Title: A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Unsupervised Skill Discovery

Determinantal Point Process

Options
Abstract: Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational option discovery that maximizes the diversity of the options through a mutual information loss (while ignoring coverage) and Laplacian-based methods that focus on improving the coverage of options by increasing connectivity of the state space (while ignoring diversity). In this paper, we show that diversity and coverage in unsupervised option discovery can indeed be unified under the same mathematical framework. To be specific, we explicitly quantify the diversity and coverage of the learned options through a novel use of Determinantal Point Process (DPP) and optimize these objectives to discover options with both superior diversity and coverage. Our proposed algorithm, ODPP, has undergone extensive evaluation on challenging tasks created with Mujoco and Atari. The results demonstrate that our algorithm outperforms state-of-the-art baselines in both diversity- and coverage-driven categories.

A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process

Abstract

Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational option discovery that maximizes the diversity of the options through a mutual information loss (while ignoring coverage) and Laplacian-based methods that focus on improving the coverage of options by increasing connectivity of the state space (while ignoring diversity). In this paper, we show that diversity and coverage in unsupervised option discovery can indeed be unified under the same mathematical framework. To be specific, we explicitly quantify the diversity and coverage of the learned options through a novel use of Determinantal Point Process (DPP) and optimize these objectives to discover options with both superior diversity and coverage. Our proposed algorithm, ODPP, has undergone extensive evaluation on challenging tasks created with Mujoco and Atari. The results demonstrate that our algorithm outperforms state-of-the-art baselines in both diversity- and coverage-driven categories.

1 Introduction

Reinforcement Learning (RL) has achieved impressive performance in a variety of scenarios, such as games [1, 2], robotic control [3, 4], and transportation [5, 6]. However, most of its applications rely on carefully-crafted, task-specific rewards to drive exploration and learning, limiting its use in real-life scenarios often with sparse or no rewards. To this end, utilizing unsupervised option discovery - acquiring rich skills without supervision of environmental rewards by building temporal action-sequence abstractions (denoted as options), to support efficient learning can be essential. The acquired skills are not specific to a single task and thus can be utilized to solve multiple downstream tasks by implementing a corresponding meta-controller that operates hierarchically on these skills.

Existing unsupervised option discovery approaches broadly fall into two categories: (1) Variational option discovery, e.g., [7, 8, 9], which aims to improve diversity of discovered options by maximizing the mutual information [10] between the options and trajectories they generate. It tends to reinforce already discovered behaviors for improved diversity rather than exploring (e.g., visiting poorlyconnected states) to discover new ones, so the learned options may have limited coverage of the state space. (2) Laplacian-based option discovery, e.g., [11, 12], which clusters the state space using a Laplacian spectrum embedding of the state transition graph, and then learns options to connect different clusters. This approach is shown to improve the algebraic connectivity of the state space [13] and reduce expected covering time during exploration. However, the discovered options focus on improving connectivity between certain clusters and thus could be homogeneous and lack diversity. Note that coverage in this paper is defined as a property with respect to a single option, which can be measured as the number of state clusters traversed by an option trajectory. By maximizing coverage

Tian Lan

(a)

(b)

(c)

20-

(d)

Diversity

Coverage

Diversity

15-

Coverage

10-

2

0

(a)

(b)

(c)

O

Figure Index

of each single option and diversity among different options in the meanwhile, the overall span of all options can be maximized. However, diversity and coverage may not go hand-in-hand in option discovery, as visualized in Figure 1(a)(b). Attempts such as [14, 15] have been made to address this gap, but they rely on expert datasets that contain diverse trajectories spanning the whole state spaces and lack an analytical framework for diversity and coverage of the discovered options.

This paper introduces a novel framework for unsupervised option discovery by utilizing Determinantal Point Process (DPP) to quantify and optimize both diversity and (single-option) coverage of the learned options. A DPP establishes a probability measure for subsets of a set of items. The expected cardinality, representing the average size of random subsets drawn according to the DPP, serves as a diversity measure, as it reflects the likelihood of sampling diverse items and the number of distinct items in the set. First , to enhance option diversity, we apply a DPP to the set of trajectories for different options. Maximizing the expected cardinality under this DPP encourages agents to explore diverse trajectories under various options. Second , we create another DPP for the set of visited states in a trajectory and maximize its expected cardinality. This prompts the agent to visit distant states from distinct state clusters through each trajectory, leading to higher single-option coverage. As shown in Figure 1(d), these objectives effectively measure diversity and coverage of the options. Lastly , to establish the option-policy mapping and so learn multiple options simultaneously, we maximize the mutual information between the option choice and related trajectories, as in variational option discovery. Rather than using the whole trajectory [16] or only the goal state [7], our solution extract critical landmark states from a trajectory via a maximum a posteriori (MAP) inference of a DPP, allowing noise mitigation while retaining vital information. In conclusion , our proposed framework unifies diversity and coverage in option discovery using a mathematical modeling tool-DPP. Our key contributions are as follows. (1) To the best of our knowledge, this is the first work to adopt DPP for option discovery. (2) The proposed unified framework enables explicit maximization of option diversity and coverage, capturing advantages of both variational and Laplacian-based methods. (3) Empirical results on a series of challenging RL tasks demonstrates the superiority of our algorithm over state-of-the-art (SOTA) baselines.

2 Background and Related Works

2.1 Unsupervised Option Discovery

As proposed in [17], the option framework consists of three components: an intra-option policy π : S × A → [0 , 1] , a termination condition β : S → { 0 , 1 } , and an initiation set I ⊆ S . An option < I, π, β > is available in state s if and only if s ∈ I . If the option is taken, actions are selected according to π until it terminates according to β (i.e., β = 1 ). The option framework enables learning and planning at multiple temporal levels and has been widely adopted in RL. Multiple research areas centered on this framework have been developed. Unsupervised Option Discovery aims at discovering skills that are diverse and efficient for downstream task learning without supervision from rewards, for which algorithms have been proposed for both single-agent and multi-agent scenarios [18, 19, 20, 21]. Hierarchical Reinforcement Learning [22, 23] and Hierarchical Imitation Learning [24, 25, 26], on the other hand, aim at directly learning a hierarchical policy incorporated with skills, either from interactions with the environment or expert demonstrations.

We focus on single-agent unsupervised option discovery. One primary research direction in this field is variational option discovery, which aims to learn a latent-conditioned option policy π ( a | s, c ) , where c represents the option latent. Diverse options can be learned by maximizing the mutual information between the option choice c and the trajectory generated by the policy conditioned on c . As shown in [27], this objective is equivalent to a Variational Autoencoder (V AE) [28]. Variational option discovery can be categorized into two groups based on the structure of the VAE used: (1) Trajectory-first methods, such as EDL [14] and OPAL [15], follow the structure τ E -→ c D -→ τ . Here, τ represents the agent's trajectory, while D and E denote the VAE's decoder and encoder, respectively. In this case, the policy network π ( a | s, c ) serves as the decoder D . The effectiveness of these methods rely on the quality of the expert trajectory set used as the encoder's input. For instance, OPAL employed for Offline RL [29] assumes access to a trajectory dataset generated by a mix of diverse policies starting from various initial states. Similarly, in EDL, highly-efficient exploration strategies such as State Marginal Matching [16] are utilized to simulate perfect exploration and obtain a set of trajectories ( τ ) with good state space coverage for option learning. (2) Option-first methods, such as VIC [7], DIAYN [8], VALOR [27], and DADS [9], follow the structure c E -→ τ D -→ c and learn π ( a | s, c ) as the encoder E . Unlike EDL and OPAL learning from "exploration experts", these methods start from a random policy to discover diverse options. However, due to challenges in exploration, they fail to expand the set of states visited by the random policy, leading to poor state space coverage. For instance, in Figure 1(a), though trajectories of different options are diverse, none of them explore the corridor. There have been more notable advancements in variational option discovery methods, as cited in [30, 31, 32, 33, 34, 35]. A detailed comparison between these methods and our algorithm, highlighting our contributions, can be found in Appendix B.

In this paper, we tackle a more challenging scenario where the agent must learn to identify diverse options and thoroughly explore the environment, starting from a random policy. This approach supports self-driven learning and exploration, rather than relying on expert policy/trajectories like trajectory-first methods. We achieve this by integrating the variational method with another key branch in unsupervised option discovery-Laplacian-based option discovery [11, 12]. This approach is based on the Laplacian spectrum of the state transition graph, which can be estimated using state transitions in the replay buffer. The state transition graph and its Laplacian matrix are formally defined in Appendix A.1. Specifically, the approach presented in [12] first estimates the Fiedler vector - the eigenvector associated with the second smallest eigenvalue of the Laplacian. Options are then trained to connect states with high and low values in the Fiedler vector. These states are loosely connected by "bottleneck" states. By connecting them with options, the algebraic connectivity of the state space is enhanced, and thus the exploration can be accelerated [13]. However, these options may be homogeneous. For instance, in Figure 1(b), multiple options are found to explore the right corridor, but they all follow the same direction given by the Fiedler vector. Inspired by variational and Laplacian-based methods, we propose a new option discovery framework that unifies the optimization of diversity and coverage, through a novel use of DPP. As shown in Figure 1(c), multiple diverse options are discovered, most of which traverse the "bottleneck" states to enhance coverage.

2.2 Determinantal Point Process

According to [36], given a set of items W = { w 1 , · · · , w N } , a point process P on W is a probability measure on the set of all the subsets of W . P is called a Determinantal Point Process (DPP) if a random subset W drawn according to P has probability:

I ∈ R N × N is the identity matrix. L = L ( W ) ∈ R N × N is the DPP kernel matrix, which should be symmetric and positive semidefinite. L W ∈ R | W |×| W | is the sub-matrix of L indexed by elements in W . Specifically, P L ( W = { w i } ) ∝ L ii and P L ( W = { w i , w j } ) ∝ L ii L jj -L 2 ij where L ij measures the similarity between item i and j . Since the inclusion of one item reduces the probability of including similar items, sets consisting of diverse items are more likely to be sampled by a DPP.

The DPP kernel matrix L can be constructed as a Gram Matrix [37]: L = ˜ B T ˜ B = Diag ( - → q ) · B T B · Diag ( - → q ) . - → q = [ q 1 , · · · , q N ] ∈ R N with q i ≥ 0 denotes the quality measure. B = [ - → b 1 · · · - → b N ] ∈ R D × N is the stacked feature matrix where - → b i ∈ R D is the feature vector corresponding to w i . The

inner product of feature vectors, e.g., - → b i T - → b j , is used as the similarity measure between items in W . From Eq. (1), we can see that P L ( W = W ) is propotional to the squared | W | -dimension volume of the parallelepiped spanned by the columns of ˜ B corresponding to the elements in W . Diverse sets have feature vectors that are more orthogonal and span larger volumes, making them more probable.

The expected cardinality of samples from a DPP is an effective measure of the diversity of W and reflects the number of modes in W [38]. We provide detailed reasons why we choose the expected cardinality instead of the likelihood in Eq. (1) as the diversity measure in Appendix C.1. According to [36], the expected cardinality of a set sampled from a DPP can be calculated with Eq. (2), where λ W i is the i -th eigenvalue of L ( W ) . DPPs have found applications across a wide array of domains to promote diversity due to their unique ability to model diverse subsets, such as information retrieval [39, 40], computer vision [38], natural language processing [41, 42], and reinforcement learning [43, 44, 45]. This motivates us to employ DPPs in skill discovery for diversity enhancement.

3 Proposed Approach

In this section, we propose a DPP-based framework that unifies the variational and Laplacian-based option discovery to get options with both superior diversity and coverage. As discussed in Section 2.1, defining an option requires relating it to its initial states s 0 , specifying its termination condition, and learning the intra-option policy. Our algorithm learns a prior network P ω ( c | s 0 ) to determine the option choice c ∈ C at an initial state, and an intra-option policy network π θ ( a | s, c ) to interact with the environment using a certain option for a fixed number of steps (i.e., the termination condition). With P ω and π θ , we can collect trajectories τ = ( s 0 , a 0 , · · · , s T ) corresponding to different options.

In Section 3.1, we propose L IB , a lower bound for the mutual information between the option and the landmark states in its trajectory, inspired by variational option discovery. By introducing the conditional variable c and maximizing L IB , we can establish the option-policy mapping and learn multiple options simultaneously. Each option can generate specific trajectories. However, the mutual information objective only implicitly measures option diversity as the difficulty of distinguishing them via the variational decoder, and does not model the coverage. Thus, in Section 3.2, we additionally introduce explicit coverage and diversity measures based on DPP as objectives. In particular, L DPP 1 measures single-agent coverage as the number of landmark states traversed by an option trajectory. Maximizing this metric encourages each option to cover a broader area in the state space. Notably, L DPP 1 generalizes the Laplacian-based option discovery objectives by employing the Laplacian spectrum as the feature vector to construct the DPP kernel matrix. L DPP 2 measures the diversity among trajectories of the same option. By minimizing it, the consistency these trajectories can be enhanced. L DPP 3 measures the diversity among trajectories corresponding to different options, which should be maximized to improve the diversity among various options. The rationale behind introducing L DPP 1:3 is to maximize both the coverage of each individual option and the diversity among various options. By doing so, we can maximize the overall span of all options, thereby fostering the formation of diverse skills that fully explore the state space.

3.1 Landmark-based Mutual Information Maximization

Previous works tried to improve the diversity of learned options by maximizing the mutual information between the option choice c and trajectory τ [27] or goal state s T [46] generated by the corresponding intra-option policy. However, the whole trajectory contains noise and the goal state only cannot sufficiently represent the option policy. In this paper, we propose to maximize the mutual information between c and landmark states G in τ instead. G is a set of distinct, representative states. Specifically, after clustering all states according to their features, a diverse set of landmarks with varied feature embeddings is identified to represent each different cluster. Notably, G can be extracted from τ through the maximum a posteriori (MAP) inference of a DPP [36], shown as Eq. (3) where X denotes the set of states in τ . The intuition is that these landmarks should constitute the most diverse subset of X and thus should be the most probable under this DPP.

In order to maximize the mutual information between G and c while filtering out the redundant information for option discovery in τ . According to the Information Bottleneck framework [47], this can be realized through Eq. (4), where µ ( · ) is the initial state distribution, I ( · ) denotes the mutual information, and I ct is the information constraint.

Equivalently, with the introduction of a Lagrange multiplier β ≥ 0 , we can optimize:

It is infeasible to directly compute and optimize Eq. (5), so we have the following proposition.

Proposition 1. The optimization problem as Eq. (5) can be solved by maximizing L IB ( ω, θ, ϕ ) :

Here, s 0 ∼ µ ( · ) , c ∼ P ω ( ·| s 0 ) , τ ∼ P θ ( ·| s 0 , c ) . H ( C|S ) represents the entropy associated with the option choice distribution at a given state. P DPP ( G | τ ) = det( L G ) / det( L + I ) is the probability of extracting G from τ , under a DPP on the set of states in τ . P ϕ ( c | s 0 , G ) serves as a variational estimation of the posterior term P ( c | s 0 , G ) in I ( c, G | s 0 ) . Unif ( τ | s 0 ) denotes the probability of sampling trajectory τ given s 0 under a uniformly random walk policy.

The derivation can be found in Appendix A.2. (1) The first two terms of Eq. (6) constitute a lower bound of the first term in Eq. (5). H ( C|S ) and P ϕ ( c | s 0 , G ) can be estimated using the output from the prior network and variational posterior network, respectively. (2) The third term in Eq. (6) corresponds to a lower bound of the second term in Eq. (5). Instead of directly calculating -βI ( c, τ | s 0 ) which is implausible, we introduce Unif ( τ | s 0 ) to convert it to a regularization term as in Eq. (6). Specifically, by minimizing D KL ( P θ ( τ | s 0 , c ) || Unif ( τ | s 0 )) , which is the KL Divergence [10] between the distribution of trajectories under our policy and a random walk policy, exploration of the trained policy π θ can be improved. Note that P θ ( τ | s 0 , c ) = ∏ T -1 t =0 π θ ( a t | s t , c ) P ( s t +1 | s t , a t ) , where P ( s t +1 | s t , a t ) is the transition function in the MDP.

Another challenge in calculating L IB is to infer the landmarks G with Eq. (3). ( L is the kernel matrix of the DPP, of which the construction is further introduced in the next section.) The MAP inference problem related to a DPP is NP-hard [48], and greedy algorithms have shown to be promising as a solution. In this work, we adopt the fast greedy method proposed in [39] for the MAP inference, which is further introduced in Appendix C.2 with its pseudo code and complexity analysis.

3.2 Quantifying Diversity and Coverage via DPP

In this section, we propose three optimization terms defined with DPP to explicitly model the coverage and diversity of the learned options. Jointly optimizing these terms with Eq. (6), the discovered options are expected to exhibit improved state space coverage and enhanced diversity.

The first term relates to improving the coverage of each option by maximizing:

where X is the set of states in τ , L ( X ) is the kernel matrix built with feature vectors of states in X , f ( τ ) is the expected number of modes (landmark states) covered in a trajectory τ . Thus, L DPP 1 measures single-option coverage, by maximizing which we can enhance exploration of each option.

As for the kernel matrix, according to Section 2.2, we can construct L ( X ) by defining the quality measure - → q and normalized feature vector for each state in X . States with higher expected returns should be visited more frequently and thus be assigned with higher quality values. However, given that there is no prior knowledge on the reward function, we assign equal quality measure to each state as 1. As for the features of each state, we define them using the Laplacian spectrum, i.e., eigenvectors corresponding to the D -smallest eigenvalues of the Laplacian matrix of the state transition graph, denoted as [ - → v 1 , · · · , - → v D ] . To be specific, for each state s , its normalized feature is defined as: - → b ( s ) = [ - → v 1 ( s ) , · · · , - → v D ( s )] / √ ∑ D j =1 ( - → v j ( s )) 2 . The reasons for this feature design are as follows:

(1) As shown in Spectral Clustering [49], states with high similarity in this feature embedding fall in

Algorithm 1 Unsupervised Option Discovery based on DPP (ODPP)

1: Initialize the prior network P ω , policy network π θ , variational decoder P ϕ , and trajectory set - → τ

2: for each training episode do

3: for i = 1 , 2 , . . . , N do

4: Sample s i 0 ∼ µ ( · ) and c ∼ P ω ( ·| s i 0 ) or use the ones from the previous episode to collect trajectories subject to the same option and starting point

5: Collect a trajectory τ i = ( s i 0 , a i 0 , · · · , s i T -1 , a i T -1 , s i T ) , where a i t ∼ π θ ( ·| s i t , c )

6: - → τ ←- - → τ ∪ { τ i }

7: end for

8: Update P ω , π θ with PPO and P ϕ with SGD, based on - → τ and Eq. (11)-(13); - → τ ←- {}

9: end for

the same cluster. Under a DPP with this feature embedding, the sampled states with distinct features should belong to different clusters, i.e., the landmarks. Then, by maximizing L DPP 1 (i.e., the expected number of landmarks covered in X ), the agent is encouraged to traverse multiple clusters within the state space. (2) With this feature design, our algorithm generalizes of the Laplacian-based option discovery [12]. In [12], they set a threshold to partition the state space into two parts - the set of states with higher values in the Fiedler vector (i.e., - → v 2 ) than the threshold is used as the initiation set and the other states are used as the termination set, then the option policy is trained to connect states within these two areas. We note that, as a special case of our algorithm, when setting the feature dimension D = 2 , we can get similar options with the Laplacian-based methods through maximizing L DPP 1 . Given that the eigenvector corresponding to the smallest eigenvalue of a Laplacian matrix is - → v 1 = ⃗ 1 , states with diverse feature embeddings encoded by [ - → v 1 , - → v 2 ] (i.e., D = 2 ) tend to differ in - → v 2 . By maximizing L DPP 1 , the options are trained to visit the states that are as distinct in - → v 2 as possible in a trajectory, which is similar with the ones learned in [12]. We empirically demonstrate this in Section 4. Note that the Laplacian spectrum [ - → v 1 , · · · , - → v D ] for infinite-scale state space can be learned as a neural network through representation learning [50] which is introduced in Appendix C.3. Thus, our algorithm can be applied to tasks with large-scale state spaces.

Next, we expect the set of sampled trajectories related to the same option c and starting from the same state s 0 , i.e., - → τ ( s 0 ,c ) , to be consistent and thus hard to be distinguished by a DPP, which is important given the stochasticity of the policy output. This is realized by minimizing the expected mode number in each - → τ ( s 0 ,c ) : ( s 0 ∼ µ ( · ) , c ∼ P ω ( ·| s 0 ) , - → τ ( s 0 ,c ) ∼ P θ ( ·| s 0 , c ) )

where Y is the set of M trajectories related to c starting at s 0 (i.e., { τ 1 , · · · , τ M } = - → τ ( s 0 ,c ) ), L ( Y ) is the kernel matrix built with the feature vectors of each trajectory, i.e., - - → b ( τ i ) . The feature vector of a trajectory can be obtained based on the ones of its landmark states G i through the Structured DPP framework [51]: - - → b ( τ i ) = ∑ s ∈ G i - - → b ( s ) . ( - - → b ( s ) is defined above.) Alternatively, we can use the hidden layer output of the decoder P ϕ ( c | G,s 0 ) , which embeds the information contained in G . This design is commonly adopted in DPP-related works [52, 53]. We will compare these two choices in Section 4. Notably, our kernel matrix design, i.e, L ( X ) and L ( Y ) , is task-irrelevant and domain-knowledge-free.

Last, the set of sampled trajectories subject to different options should be diverse. This is achieved by maximizing its expected mode number (i.e., Eq. (9)). Here, Z is the union set of Y related to different options, i.e., ∪ c ′ - → τ ( s 0 ,c ′ ) , and L ( Z ) is the kernel matrix corresponding to trajectories in Z .

3.3 Overall Algorithm Framework

The overall objective is to maximize Eq. (10). The hyperparameters α 1:3 ≥ 0 (provided in Appendix C.6) are the weights for each DPP term and can be fine-tuned to enable a tradeoff between coverage and diversity of the learned options.

Based on Eq. (10), we can calculate the gradients with respect to ω, θ, ϕ , i.e., the parameters of the prior network, policy network and variational decoder, and then apply corresponding algorithms for optimization. First, ∇ ϕ L = ∇ ϕ L IB , so P ϕ can be optimized as a standard likelihood maximization problem with SGD [54]. Next, regarding ω and θ , we have Proposition 2, which is proved in Appendix A.3. Notably, we select PPO [55] to update ω and θ . The overall algorithm (ODPP) is summarized as Algorithm 1, where the main learning outcome is the intra-option policy π θ . When applied to downstream tasks, π θ ( a | s, c ) can be fixed and we only need to learn a high-level policy P ψ ( c | s ) to select among options. We note that our algorithm is highly salable and only slightly increases the time complexity compared with previous algorithms in this field. Detailed discussion on the complexity and scalability are provided in Appendix C.4 and C.5.

Proposition 2. The gradients of the overall objective L ( ω, θ, ϕ ) with respect to ω and θ can be unbiasedly estimated using Eq. (11). Here, Q P ω and Q π θ m are the Q-functions for the prior and policy networks, respectively. Consequently, both networks can be trained using reinforcement learning.

4 Evaluation and Main Results

In this section, we compare ODPP with SOTA baselines on a series of RL tasks. (1) For intuitive visualization, we test these algorithms on maze tasks built with Mujoco [56]. In particular, we select the Point and Ant as training agents, and put them in complex 3D Mujoco Maze environments (Figure 4(a) and 4(d)). We evaluate the diversity and coverage of the options learned with different algorithms, through visualizing corresponding trajectories. Then, we provide a quantitative study to see if these options can aid learning in downstream tasks - goal-achieving or exploration tasks in the maze environments. Both tasks are long-horizon with an episode length of 500. (2) To show the applicability of our algorithm on a wide range of tasks, we test it on 3D Ant locomotion tasks (Figure 5(a)) and Atari video games. In this part, we focus on evaluating if the agent can learn effective skills without supervision of task-specific rewards.

As mentioned in Section 2.1, our algorithm follows the option-first variational option discovery, which starts from a random policy rather than an efficient exploration policy like the trajectory-first methods. To keep it fair, we compare our algorithm with several SOTA option-first methods: VIC [7], DIAYN [8], VALOR [27], DADS [9], and APS [57]. Further, our algorithm integrates the variational and Laplacian-based option discovery, so we compare it with a SOTA Laplacian-based method as well: DCO [12]. The codes are available at https://github.com/LucasCJYSDL/ODPP.

4.1 Ablation Study

In Figure 2, we visualize trajectories of options learned with different algorithms in the Point Room task, which start from the same initial state and have the same horizon (i.e., 50 steps). Note that the visualizations in Figure 1-3 are aerial views of the 3D Mujoco Room/Corridor. As mentioned in Section 3.2, the objective of DCO is to train options that can connect states with high and low values in the Fielder vector of the state transition graph. Due to its algorithm design [12], we can only learn one option with DCO at a time. While, with the others, we can learn multiple options simultaneously. (1) From (a)-(c), we can see that variational option discovery methods can discover diverse options, but these options can hardly approach the "bottleneck" states in the environment which restricts their coverage of the state space. On the contrary, in (d), options trained with DCO can go through two "bottleneck" states but lack diversity, since they stick to the direction given by the Fiedler vector. While, as shown in (h), options learnt with our algorithm have both superior diversity

(a)

(b)

(C)

(d)

(e)

(

(g)

(h)

and coverage. (2) In (e), we can already get significant better options than the baselines by only using L IB as the objective. From (e) to (f), it can be observed that additionally introducing the objective term L DPP 1 can encourage options to cover more landmark states. From (f) to (h), we further add L DPP 2 and L DPP 3 , which makes the option trajectories more distinguishable from the view of DPP. Also, in (g) and (h), we adopt different designs of the trajectory feature. It shows that using trajectory features defined with the Structured DPP framework (introduced in Section 3.2) is better than using the hidden layer output of the decoder P ϕ . These results show the effectiveness of each component in our algorithm design and demonstrate that our algorithm can construct options with higher diversity and coverage than baselines. In Appendix D.3, we provide quantitative results to further show the improvement brought by each objective term of ODPP.

Next, as claimed in Section 3.2, if setting the feature dimension D = 2 , ODPP is expected to learn similar options with DCO, thus ODPP generalizes the Laplacian-based method. In Figure 3, we visualize the value of each state in the Fiedler vector as the background color (the darker the higher), and the options learned with DCO and ODPP in the Point Corridor task starting from different locations. We can observe from (a) and (b) that most of the learned options are similar both in direction and

(a) DCO

(b) ODPP (

D

= 2

)

(c) ODPP

口

G

length. Further, if we adopt the normal setting, where D = 30 and the number of options to learn at a time is 10, we can get diverse options shown as (c), which can be beneficial for various downstream tasks. In this case, ODPP can be viewed as a generalization and extension of DCO through variational tools (e.g., L IB ) to learn multiple diverse options at a time.

4.2 Evaluation in Downstream Tasks

In Figure 4, we evaluate the options learned with different algorithms on a series of downstream tasks. These options are trained without task-specific rewards, and thus potentially applicable to different downstream tasks in the environment. Firstly, we test the options in Point Room/Corridor goal-achieving tasks where a point agent is trained to achieve a certain goal (i.e., red points in Figure 4(a) and 4(d)). This task is quite challenging since: (1) The location of the goal is not included in the observation. (2) The agent will get a positive reward only when it reaches the goal area; otherwise, it will receive a penalty. Hence, the reward setting is highly sparse and delayed, and the agent needs to fully explore the environment for the goal state. In Figure 4(b)-4(c), we compare the mean and standard deviation of the performance (i.e., mean return of a decision step) of different algorithms

(a) Mujoco Room

(b) Point Room Task

(c) Point Corridor Task

(d) Mujoco Corridor

(e) Ant Room Task

0.20

algorithm

0.20

algorithm

0.15

DCO

DIAYN

VALOR

ODPP

0.15

DIAYN

VALOR

ODPP

DCO

Reward

0.10

VIC

Base RL

0.10

vBaseRL

0.05

0.05

0.00

0.00

2000

4000

Episode

6000

8000

10000

2000

4000

Episode

6000

8000

10000

0.6

0.5

algorithm

0.4

Reward

0.3

0.2

0.1

0.4

DIAYN

VALOR

DCO

VIC

Base-RL

ODPP

Reward

0.3

0.1

0.2

algorithm

VALOR

ODPP

DIAYN

DCO

0.0

0.0

VIC

Base RL

1000

2000

Episode

3000

4000

5000

1000

2000

Episode

3000

4000

5000

in the training process, which is repeated four times (each time with a different goal). It can be observed that, with options learned by ODPP, the convergence speed and return value can be much higher. Note that the pretrained options are fixed for downstream tasks and we only need to learn an option selector P ψ ( c | s ) which gives out option choice c at state s . In this way, we can simplify a continuous control task to a discrete task, and the advantage can be shown through the comparison with using PPO (i.e., 'Base RL") directly on the task. To keep it fair, the PPO agent is pretrained for the same number of episodes as the option learning. Moreover, we evaluate these algorithms in Ant Room/Corridor exploration tasks where an Ant agent is trained to explore the areas as far from the start point (center) as possible. The reward for a trajectory is defined with the largest distance that the agent has ever reached during this training episode. In Figure 4(e)-4(f), we present the change of the trajectory return during the training process of these algorithms (repeated five times with different random seeds). The options trained with ODPP provide a good initialization of the policy for this exploration task. With this policy, the Ant agent can explore a much larger area in the state space.

As mentioned in Section 3, we learn a prior network P ω together with the option policy network π θ . This design is different from previous algorithms which choose to fix the prior distribution. In Appendix D.1, we provide a detailed discussion on this and empirically show that we can get a further performance improvement in the downstream task by initializing the option selector P ψ with P ω .

4.3 Performance of Unsupervised Skill Discovery Across Various Benchmarks

In the 3D Locomotion task, an Ant agent needs to coordinate its four legs to move. In Figure 5(a), we visualize two of the controlling behaviors, learned by ODPP without supervision of any extrinsic rewards. Please refer to Appendix D.2 for more visualizations. The picture above shows that the Ant rolls to the right by running on Leg 1 first and then Leg 4, which is a more speedy way to move ahead. While, the picture below shows that it learns how to walk to the right by stepping on its front legs (2&3) and back legs (1&4) by turn. These complex behaviors are learned with only intrinsic objectives on diversity and coverage, which is quite meaningful. Further, we show numeric results to evaluate the learned behaviors. We use the reward defined in [58] as the metric, which is designed to encourage the Ant agent to move as fast as possible at the least control cost. In Figure 5(b), we take the average of five different random seeds, and the result shows that options trained by ODPP outperform the baselines. The reward drops during training for some baselines, which is reasonable since this is not the reward function used for option learning. Finally, to see whether we can learn a large number of options in the meantime with ODPP, we test the performance of the discovered options when setting the number of options to learn as 10, 20, 40, 60. From Figure 5(c), it can be observed that even when learning a large number of options at the same time, we can still get options with high quality (mean) and diversity (standard deviation) which increase during the training.

a

3

3

50

O

(b)

60

40

(c)

algorithm

20

1

3

4

1

3

1

4

3

4

3

Reward

-100

150

50

DIAYN

DGO

VIC

VALOR

ODPP

Reward

40

60

20

O

algorithm

40 Skills

60Skills

10 Skills

1

2

2

1

2

1

2

1

2

1

2

-200

O

00000000000900000000000000000000

Step

-80

00000000000900000007000000000000

Step

20 Skills

(a) CartPole

(b) RiverRaid

(c) AirRaid

100

06

140

80

80

120

70

Reward

Reward

Reward

60

100

09

50

40

algorithm

80

algorithm

40

algorithm

mmTm

ODPP

ODPP

ODPP

20

0

2000

4000

Step

6000

8000

APS

DADS

10000

60

2000

4000

Step

0009

8000

APS

DADS

10000

30

20

0

2000

4000

Step

6000

8000

APS

DADS

10000

Figure 6: Evaluation results on OpenAI Gym and Atari games to show the superiority of our algorithm on more general tasks. Our algorithm performs the best in all the three tasks, and the performance improvement becomes more significant as the task difficulty increases.

To further demonstrate the applicability of ODPP, we compare it with SOTA baselines on more general OpenAI Gym and Atari tasks [59]. Notably, we adopt two more advanced skill discovery algorithms as baselines: DADS and APS. Comparisons among these two and previous baselines are in Appendix D.4. Skills discovered with different algorithms are evaluated with reward functions carefully-crafted for each task, provided by OpenAI Gym. For each algorithm, we learn 10 skills, of which the average cumulative rewards (i.e., sum of rewards within a skill duration) in the training process are shown in Figure 6. The skill duration is set as 100 for CartPole and 50 for the other two. Note that the complete episode horizon is 200 for CartPole and 10000 for AirRaid and RiverRaid. Thus, it would be unfair to compare the cumulative reward of a skill with the one of a whole episode. Our algorithm performs the best in all the three tasks, and the improvement becomes more significant as the task difficulty increases. When relying solely on MI-based objectives, the agent tends to reinforce already discovered behaviors for improved diversity rather than exploration. The explicit incorporation of coverage and diversity objectives in our algorithm proves beneficial in this case.

5 Conclusion and Discussion

ODPP is a novel unsupervised option discovery framework based on DPP, which unifies variational and Laplacian-based option discovery methods. Building upon the information-theoretic objectives in prior variational research, we propose three DPP-related measures to explicitly quantify and optimize diversity and coverage of the discovered options. Through a novel design of the DPP kernel matrix based on the Laplacian spectrum of the state transition graph, ODPP generalizes SOTA Laplacianbased option discovery algorithms. We demonstrate the superior performance of ODPP over SOTA baselines using variational and Laplacian-based methods on a series of challenging benchmarks. Regarding limitations of ODPP, the primary one is to assign suitable weights for each objective term: α 1:3 in Eq. (10) and β in Eq. (6). These hyperparameters are crucial, yet conducting a grid search for their joint selection would be exhaustive. Instead, we employ a sequential, greedy method to select hyperparameters in accordance with our ablation study's procedure, further detailed in Appendix C.6.

Acknowledgments and Disclosure of Funding

This project is supported by ONR award N00014-23-1-2850 and CISCO research award 76934189.

References

Brown, N., T. Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):885-890, 2019.

Silver, D., A. Huang, C. J. Maddison, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

Ebert, F., C. Finn, S. Dasari, A. Xie, A. Lee, S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568 , 2018.

Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Chen, J., A. K. Umrawal, T. Lan, V. Aggarwal. Deepfreight: A model-free deep-reinforcementlearning-based algorithm for multi-transfer freight delivery. In Proceedings of the 31st International Conference on Automated Planning and Scheduling , pages 510-518. AAAI Press, 2021.

Al-Abbasi, A. O., A. Ghosh, V. Aggarwal. Deeppool: Distributed model-free algorithm for ridesharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 20(12):4714-4727, 2019.

Gregor, K., D. J. Rezende, D. Wierstra. Variational intrinsic control. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017 . OpenReview.net, 2017.

Eysenbach, B., A. Gupta, J. Ibarz, S. Levine. Diversity is all you need: Learning skills without a reward function. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019 . OpenReview.net, 2019.

Sharma, A., S. Gu, S. Levine, V. Kumar, K. Hausman. Dynamics-aware unsupervised discovery of skills. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, 2020.

Cover, T. M. Elements of information theory . John Wiley & Sons, 1999.

Jinnai, Y., J. W. Park, D. Abel, G. D. Konidaris. Discovering options for exploration by minimizing cover time. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019 , vol. 97 of Proceedings of Machine Learning Research , pages 3130-3139. PMLR, 2019.

Jinnai, Y., J. W. Park, M. C. Machado, G. D. Konidaris. Exploration in reinforcement learning with deep covering options. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, 2020.

Ghosh, A., S. P. Boyd. Growing well-connected graphs. In Proceedings of the 45th IEEE Conference on Decision and Control, CDC 2006 , pages 6605-6611. IEEE, 2006.

Campos, V., A. Trott, C. Xiong, R. Socher, X. Giró-i-Nieto, J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , vol. 119 of Proceedings of Machine Learning Research , pages 1317-1327. PMLR, 2020.

Ajay, A., A. Kumar, P. Agrawal, S. Levine, O. Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021.

Lee, L., B. Eysenbach, E. Parisotto, E. P. Xing, S. Levine, R. Salakhutdinov. Efficient exploration via state marginal matching. CoRR , abs/1906.05274, 2019.

Sutton, R. S., D. Precup, S. P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112(1-2):181-211, 1999.

Chen, J., J. Chen, T. Lan, V. Aggarwal. Multi-agent covering option discovery based on kronecker product of factor graphs. IEEE Transactions on Artificial Intelligence , 2022.

-. Learning multi-agent options for tabular reinforcement learning using factor graphs. IEEE Transactions on Artificial Intelligence , pages 1-13, 2022.

Zhang, F., C. Jia, Y.-C. Li, L. Yuan, Y. Yu, Z. Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In Proceedings of the 11th International Conference on Learning Representations . 2022.

Chen, J., M. Haliem, T. Lan, V. Aggarwal. Multi-agent deep covering option discovery. arXiv preprint arXiv:2210.03269 , 2022.

Zhang, S., S. Whiteson. DAC: the double actor-critic architecture for learning options. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems , pages 2010-2020. 2019.

Li, C., D. Song, D. Tao. The skill-action architecture: Learning abstract action embeddings for reinforcement learning. In Submissions of the 9th International Conference on Learning Representations . 2021.

Jing, M., W. Huang, F. Sun, X. Ma, T. Kong, C. Gan, L. Li. Adversarial option-aware hierarchical imitation learning. In Proceedings of the 38th International Conference on Machine Learning , pages 5097-5106. PMLR, 2021.

Chen, J., T. Lan, V. Aggarwal. Option-aware adversarial inverse reinforcement learning for robotic control. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5902-5908. IEEE, 2023.

Chen, J., D. Tamboli, T. Lan, V. Aggarwal. Multi-task hierarchical adversarial inverse reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning , vol. 202 of Proceedings of Machine Learning Research , pages 4895-4920. PMLR, 2023.

Achiam, J., H. Edwards, D. Amodei, P. Abbeel. Variational option discovery algorithms. CoRR , abs/1807.10299, 2018.

Kingma, D. P., M. Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014 . 2014.

Levine, S., A. Kumar, G. Tucker, J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR , abs/2005.01643, 2020.

Hansen, S., W. Dabney, A. Barreto, D. Warde-Farley, T. V. de Wiele, V. Mnih. Fast task inference with variational intrinsic successor features. In Proceedings of the 8th International Conference on Learning Representations . OpenReview.net, 2020.

Baumli, K., D. Warde-Farley, S. Hansen, V. Mnih. Relative variational intrinsic control. In Thirty-Fifth AAAI Conference on Artificial Intelligence , pages 6732-6740. AAAI Press, 2021.

Hansen, S., G. Desjardins, K. Baumli, D. Warde-Farley, N. Heess, S. Osindero, V. Mnih. Entropic desired dynamics for intrinsic control. In Advances in Neural Information Processing Systems 34 , pages 11436-11448. 2021.

Kamienny, P., J. Tarbouriech, S. Lamprier, A. Lazaric, L. Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. In Proceedings of the 10th International Conference on Learning Representations . OpenReview.net, 2022.

Strouse, D., K. Baumli, D. Warde-Farley, V. Mnih, S. S. Hansen. Learning more skills through optimistic exploration. In Proceedings of the 10th International Conference on Learning Representations . OpenReview.net, 2022.

Liu, H., P. Abbeel. APS: active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning , vol. 139, pages 6736-6747. PMLR, 2021.

Kulesza, A., B. Taskar. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning , 5(2-3):123-286, 2012.

Elfeki, M., C. Couprie, M. Rivière, M. Elhoseiny. GDPP: learning diverse generations using determinantal point processes. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019 , vol. 97 of Proceedings of Machine Learning Research , pages 1774-1783. PMLR, 2019.

Gong, B., W. Chao, K. Grauman, F. Sha. Diverse sequential subset selection for supervised video summarization. In Advances in Neural Information Processing Systems 27, NeurIPS 2014 , pages 2069-2077. 2014.

Chen, L., G. Zhang, E. Zhou. Fast greedy MAP inference for determinantal point process to improve recommendation diversity. In Advances in Neural Information Processing Systems 31, NeurIPS 2018 , pages 5627-5638. 2018.

Wilhelm, M., A. Ramanathan, A. Bonomo, S. Jain, E. H. Chi, J. Gillenwater. Practical diversified recommendations on youtube with determinantal point processes. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management , pages 2165-2173. ACM, 2018.

Perez-Beltrachini, L., M. Lapata. Multi-document summarization with determinantal point process attention. Journal of Artificial Intelligence Research , 71:371-399, 2021.

Song, Y., R. Yan, Y. Feng, Y. Zhang, D. Zhao, M. Zhang. Towards a neural conversation model with diversity net using determinantal point processes. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence , pages 5932-5939. AAAI Press, 2018.

Parker-Holder, J., A. Pacchiano, K. M. Choromanski, S. J. Roberts. Effective diversity in population based reinforcement learning. In Advances in Neural Information Processing Systems 33 . 2020.

Nieves, N. P., Y. Yang, O. Slumbers, D. H. Mguni, Y. Wen, J. Wang. Modelling behavioural diversity for learning in open-ended games. In Proceedings of the 38th International Conference on Machine Learning , vol. 139, pages 8514-8524. PMLR, 2021.

Wu, S., J. Yao, H. Fu, Y. Tian, C. Qian, Y. Yang, Q. Fu, W. Yang. Quality-similar diversity via population based reinforcement learning. In Proceedings of the 11th International Conference on Learning Representations . OpenReview.net, 2023.

Kwon, T. Variational intrinsic control revisited. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021.

Alemi, A. A., I. Fischer, J. V. Dillon, K. Murphy. Deep variational information bottleneck. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017 . OpenReview.net, 2017.

Ko, C., J. Lee, M. Queyranne. An exact algorithm for maximum entropy sampling. Operation Research , 43(4):684-691, 1995.

Ng, A. Y., M. I. Jordan, Y. Weiss. On spectral clustering: Analysis and an algorithm. In Advances in Neural Information Processing Systems 14, NeurIPS 2001 , pages 849-856. MIT Press, 2001.

Wang, K., K. Zhou, Q. Zhang, J. Shao, B. Hooi, J. Feng. Towards better laplacian representation in reinforcement learning with generalized graph drawing. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021 , vol. 139 of Proceedings of Machine Learning Research , pages 11003-11012. PMLR, 2021.

Kulesza, A., B. Taskar. Structured determinantal point processes. In Advances in Neural Information Processing Systems 23, NeurIPS 2010 , pages 1171-1179. Curran Associates, Inc., 2010.

Yuan, Y., K. M. Kitani. Diverse trajectory forecasting with determinantal point processes. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, 2020.

Yang, Y., Y. Wen, J. Wang, L. Chen, K. Shao, D. Mguni, W. Zhang. Multi-agent determinantal q-learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , vol. 119 of Proceedings of Machine Learning Research , pages 10757-10766. PMLR, 2020.

Amari, S. Backpropagation and stochastic gradient descent method. Neurocomputing , 5(3):185196, 1993.

Schulman, J., F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Todorov, E., T. Erez, Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

Liu, H., P. Abbeel. APS: active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning , vol. 139, pages 6736-6747. PMLR, 2021.

Schulman, J., P. Moritz, S. Levine, M. I. Jordan, P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the 4th International Conference on Learning Representations, ICLR 2016 . 2016.

Hosu, I., T. Rebedea. Playing atari games with deep reinforcement learning and human checkpoint replay. CoRR , abs/1607.05077, 2016.

Fiedler, M. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal , 23(2):298305, 1973.

A Notations and Proof

A.1 Basic Concepts and Notations

Markov Decision Process (MDP): The reinforcement learning problem can be described with an MDP, denoted by M = ( S , A , P , R , γ ) , where S is the state space, A is the action space, P : S × A × S → [0 , 1] is the state transition function, R : S × A → R 1 is the reward function, and γ ∈ (0 , 1] is the discount factor.

State transition graph in an MDP: The state transitions in M can be modelled as a state transition graph G = ( V G , E G ) , where V G is a set of vertices representing the states in S , and E G is a set of undirected edges representing state adjacency in M . We note that:

Remark. There is an edge between state s and s ′ (i.e., s and s ′ are adjacent) if and only if ∃ a ∈ A , s.t. P ( s, a, s ′ ) > 0 ∨ P ( s ′ , a, s ) > 0 .

The adjacency matrix A of G is an |S| × |S| matrix whose ( i, j ) entry is 1 when s i and s j are adjacent, and 0 otherwise. The degree matrix D is a diagonal matrix whose entry ( i, i ) equals the number of edges incident to s i . The Laplacian matrix of G is defined as L = D -A . Its second smallest eigenvalue λ 2 ( L ) is called the algebraic connectivity of the graph G , and the corresponding normalized eigenvector is called the Fiedler vector [60]. Last, the normalized Laplacian matrix is defined as L = D -1 2 LD -1 2 .

A.2 Proof of Proposition 1

To start with, we can find a lower bound of the second term in Eq. (5) as follows:

(14)

where D KL ( · ) denotes the Kullback-Leibler (KL) Divergence which is non-negative, P θ ( τ | s 0 , c ) = ∏ T -1 t =0 π θ ( a t | s t , c ) P ( s t +1 | s t , a t ) is the probability of the trajectory τ given s 0 and c under the option policy π θ , and Unif ( τ | s 0 ) is the probability of the same trajectory given s 0 under the random walk policy. Instead of explicitly calculating P ( τ | s 0 ) which is impractical, we introduce Unif ( τ | s 0 ) to convert the second term in Eq. (5) into a regularization term to encourage exploration and diversity.

As for the first term in Eq. (5), we can deal with it as Eq. (15), where we introduce P ϕ ( c | s 0 , G ) as the variational estimation of P ( c | s 0 , G ) which is hard to acquire. The first inequality in Eq. (15) is based on the fact that KL Divergence is non-negative. While, the second inequality holds because we only keep the trajectory τ from which G is sampled, so the trajectory and its corresponding landmark states form a bijection.

A.3 Proof of Proposition 2

First, we take the gradient with respect to ω and get the following result:

Given that ∇ ω P ω ( c | s 0 ) = P ω ( c | s 0 ) ∇ ω log P ω ( c | s 0 ) and the definition of KL Divergence, i.e., D KL ( P θ ( τ | s 0 , c ) || Unif ( τ | s 0 )) = ∑ τ P θ ( τ | s 0 , c ) ∑ T -1 t =0 [log π θ ( a t | s t , c ) -log π unif ( a t | s t )] , we can simplify Eq. (16) as:

where the related Q-function Q P ω ( c, s 0 ) is defined as:

Next, we calculate the gradient with respect to θ as follows:

With ∇ θ P θ ( τ | s 0 , c ) = P θ ( τ | s 0 , c ) ∇ θ log P θ ( τ | s 0 , c ) = P θ ( τ | s 0 , c ) ∑ T -1 t =0 ∇ θ log π θ ( a t | s t , c ) , and ∇ θ P θ ( - → τ | s 0 , c ) = P θ ( - → τ | s 0 , c ) ∑ M m =1 ∑ T -1 t =0 ∇ θ log π θ ( a t m | s t m , c ) where s t m ( a t m ) is the state (action) at step t in trajectory m , Eq. (19) can be written as follows:

(20) where the advantage term is as Eq. (21), - → τ = { τ 1 , · · · , τ M } , τ m = ( s m 0 , a m 0 , · · · , s m T -1 , a m T -1 , s m T ) :

Then, it's not hard to see the relationship between Q P ω and Q π θ m as:

B Comparisons with Recent Variational Option Discovery Algorithms

Here, we provide comparisons of our algorithm with more recent variational option discovery methods [30, 31, 32, 33, 34, 35].

In [30], the authors focus on alternative approaches to leverage options learned during the unsupervised phase, rather than new option discovery algorithms. They still adopt objectives based on Mutual Information (MI) as previous works without solving the exploration issue.

The authors of [31] propose a slightly-modified version of VIC to improve usefulness of the discovered options by introducing an extra posterior. Still, their options are not explicitly trained for better coverage/exploration like ours.

The authors of [32] propose to replace the fixed prior distribution P ( c ) with a fixed dynamics model over the option latent codes P ( c t | c t -1 ) . Each latent code corresponds to a sub-trajectory. By concatenating sub-trajectories, the agent can reach much further states. They only rely on MI-based objectives, so they cannot model the coverage of each option (like ours) and instead choose to chain options for better overall coverage. The fixed P ( c t | c t -1 ) can result in inflexibility when applying options in downstream tasks.

In [33], they employ a multi-step protocol to generate options organized in a tree-like structure. Heuristics and structural limits are involved in each step, which may hinder its generality. Also, they propose to optimize the local coverage around the final state rather than the overall trajectory coverage like ours.

The authors of [34] note that, with variational objectives proposed in VIC and DIAYN, the agent can be discouraged from seeking out new states, since the variational posterior P ϕ ( c | s ) is likely to make poor predictions when presented with trajectories containing previously unseen states, resulting in low rewards for the policy. Thus, an exploration bonus should be introduced as an extra reward term. In [34], they choose to train ensembles of discriminators and adopt their disagreement as a measure of state uncertainty. States of higher uncertainty are assigned with higher exploration bonus. Similar to theirs, our algorithm introduces additional bonus as a reimbursement of the unnecessary pessimistic exploration. The difference lies that they adopt an implicit exploration measure (i.e., the disagreement among ensembles) while we use explicit ones based on DPP.

In [35], they optimize the MI I ( s, c ) = H ( s ) -H ( s | c ) , where H ( s ) is for improving exploration of the learned options. They adopt a variational posterior P ϕ ( s | c ) to estimate H ( s | c ) . Learning P ϕ ( s | c ) can be challenging when the state space is high-dimensional, compared with P ϕ ( c | s ) used in our paper, which can hinder the optimization. Further, in DADS, they categorize algorithms utilizing I ( s, c ) = H ( s ) -H ( s | c ) as the forward form of MI-based option discovery, and they empirically and theoretically show the limited capability of these algorithms for exploration even with H ( s ) in the objective.

C Implementation Details and Analysis of ODPP

C.1 The Choice of Diversity Measure

The expected cardinality is a better choice for the diversity measure than the likelihood shown as Eq. (1). Using the log-likelihood based on the Determinant of the DPP kernel matrix directly would heavily penalize repeated items in the sampled set W in Eq. (1). For example, if there are very similar points in W , the corresponding rows in the kernel matrix will be almost identical and lead to a zero determinant, which will cause numerical issues for the logarithm function. Take our work as an example: at the beginning of the training stage, the moving range of the Mujoco agent is very limited, then we always include very close states in a trajectory, which will always lead to a zero determinant and thus cannot provide training signals. However, the expected cardinality only counts the number of diverse states in a trajectory that will not be heavily influenced by the repeated items. Therefore, we select the expected cardinality as the diversity measure in this paper.

C.2 Fast Greedy MAP Inference for DPP

The maximum a posteriori (MAP) inference of DPP aims at finding the subset of items with the highest possibility under the DPP measure, which is NP-hard [48]. The log-probability function in

Thus, the MAP inference for DPP can be converted to a submodular maximization problem, where greedy algorithms have shown promising empirical success. Recently, the authors of [39] propose a fast greedy method for MAP inference in DPP with time complexity O ( S 2 N ) to return S items out of a sample space of size N . The key step of their algorithm is that for each iteration, the item which maximizes the marginal gain:

is added to W map starting from an initial set ∅ , until the maximal marginal gain becomes negative or the target sample number is reached (i.e., stopping criteria ). This part is not our contribution. We provide its detailed pseudo code as Algorithm 2. For the derivation, please refer to the original paper [39]. We also provide its implementation code as a part of the complete code of ODPP.

Algorithm 2 Fast Greedy MAP Inference for DPP 1: Input: The set of items W and its kernel matrix L , stopping criteria 2: Initialize: For i ∈ W , c i = [] , d 2 i = L ii ; W map = { j } , where j = arg max i ∈W log( d 2 i ) 3: while stopping criteria not satisfied do 4: for i ∈ W\ W map do 5: e i = ( L ji -< c j , c i > ) /d j 6: c i = [ c i e i ] , d 2 i = d 2 i -e 2 i 7: end for 8: j = arg max i ∈W\ W map log( d 2 i ) , W map = W map ∪ { j } 9: end while 10: Return W map

C.3 Computation of the Laplacian Spectrum for the Infinite-scale State Spaces

As mentioned in Section 3.2, the feature vector of each state - → b i is defined with the eigenvectors corresponding to the D -smallest eigenvalues of the Laplacian matrix of the state transition graph. However, for the infinite-scale state spaces, we cannot obtain this Laplacian spectrum through matrixbased methods, so we adopt the NN-based method proposed in [50] for estimating the Laplacian spectrum, which has been proved to be scalable for infinite-scale state spaces and sufficiently accurate compared with the groundtruth. Since this algorithm is not our contribution, we only provide the take-away messages here for implementation.

According to [50], the k smallest eigenvalues λ 1: k and corresponding eigenvectors v 1: k of the Laplacian L can be estimated by: ( k = D for our case)

For the large-scale state space, the eigenvectors can be represented as a neural network that takes a state s as input and outputs a k -dimension vector [ f 1 ( s ) , · · · , f k ( s )] as an estimation of [ v 1 ( s ) , · · · , v k ( s )] . Accordingly, the objective in Equation (25) can be expressed as: (please refer to [50] for details)

where T is a set of state-transitions collected by interacting with the environment through a random policy. Further, the orthonormal constraints in Equation (25) are implemented as a penalty term:

where α is the weight term and ρ is the distribution of states in T . To sum up, the eigenfunctions f can be trained as an NN by minimizing the loss function:

C.4 Analysis on Computation Complexity

The learning target of ODPP is an intra-option policy π θ ( a | s, c ) conditioned on the option choice c . As in Section 3.3, this policy is learned with an Actor-Critic algorithm for which the Q-function is defined as Eq. (13). This Q-function contains variational and DPP-based objectives. Compared with previous variational option discovery algorithms (e.g., DIAYN, VIC, V ALOR), we additionally need to (a) sample landmark states from each trajectory and (b) calculate the DPP-related terms: f ( · ) , g ( · ) , h ( · ) .

For (a), we adopt a fast greedy MAP inference algorithm for DPP. As mentioned in Appendix C.2, it takes O ( S 2 N ) to sample S landmark states from an option trajectory of length N . In our setting, N = 50 , S = 10 , so the process can be done in real-time.

For (b), we need to build DPP kernel matrices, and then compute f ( · ) , g ( · ) , h ( · ) based on eigenvalues of the corresponding kernel matrix as in Eq. (7)-(9). The time complexity for eigen decomposition is O ( N 3 ) , where N is the size of the matrix. For the state kernel matrix, N is the number of states in an option trajectory (i.e., the option horizon) which we set as 50. For the trajectory kernel matrix, N corresponds to the number of trajectories collected in each training iteration, which is set as 100. Thus, f ( · ) , g ( · ) , h ( · ) can be computed in real-time.

To build the kernel matrix, we need feature vectors for each state. As introduced in Appendix C.3, the feature vector is the output of a pre-trained neural network which takes the state as input. The training of this feature function is based on state transitions in the replay buffer and only needs to be done for once or twice in the whole option discovery process, of which the time cost is within 30 minutes.

To sum up, compared with previous variational methods, we additionally introduce three DPP items to explicitly model the option diversity and coverage, of which the involvement only slightly increases the time complexity.

C.5 Analysis on Scalability

ODPP can be adapted to more intricate setups encompassing longer option horizons, a greater number of skills, or visual domains. We elaborate on the scalability of ODPP as follows.

(a) The skill horizon is constrained by the MAP inference and eigen decomposition operations previously described. Given their time complexity, the skill horizon could readily be expanded from 50 to 100 or even 500. This augmentation would necessitate an additional time of O (10 -3 ) or O (10 -2 ) seconds per training iteration, compared with previous variational methods. These estimations are based on computations on a machine with a single Intel i7 CPU and four GeForce RTX 2060 GPUs. Note that a skill horizon longer than 100 is rarely necessary. Employing a skill with an excessively long horizon may compromise flexibility in decision-making.

(b) Compared with variational methods, our algorithm does not introduce extra limitations on the number of learned skills. Moreover, in Figure 5(c), we show that even when learning a large number of options at the same time (as much as 60), we can still get options with high quality (mean) and diversity (standard deviation) which increase during the training process.

(c) ODPP needs to learn the Laplacian feature embeddings. For visual domains, this process can incorporate a pretrained CNN model as a feature extractor, which serves to convert visual input into feature vectors. Subsequently, the original algorithm can be applied. Applications in visual domains could pose a common challenge for all option discovery algorithms and present an exciting avenue for future research.

C.6 Important Hyperparameters

First, we introduce the structure of the networks used in our algorithm and the baselines as follows. We use s _ dim , a _ dim to represent the dimension of the state space and action space respectively, and use c _ num to represent the number of options to learn at a time, which can be 10, 20, 40 or 60 in our experiments. Also, we use tanh and relu to denote the hyperbolic tangent function and rectified linear unit used as the activation functions, FC ( X,Y ) , BiLSTM ( X,Y ) to denote the fully-connected and bidirectional LSTM layer with the input size X and output size Y .

The prior network P ω is used in all the algorithms other than DCO and its structure is [ FC ( s _ dim, 64) , tanh, FC (64 , 64) , tanh, FC (64 , 64) , tanh, FC (64 , c _ num )] . The value network corresponding to P ω has the same structure as P ω , except that the output is of size 1.

The policy network π θ is used in all the algorithms, with the structure [ FC ( s _ dim + c _ num, 64) , tanh, FC (64 , 64) , tanh, FC (64 , 64) , tanh, FC (64 , a _ dim ) , tanh ] . Its corresponding value network has the same structure except that the output is of size 1 and there is not tanh at the end.

The decoder P ϕ used in ODPP and VALOR takes a sequence of states in the trajectory as input, so it uses bidirectional LSTM as part of the network, i.e., [ BiLSTM ( s _ dim, 64) , FC (2 ∗ 64 , c _ num )] . While, the decoder of VIC and DIAYN takes one state as input rather than sequential data, so it uses the fully-connected layer instead, i.e., [ FC ( s _ dim, 180) , tanh, FC (180 , 180) , tanh, FC (180 , 180) , tanh, FC (180 , c _ num )] .

The option selector P ψ has the same structure as P ω and is used in all the algorithms.

The eigenfunction network introduced in Section C.3 is used in ODPP and DCO to estimate the Laplacian spectrum. Its structure is [ FC ( s _ dim, 256) , relu, FC (256 , 256) , relu, FC (256 , 256) , relu, FC (256 , 256) , relu, F C (256 , 30)] , where 30 denotes the dimension of the feature vector - → b i mentioned in Section 3.2.

As noted in the paper, the crucial hyperparameters are β, α 1:3 in Eq. (6) and (10) which control the importance of each objective term, relating to diversity and coverage. Conducting a grid search on the set of parameters can be exhaustive. Therefore, we follow the process of the ablation study shown in Figure 2, add objective terms and adjust their corresponding weights one by one. In particular, in Figure 2(e), we retain only the L IB objective and select its weight β = 10 -3 from five possible choices: 1 , 10 -1 , 10 -2 , 10 -3 , 10 -4 , guided by the visualization results. Next, for Figure 2(f), we introduce L DPP 1 and fine-tune the corresponding weight α 1 while keeping β fixed at 10 -3 . Last, we incorporate L DPP 2 and L DPP 3 and adjust α 2 and α 3 accordingly, while keeping β and α 1 fixed. Note that the final two terms must work in tandem to ensure that the discovered options exhibit diversity across different options and consistency for a specific option choice.

After fine-tuning, we set β = 10 -3 , α 1 = 10 -4 , α 2 = 10 -2 , α 3 = 10 -2 . It is worthy noting that our evaluations across various challenging RL tasks utilize the same hyperparameter set, highlighting the robustness of our algorithm. This is because our proposed (DPP-based) coverage and diversity measures are task-agnostic and universally applicable to RL tasks.

D Additional Evaluation Results

D.1 Complementary Results on the Effect of the Prior Network

As discussed in Section 3, we learn a prior network P ω concurrently with the option policy network π θ . Figure 7 demonstrates how initializing the option selector P ψ with P ω can lead to further performance improvement in the downstream task, using the Point Corridor goal-achieving task as an example. This is based on the fact that both networks share the same structure.

First, in (a), we sample 10,000 trajectories for each option and visualize the agent orientation distribution corresponding to different options. In (b), we present the coordinate system setup along with the four turning points for evaluation. The option choices at these turning points provided by the prior network are displayed in (c). It can be observed that P ω favors the most significant options at a given state. For instance, at Location #1, Option #5, which tends to go left or down as shown in (a), is preferred, while at Location #3, Option #3 is favored, guiding the agent to go up or right. Lastly, in (d), we demonstrate that initializing the option selector with the prior network can further enhance the agent's performance in the downstream goal-achieving task.

Previous variational methods choose to fix the prior distribution to avoid a collapse for the prior network to sample only a handful of skills. Our algorithm pretends that because our algorithm additionally introduces three DPP-based terms to explicitly maximize the coverage and diversity of the learned options. With L DPP 1:3 , each option is expected to cover multiple state clusters with a long range and different options tend to visit different regions in the state space. In this case, the prior network would tend to select multiple diverse skills to improve its learning objective (i.e., Eq.

Option#0

Option#1

Option#2

Option#3

Option#4

0.010

Density

0.005

0.000

-180-90

0

90180

-180-90

0

90

180

-180-90

0

90

180

-180-90

0

06

180

-180-90

0

90

180

Option#5

Option#6

Option#7

Option #8

Option#9

0.010

Density

0.005

0.000

-180-90

0

90

180

-180-90

0

06

180

-180-90

0

06

180

-180-90

0

06

180

-180-90

0

180

Orientation

Location #1

Location#2

0.20

#1

C

(d)

algorithm

0.50

ODPP

0.15

ODPP_prior

0.25

#4

90%

0.00

123456789

0

123456789

00

Location #3

Location #4

#2

0.05

0.50

#3

0.25

0.00

0.00

234

56789

0123456789

0

2000

4000

6000

8000

10000

OptionChoice

Episode

(12)) rather than sampling only few of them. The collapse happens because the mutual information objective only implicitly measures option diversity as the difficulty of distinguishing them via the variational decoder and does not model the coverage, as noted in Section 3. This motivates us to introduce explicit diversity and coverage measures as regularizers and enhancement.

D.2 More Visualization of the Learned Ant Locomotion Behaviors

In Figure 8, we show more visualization results of the learned Ant locomotion behaviors without the supervision of task-specific rewards.

(a) Flip

(b) Rotation

3

2

y

3

K

2

3

2

2

3

2

3

3

3

3

十

3

3

3

4

2

2

1

1

4

1

1

2

1

2

1

2

1

4

1

2

4

1

D.3 Quantitative Ablation Study Results

D.4 Baseline Performance on OpenAI Gym and Atari Games

(a) CartPole

(b) RiverRaid

(c) AirRaid

100

100

80

80

Reward

60

Reward

60

algorithm

VALOR

VALOR

MIC

VIC

40

DIAYN

40

DIAYN

DCO

DCO

DADS

20

DADS

20

APS

APS

0

2000

4000

6000

8000

10000

2000

4000

6000

8000

10000

Step

Step

60

50

mummm

Reward

40

algorithm

VALOR

30

VIC

20

DIAYN

DCO-

10

DADS

APS

0

Q

2000

4000

6000

8000

10000

Step

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on unsupervised option discovery. It uses the framework of Determinantal Point Processes (DPP) with the aim of combining the advantages of variational and Laplacian-based methods, and unify the desiderata of coverage and diversity of the learned options. Empirical validation shows the benefits of the approach over baselines in continuous control tasks.\n\nSTRENGTHS:\n- The idea of adopting DPP for option discovery is novel and quite interesting. Ensuring coverage and diversity has been a challenging open question in the unsupervised RL community and the use of DDP is a mathematically clever way to tackle it. The fact that the proposed approach seeks to capture advantages of both variational and Laplacian-based methods, usually part of distinct streams of research, also has a unifying value.\n- The paper is quite clear, well motivated and well written.\n- Experiments show improvements over baselines like DIAYN.\n\nWEAKNESSES:\n- Some related work is missing, including more recent and improved variational methods, such as [1-6]. It would in particular be valuable to discuss the connections with option-discovery approaches like [6-8] that build on the idea of Maximum Entropy Exploration and explicitly seek to optimize coverage, since the latter is a key feature targeted in the paper; see also my first question below. \n- The proposed method is quite complicated, with somewhat ad hoc design. While I appreciate the effort to isolate each desiderata (coverage/diversity, intra/inter-option), the final objective function in Eq.(9) is rather involved. In particular, the choice of hyperparameters to trade-off each term looks non-trivial, as acknowledged in the last section of the paper. The method seems quite computationally involved compared to standard variational approaches, and it is unclear how it would scale to more complex setups (e.g, longer horizon, more skills, visual domains).\n- The empirical part of the paper could be improved. There have been numerous option-discovery works that clearly outperform VIC/DIAYN and it would be relevant to compare to such stronger baselines. For example: DISDAIN [5], EDL [7] (with SMM [6] instead of the oracle version, despite the fact that it is a 'trajectory-first' method as categorized in the paper), DADS (despite the author's argument of distinction between model-free / model-based). As a small side note, it's also nice to add the random policy as a 'free' baseline (as it sometimes outperforms VIC/DIAYN in coverage objectives).\n- Very minor typos: 'extracts' (l.51), 'generalizes the\" (l.202), 'long-horizon' (l.251)).\n\n[1] Fast task inference with variational intrinsic successor features, Hansen et al., ICLR 2020\n\n[2] Relative variational intrinsic control, Baumli et al., AAAI 2021\n\n[3] Entropic desired dynamics for intrinsic control, Hansen et al., NeurIPS 2021\n\n[4] Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching, Kamienny et al., ICLR 2022\n\n[5] Learning more skills through optimistic exploration, Strouse et al., ICLR 2022\n\n[6] Active Pretraining with Successor Features, Liu and Abbeel, ICML 2021\n\n[7] Efficient Exploration via State Marginal Matching, Lee et al., arXiv 2019\n\n[8] Explore, discover and learn: Unsupervised discovery of state-covering skills, Campos et al., ICML 2020\n\nQUESTIONS:\n- Could the authors provide a rigorous/mathematical definition of what they mean by 'coverage' throughout the paper. Is it in terms of entropy of states visited by the options H(S)? If so, then it's a term explicitly optimized by diversity I(S,Z) = H(S) - H(S|Z), which would make confusing the statement that \"variational option discovery maximizes the diversity of the options through a mutual information loss (while ignoring coverage)\" (or maybe the authors mean that optimizing MI exactly is difficult and the approximations performed in the literature tend to have poor coverage, which is different from arguing that the targeted objective ignores coverage in the first place). Also, using a Laplacian-based idea that reduces the cover time of the graph induced by the random policy may be a poor proxy of state space coverage in the entropic sense, while some diversity-based works like [6-8] explicitly target this component with good empirical results. As for the coverage visualizations, Figure 1) b) is said to improve coverage but the options are all going to the same corridor, which does not look desirable in terms of state space coverage. Meanwhile on Figure 3, the paper implies that options learned by (c) are more \"diverse\" and \"beneficial for various downstream tasks\" than (a) or (b), although one may argue otherwise, as the latter show less clutter around the initial state and attain more distant regions of the state space. Thanks for the clarifications.\n- Isn't the evaluation on Appendix D.3 on OpenAI Gym rather than Atari? (Is Cartpole an Atari game?)\n- Although there is a computational complexity discussion in Appendix C.4, could you elaborate more on the scalability of the approach and for example provide numbers on its computational requirements compared to VIC/DIAYN given the same number of environment interactions?\n- As first observed in the DIAYN paper, fixing the prior distribution (over option choice at the initial state) rather than learning it prevents a collapse to sampling only a handful of skills (as it can occur for VIC), and this has become standard practice in most subsequent works (although it is a loose lower bound on the original MI objective that should also optimize for the prior distribution). Here you explicitly learn the prior distribution, which is an interesting but undiscussed choice, could you explain more if, and why, you can overcome the aforementioned collapse?\n\nLIMITATIONS:\nA limitation is discussed in Section 5. Not applicable regarding potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for unsupervised option discovery by utilizing Determinantal Point Process (DPP) to quantify and optimize both the diversity and the coverage of the learned options. The proposed unified option discovery framework captures the advantages of both variational and Laplacian-based methods, which is the major tools for existing unsupervised option discovery approaches. The experiment results in both MuJoCo and Atari demonstrate the superiority of the proposed algorithm.\n\nSTRENGTHS:\n1. The motivation to propose the unified algorithm framework is convincing and the paper is well-written and easy to follow. The author well illustrates the main idea of this paper using a toy example. \n2. The proposed option discovery framework unifies both variational and Laplacian-based methods and enables explicit maximization of diversity and coverage of the options.\n3. Though DPP is widely used in methods to promote diversity, it is the first work to adopt it into option diversity.\n4. Both the theoretical and experimental results show the superiority of the algorithms.\n\nWEAKNESSES:\n1. The major concern to me is the application of the proposed framework. In Equ 9. there are three hyper-parameters in the loss function. How to tune these hyper-parameters?  In L586 you say fine-tune important hyperparameters using a sequential, greedy method based on options’ visualization, such as in Fig2. Could you provide more details? \n2. DPP is widely used in the work to promote diversity. It would be better to discuss them in related work, e.g., in promoting diverse policies in population-based RL[1, 2], diverse policies in games [3], recommendation diversity [4], etc. \n3. Some theoretical results can be highlighted (such as using a Proposition station) in the main text. \n4. In Sec4, instead of some visualization results, are there some diversity metrics that could be proposed to measure the diversity? It would be better to see these numerical results in the main text. \n\n[1] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050–18062, 2020.\n\n[2] Wu S, Yao J, Fu H, et al. Quality-Similar Diversity via Population Based Reinforcement Learning. In The Eleventh International Conference on Learning Representations, (2023).\n\n[3] Perez-Nieves, Nicolas, et al. \"Modelling behavioural diversity for learning in open-ended games.\" International conference on machine learning. PMLR, 2021.\n\n[4]. Chen, L., G. Zhang, E. Zhou. Fast greedy MAP inference for determinantal point process to improve recommendation diversity. In Advances in Neural Information Processing Systems 31, NeurIPS 2018, pages 5627–5638. 2018.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an unsupervised option discovery framework based on a combination of Determinantal Point Process and Laplacian spectral features. The main idea is to combine variational methods and  Laplacian methods in order to control for coverage and diversity of the options. The proposed method has been validated experimentally on Mujoco 3D navigation environments where it shows superior performance to other approaches. Further experiments have been done on Atari environments as well.\n\nSTRENGTHS:\nThe proposed combination of DPP and Laplacian features is reasonable and shows promising results. The novelty stems mainly from the modified DPP kernel similarity matrix which makes use of the Laplacian spectrum. I also think that maximizing the cardinality of the the landmark set as a diversity signal is a nice idea.\n\nAll components that were introduced in the method are well argued. Especially the overarching motivation of maximizing coverage and diversity in the same time (reaping the benefits of variational and Laplacian methods)\n\nExperiments are convincing.\n\nWEAKNESSES:\nAt some points the paper lacks clarity, for example in the exposition of the DPP, terms were introduced that were not properly explained and that I would not consider common knowledge.\n\nSome probabilistic framing that was used in the paper was off, but we can clarify this in the rebuttal phase hopefully.\n\nThe Atari experiments should appear in the main paper in some form, since they are already announced in the abstract.\n\nAs the authors have already noted in the paper, the method needs balancing of 3 terms in the objective which might make it impractical, however the hyperparameters are stable across different environments (shows robustness).\n\nNo significant theoretical insights, rather a combination of existing works in order to devise and option discovery algorithm.\n\nThe writing in general could be improved, I found it a bit hard to keep track of all the introduced terms and the connection to the main idea of the paper.\n\nQUESTIONS:\nline 112: the explanation of this Gram matrix seems lacking, for instance the quality measure is just introduced in the text here and used once more later, without any explanation how it related to reward and the DPP probability of a subset.\n\neq.3: it is not clear to me from this equation that you are doing MAP inference,  can you specify in clear terms what is the posterior here and what is the prior (also in the paper)? \n\neq. 4: what is being maximized over here? And a follow-up, in which sense is beta a Lagrange multiplier? What is the constrained optimization problem that is being solved here, can you actually write it this way? Please clarify this in your response and ideally in the main text.\n\nline 219: the notation here is confusing, is this to define a conditional random variable given s_0 and c? (the trajectory). Wouldn’t it be better to just define this as sampled from a conditional distribution, then take the expectation in eq.7 over it. (might be less confusing)\n\neq.8: same comment as for line 219. \n\neq.9: perhaps you should consider putting the minus sign into the second loss, so that it remains a proper loss (something that you want to minimize) and adjust the explanation.\n\nfigure 3 caption - “in the normal setting”\n\nthe legends in figure 4 are not well visible and the fonts are off. I suggest you place the legends outside of the figure.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses reward-free options discovery for RL. First, it notes that prior work would prioritize either state coverage or diversity into the options discovery procedure. Hence, it proposes a new loss function that fosters coverage and diversity simultaneously by exploiting DPPs on both the trajectories generated by the options and the states within a trajectory. Finally, it presents an algorithm to optimize this objective, which is evaluated in continuous control domains against standard baselines, such as VIC, VALOR, DIAYN, and Laplacian methods.\n\nSTRENGTHS:\n- Exploiting tools from DPPs for the options discovery is interesting and, to the best of my knowledge, novel and original;\n- The experimental results are at least promising;\n- The paper includes a neat presentation of the previous works and approaches for unsupervised options discovery.\n\nWEAKNESSES:\n- The learning objective and the corresponding algorithm are quite convoluted. They requires several layers of approximation to make the learning tractable, as well as a handful of hyper-parameters to be tuned and the definition of a suitable kernel over trajectories and states;\n- The experimental evaluation is not thorough. Most of all, the ablation study is limited to a single run in a single domain, and leaves the reader wondering whether all of the introduced ingredients would be needed in general;\n- The intuition behind the loss function is not always pristine. It is easy to lose focus going through Section 3 of the paper;\n- The visualization of the experimental results could be further polished.\n\nQUESTIONS:\nThe paper is an interesting contribution to a problem that is somehow far from being solved despite receiving a considerable attention recently. While the main contribution of this paper is algorithmic, it is hard to assess its value without a stronger empirical study (or more theoretical corroboration). Thus, I am currently providing a slightly negative score, even if a case for acceptance could be made and I am open to change my score after the authors' response. As is, the paper looks like a nice engineering feat to me, but lacks some convincing intuition and experiments on why all those tricks would work in general.\n\n**Algorithm**\n\n- There is one aspect that is not completely clear to me on how the loss is presented. The authors motivates the work as going beyond previous variational or Laplacian approaches with a new framework that comprises both, but then the main ingredient of the loss, i.e., $\\mathcal{L}_{IB}$, looks a standard variational loss with some variation (landmark states). Moreover, the components $\\mathcal{L}_1, \\mathcal{L}_2, \\mathcal{L}_3$ already provide incentives for coverage and diversity, so why one should also add the information bottleneck on top of them? Overall, I think that presenting this work as an evolution of variational option discovery would help its presentation.\n- From my understanding, to define the relevant DPPs a proper kernel on trajectories and states is needed. Can the authors discuss how this would be designed in the absence of any domain knowledge?\n- I am wondering whether the presented loss function induces a (hidden) RL problem or not, i.e., whether the loss function could be incorporated into a standard reward function in hindsight. Is the approach producing a set of deterministic options or the discovered options may be stochastic?\n\n**Experiments**\n\n- The presented approach is fairly complicated and I think the ablation study in Section 4.1 is not enough to state that all of the introduced ingredients would benefit the option discovery in general. It is somehow underwhelming that the paper only presents one result, a single domain and a single seed, through an illustration instead of some quantitative analysis and learning curves. I believe this does not meet the bar to motivate the added complexity of the loss function, especially on top of the $\\mathcal{L}_{IB}$ that seems to lead to a good options discovery already.\n- All of the approach is predicated on the need for coverage and diversity simultaneously, but it is hard to evaluate them beyond qualitative assessments. However, I think the paper could put more effort in designing quantitative measures to evaluate diversity and coverage.\n- I would suggest to report the learning curves with average and confidence intervals instead of one standard error, which is hardly meaningful.\n\n**Minor**\n\n- I would suggest to report the pseudocode of Algorithm 1 in the main text.\n- The overall notation, in particular the one needed to define the loss function, is sometimes convoluted.\n- Where is the DIAYN learning curve in Fig. 4b, 4c?\n- Why are the Atari results relegated to the appendix? CartPole is not an Atari game, but a continuous control task.\n- In the downstream task setting only the option selecting policy is learned? Do the authors considered also fine-tuning of the learned options?\n\nLIMITATIONS:\nThe paper explicitly addresses the limitations of the presented approach in the final paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}

comments: SUMMARY:
This paper proposes and evaluates an approach to option discovery in reinforcement learning. The aim is to autonomously identify a set of options that are diverse and that give good coverage of the state space. This aim is achieved by using the Determinantal Point Process (DPP). The proposed approach is evaluated in a range of domains.

STRENGTHS:
The proposed approach is intuitive and principled. 

While the underlying notions of diversity and coverage have been used earlier in the literature for option discovery, their combined optimization through the Determinantal Point Process is novel and returns better results than related existing methods.

The experimental evaluation is extensive and varied, showing not only learning curves but also option trajectories.

WEAKNESSES:
Only minor comments: 

An analysis of computational complexity is provided in the supplementary material. It would be useful to see a short summary of that in the main paper. 

In the learning curves, it would be useful to plot ceiling performance. 

Text in Figure 1d is too small.

QUESTIONS:
No questions.

LIMITATIONS:
Adequate.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-03