Title: Effective Exploration Based on the Structural  Information Principles
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Reinforcement Learning

Structural Information

Representation Learning

Entropy Maximization Exploration
Abstract: Traditional information theory provides a valuable foundation for Reinforcement Learning (RL), particularly through representation learning and entropy maximiza tion for agent exploration. However, existing methods primarily concentrate on modeling the uncertainty associated with RL’s random variables, neglecting the in herent structure within the state and action spaces. In this paper, we propose a novel Structural Information principles-based Effective Exploration framework, namely SI2E. Structural mutual information between two variables is defined to address the single-variable limitation in structural information, and an innovative embedding principle is presented to capture dynamics-relevant state-action representations. The SI2E analyzes value differences in the agent’s policy between state-action pairs and minimizes structural entropy to derive the hierarchical state-action struc ture, referred to as the encoding tree. Under this tree structure, value-conditional structural entropy is defined and maximized to design an intrinsic reward mechanism that avoids redundant transitions and promotes enhanced coverage in the state-action space. Theoretical connections are established between SI2E and classical information-theoretic methodologies, highlighting our framework’s rationality and advantage. Comprehensive evaluations in the MiniGrid, MetaWorld, and DeepMind Control Suite benchmarks demonstrate that SI2E significantly outperforms state-of-the-art exploration baselines regarding final performance and sample efficiency, with maximum improvements of 37.63% and 60.25%, respectively.

Effective Exploration Based on the Structural Information Principles

Abstract

Traditional information theory provides a valuable foundation for Reinforcement Learning (RL), particularly through representation learning and entropy maximization for agent exploration. However, existing methods primarily concentrate on modeling the uncertainty associated with RL's random variables, neglecting the inherent structure within the state and action spaces. In this paper, we propose a novel S tructural I nformation principles-based E ffective E xploration framework, namely SI2E . Structural mutual information between two variables is defined to address the single-variable limitation in structural information, and an innovative embedding principle is presented to capture dynamics-relevant state-action representations. The SI2E analyzes value differences in the agent's policy between state-action pairs and minimizes structural entropy to derive the hierarchical state-action structure, referred to as the encoding tree. Under this tree structure, value-conditional structural entropy is defined and maximized to design an intrinsic reward mechanism that avoids redundant transitions and promotes enhanced coverage in the state-action space. Theoretical connections are established between SI2E and classical information-theoretic methodologies, highlighting our framework's rationality and advantage. Comprehensive evaluations in the MiniGrid, MetaWorld, and DeepMind Control Suite benchmarks demonstrate that SI2E significantly outperforms state-of-the-art exploration baselines regarding final performance and sample efficiency, with maximum improvements of 37 . 63% and 60 . 25% , respectively.

1 Introduction

Reinforcement Learning (RL) has emerged as a pivotal technique for addressing sequential decisionmaking problems, including game intelligence [Vinyals et al., 2019, Badia et al., 2020], robotic control [Andrychowicz et al., 2017, Liu and Abbeel, 2021], and autonomous driving [Prathiba et al., 2021, Pérez-Gil et al., 2022]. In the realm of RL, striking a balance between exploration and exploitation is crucial for optimizing agent policies and mitigating the risk of suboptimal outcomes, especially in scenarios characterized by high dimensions and sparse rewards [Zhang et al., 2021b].

Recently, advancements in information-theoretic approaches have shown promise for exploration in self-supervised settings. The maximum entropy framework over the action space [Haarnoja et al., 2017] has led to the development of robust algorithms such as Soft Q-learning [Nachum et al., 2017], SAC [Haarnoja et al., 2018], and MPO [Abdolmaleki et al., 2018]. Additionally, various objectives focused on maximizing state entropy are utilized to ensure comprehensive state coverage [Hazan et al., 2019, Islam et al., 2019]. To facilitate the exploration of complex state-action pairs, MaxRenyi optimizes Rényi entropy across the state-action space [Zhang et al., 2021a]. However, a prevalent issue with entropy maximization strategies is their tendency to bias exploration towards low-value states, making them vulnerable to imbalanced state-value distributions in supervised settings. To mitigate

𝑠𝑠0

𝑠𝑠1

𝑠𝑠2

′

𝑠𝑠3

six-state MDP with four actions

five-state MDP with two actions

𝑠𝑠0

𝑠𝑠1

𝑠𝑠2

𝑠𝑠3

𝑠𝑠5

maximize state-action entropy

maximize structural entropy

𝑎𝑎0

𝑎𝑎1

𝑎𝑎0

,

𝑎𝑎1

/

𝑎𝑎0

′

𝑎𝑎2

𝑎𝑎3

𝑎𝑎0

,

𝑎𝑎1

𝑎𝑎0

′

𝑠𝑠2

,

𝑠𝑠5

𝑠𝑠2

′

this, value-conditional state entropy is introduced to compute intrinsic rewards based on the estimated values of visited states [Kim et al., 2023]. Due to their instability in noisy and high-dimensional environments, a Dynamic Bottleneck (DB) [Bai et al., 2021] is developed based on the Information Bottleneck (IB) principle [Tishby et al., 2000], thereby obtaining dynamics-relevant representations of state-action pairs. Despite their successes, existing information-theoretic exploration methods have a critical limitation: they often overlook the inherent structure within state and action spaces. This oversight necessitates new approaches to enhance exploration effectiveness.

Figure 1 illustrates a simple six-state Markov Decision Process (MDP) with four actions. The different densities of the blue and red lines represent different actions, as indicated in the legend, leading to state transitions aimed at optimizing the return to the initial state s 0 . Solid lines specifically denote actions a 0 and a 1 . The transitions between states s 2 and s 5 are deemed redundant as they do not facilitate the primary objective of efficiently returning to s 0 . Therefore, the state-action pairs ( s 2 , a 3 ) and ( s 5 , a 3 ) have lower policy values. A policy maximizing state-action Shannon entropy would encompass all possible transitions (blue color). In contrast, a policy incorporating the inherent state-action structure will divide these redundant state-action pairs into a vertex sub-community and minimize the entropy of this sub-community to avoid visiting it unnecessarily. Simultaneously, it maximizes state-action entropy, resulting in maximal coverage for transitions (red color) that are more likely to contribute to the desired outcome in the simplified five-state MDP.

Departing from traditional information theory applied to random variables, structural information [Li and Pan, 2016] has been devised to quantify dynamic uncertainty within complex graphs under a hierarchical partitioning structure known as an 'encoding tree". Structural entropy is conceptualized as the minimum number of bits required to encode a vertex accessible through a single-step random walk and is minimized to optimize the encoding tree. However, this definition is limited to singlevariable graphs and cannot capture the structural relationship between two variables. While the underlying graph can encompass multiple variables, current structural information principles are limited in treating these variables as a single joint variable, measuring only its structural entropy. This limitation prevents the effective quantification of structural similarity between variables, inherently imposing a single-variable constraint. Prior research on reinforcement learning using structural information principles [Zeng et al., 2023b,c] has focused on independently modeling state or action variables without simultaneously considering state-action representations.

In this work, we propose SI2E, a novel and unified framework grounded in structural information principles for effective exploration within high-dimensional and sparse-reward environments. Initially, we embed state-action pairs into a low-dimensional space and present an innovative representation learning principle to capture dynamics-relevant information and compress dynamics-irrelevant information. Then, we strategically increase the state-action pairs' structural mutual information with subsequent states while decreasing it with current states. We analyze value differences among state-action representations to form a complete graph and minimize its structural entropy to derive the

optimal encoding tree, thereby unveiling the hierarchical community structure of state-action pairs. By leveraging this identified structure, we design an intrinsic reward mechanism tailored to avoid redundant transitions and enhance maximal coverage in exploring the state-action space. Furthermore, we establish theoretical connections between our framework and classical information-theoretic methodologies, highlighting the rationality and advantage of SI2E. Our thorough evaluations across diverse and challenging tasks in the MiniGrid, MetaWorld, and DeepMind Control Suite benchmarks have consistently shown SI2E's superiority, with significant improvements in final performance and sample efficiency, surpassing state-of-the-art exploration baselines. For further research, the source code is available at 1 . Our contributions are summarized as follows:

A novel framework based on structural information principles, SI2E, is proposed for effective exploration in high-dimensional RL environments with sparse rewards.

An innovative principle of structural mutual information is introduced to overcome the singlevariable constraint inherent in existing structural information and to enhance the acquisition of dynamics-relevant representations for state-action pairs.

A unique intrinsic reward mechanism that maximizes the value-conditional structural entropy is designed to avoid redundant transitions and promote enhanced coverage in the state-action space.

Our experiments on various challenging tasks demonstrate that SI2E significantly improves final performance and sample efficiency by up to 37 . 63% and 60 . 25% , respectively, compared to state-of-the-art baselines.

2 Preliminaries

In this section, we formalize the definitions of fundamental concepts. The descriptions of primary notations are summarized in Appendix A.1 for ease of reference.

2.1 Traditional Information Principles

Consider the random variable pair Z = ( X,Y ) with a joint distribution probability denoted by p ( x, y ) ∈ (0 , 1) . The marginal probabilities, p ( x ) and p ( y ) , are defined as p ( x ) = ∑ y p ( x, y ) and p ( y ) = ∑ x p ( x, y ) , respectively. The joint Shannon entropy [Shannon, 1953] of X and Y is H ( X,Y ) = -∑ ( x,y ) [ p ( x, y ) · log p ( x, y )] , which quantifies the total uncertainty in Z . Conversely, the marginal entropies H ( X ) = -∑ x [ p ( x ) · log p ( x )] and H ( Y ) = -∑ y [ p ( y ) · log p ( y )] characterize the uncertainty in X and Y individually. The mutual information I ( X ; Y ) = ∑ x,y [ p ( x, y ) · log p ( x,y ) p ( x ) p ( y ) ] quantifies the shared uncertainty between X and Y . It satisfies the following relationship: I ( X ; Y ) = H ( X ) + H ( Y ) -H ( X,Y ) .

2.2 Reinforcement Learning

Within the context of RL, the sequential decision-making problem is formalized as a Markov Decision Process (MDP) [Bellman, 1957]. The MDP is characterized by a tuple ( O , A , P , R e , γ ) , where O denotes the observation space, A the action space, P the environmental transition function, R e the extrinsic reward function, and γ ∈ [0 , 1) the discount factor. At each discrete timestep t , the agent selects an action a t ∈ A upon observing o t ∈ O . This leads to a transition to a new observation o t +1 ∼ P ( o t , a t ) and a reward r e t ∈ R . The policy network π is optimized to maximize the cumulative long-term expected discounted reward.

Maximum State Entropy Exploration. In environments with sparse rewards, agents are encouraged to explore the state space extensively, which can be incentivized by maximizing the Shannon entropy H ( S ) of state variable S . When the prior distribution p ( s ) is not available, the non-parametric k -nearest neighbors ( k -NN) entropy estimator [Singh et al., 2003] is employed. For a given set of n independent and identically distributed samples from a d x -dimensional space { x i } n -1 i =0 , the entropy of variable X is estimated as follows:

where d ( x i ) is twice the distance from x i to its k -th nearest neighbor, and C is a constant term.

Information Bottleneck Principle. In the supervised learning paradigm, representation learning aims to transform an input source X into a representation Z , targeted towards an output source Y . The Information Bottleneck (IB) principle [Tishby et al., 2000] refines this process by maximizing the mutual information I ( Z ; Y ) between Z and Y , capturing the relevant features of Y within Z . Concurrently, the IB principle imposes a complexity constraint by minimizing the mutual information I ( Z ; X ) between Z and X , effectively discarding irrelevant features. To balance these objectives, the IB principle utilizes a Lagrangian multiplier, facilitating a balanced trade-off between the richness of the representation and its complexity.

2.3 Structural Information Principles

The encoding tree T of an undirected and weighted graph G = ( V, E ) is characterized as a rooted tree with the following properties: 1) Each tree node α in T corresponds to a subset of graph vertices V α ⊆ V . 2) The subset V λ of tree root λ encompasses all vertices in V . 3) Each subset V ν of a leaf node ν in T only contains a single vertex v , thus V ν = { v } . 4) For each non-leaf node α , the number of its children is assumed as l α , with the i -th child specified as α i . The collection of subsets V α 1 , . . . , V α l α constitutes a sub-partition of V α .

Given an encoding tree T whose height is at most K , the K -dimensional structural entropy of graph G is defined as follows:

̸

where g α is the weighted sum of all edges connecting vertices within the subset V α to vertices outside the subset V α .

3 Structural Mutual Information

In this section, we address the single-variable constraint prevalent in existing structural information principles and introduce the concept of structural mutual information for subsequent state-action representation learning within our SI2E framework.

Given the random variable pair ( X,Y ) with | X | = | Y | = n , we construct an undirected bipartite graph G xy to represent the joint distribution of X and Y . In G xy , each vertex x ∈ X connects to each vertex y ∈ Y via weighted edges, where the weight of each edge equals the joint probability p ( x, y ) . Notably, no edges connect vertices within the same set, X or Y , and the total sum of the edge weights is 1 , ∑ x,y p ( x, y ) = 1 . Each single-step random walk in G xy accesses either a vertex from X or Y . The structural entropy of variable X in G xy is defined as the number of bits required to encode all accessible vertices in the set X . It is calculated using the following formula:

where the sum of all vertex degrees is twice the total sum of edge weights, resulting in vol( G xy ) = 2 .

The structural entropy H SI ( Y ) is defined similarly. We restrict the partitioning structure of G xy to 2 -layer approximate binary trees, denoted as T 2 , to calculate the required bits to encode accessible vertices in X or Y , defined as the joint structural entropy. This tree structure mandates that each intermediate node (neither root nor leaf) has precisely two children. We begin by initializing a one-layer encoding tree, T 0 xy , designating each non-root node α 's parent as the root λ , with α -= λ . By applying the stretch operator from the HCSE algorithm [Pan et al., 2021], we pursue an iterative and greedy optimization of T 0 xy , further detailed in Appendix A.2. The optimal encoding tree, T ∗ xy , for G xy and the joint entropy under T ∗ xy are achieved through:

Utilizing 2 -layer approximate binary trees as the structural framework ensures computational tractability and more complex structures will increase the cost of increased computational complexity, which can be prohibitive for practical applications.

c) Encoding Trees

𝒯𝒯

2

d) Mutual Information

a) Embedding

Ⅰ.

State-action

Representation

𝐼𝐼

𝑆𝑆𝑆𝑆

(

𝑍𝑍

𝑡𝑡

;

𝑆𝑆

𝑡𝑡+1

)

min

𝐼𝐼

𝑆𝑆𝑆𝑆

(

𝑍𝑍

𝑡𝑡

;

𝑆𝑆

𝑡𝑡

)

𝑠𝑠

𝑡𝑡

𝑠𝑠

𝑡𝑡+1

𝑧𝑧

𝑡𝑡

a) State-action Space

Ⅱ.

Efficient Exploration

Value

Value Norm

||

𝜋𝜋

(

𝑣𝑣

𝑖𝑖

)

-𝜋𝜋

(

𝑣𝑣

𝑗𝑗

)||

2

𝑑𝑑

𝑖𝑖

=

𝑝𝑝

(

𝑠𝑠

𝑖𝑖

,

𝑎𝑎

𝑖𝑖

)

𝐻𝐻

𝑇𝑇

𝑠𝑠𝑠𝑠

∗

(

𝐺𝐺

𝑠𝑠𝑠𝑠

′

)

𝐻𝐻

𝑇𝑇

𝑠𝑠𝑠𝑠

∗

𝐺𝐺

𝑠𝑠𝑠𝑠

′

~

𝐻𝐻

(

𝑆𝑆

,

𝐴𝐴

)

�

𝐻𝐻

𝐾𝐾𝐾𝐾

𝑉𝑉 0

- �

𝐻𝐻

𝐾𝐾𝐾𝐾

𝑉𝑉 1

𝑟𝑟

𝑡𝑡

=

𝑟𝑟

𝑡𝑡

+

𝛽𝛽 ⋅ 𝑟𝑟

𝑡𝑡

𝑖𝑖

c) Optimal Encoding Tree

𝑇𝑇

𝑠𝑠𝑠𝑠

∗

b) State-action Graph

𝐺𝐺

𝑠𝑠𝑠𝑠

𝐺𝐺

𝑠𝑠𝑠𝑠

′

e) Value-conditional Entropy

f) Intrinsic Reward

HCSE

k-NN Estimator

structural

entropy

sub-communities

vertices

construction

𝐻𝐻

𝑆𝑆𝑆𝑆

𝑋𝑋

+

𝐻𝐻

𝑆𝑆𝑆𝑆

(

𝑌𝑌

)

𝐻𝐻

𝑇𝑇

𝑥𝑥𝑥𝑥

𝑙𝑙

(

𝑋𝑋

,

𝑌𝑌

)

We derive the following proposition regarding T 2 , with the detailed proof provided in Appendix B.1.

Proposition 3.1. Consider an undirected graph G = ( V, E ) with vertices v i and v j in V . If the edge ( v i , v j ) is absent from E , then in the 2 -layer approximate binary optimal encoding tree T ∗ ∈ T 2 , there does not exist any non-root node α such that both v i and v j are included in its subset V α .

Each intermediate node α ∈ T ∗ xy corresponds to a subset comprising exactly one x vertex and one y vertex, thus establishing a one-to-one matching structure between variables X and Y . The i -th intermediate node in T ∗ xy , ordered from left to right, is denoted as α i . Within this subset, the x and y vertices are labeled as x i and y i , respectively.

To define structural mutual information accurately, it is essential to consider the joint entropy of two variables under various partition structures. We introduce an l -transformation applied to T ∗ xy to systematically traverse all potential one-to-one matching of these variables, providing a comprehensive measure of their structural similarity. Given an integer parameter l > 0 , this transformation generates a new 2 -layer approximate binary tree, T l xy , representing an alternative one-to-one matching structure.

Definition 3.2. For each intermediate node α i in T l xy , the x and y vertices in T l α i are specified as T l α i = { x i ′ , y i } , where i ′ = ( i + l ) mod n .

The resulting tree T l xy is equivalent to the optimal tree T ∗ xy when l = 0 . We provide an example in Appendix A.3 with n = 4 for an intuitive understanding of the described process.

Definition 3.3. Leveraging the relationship I ( X ; Y ) = H ( X ) + H ( Y ) -H ( X,Y ) in traditional information theory, we formally define the structural mutual information, I SI ( X ; Y ) , as follows:

The detailed derivation is provided in Appendix C.1. Structural mutual information quantifies the average difference between the required encoding bits of accessible vertices of a single variable and the joint variable. The following theorem outlines the connection between I SI ( X ; Y ) and I ( X ; Y ) , with a detailed proof in Appendix B.2.

Theorem 3.4. For a tuning parameter 0 ≤ ϵ ≤ 1 , its holds for the structural mutual information I SI ( X ; Y ) , traditional mutual information I ( X ; Y ) , and joint Shannon entropy H ( X,Y ) that:

4 The Proposed SI2E Framework

In this section, we describe the detailed designs of the proposed SI2E framework, which captures dynamic-relevant state-action representations through structural mutual information (see Section

4.1) and enhances state-action coverage conditioned by the agent's policy by maximizing structural entropy (see Section 4.2). The overall architecture of our framework is illustrated in Figure 2.

4.1 State-action Representation Learning

To effectively learn dynamics-relevant state-action representations, we present an innovative embedding principle that maximizes the structural mutual information with subsequent states and minimizes it with current states.

Structural Mutual Information Principle. In this phase, the input variables at timestep t encompass the current observation O t and the action A t , with the target being the subsequent observation O t +1 . We denote the encoding of observations O t and O t +1 as states S t and S t +1 , respectively. We aim to generate a latent representation Z t for the tuple ( S t , A t ) , which preserves information relevant to S t +1 while compressing information pertinent to S t . This embedding process mentioned above is detailed as follows:

where f s and f z are the respective encoders for states and state-action pairs (step I. a in Figure 2). For the state-action embeddings Z t , we construct two undirected bipartite graphs, G zs and G zs ′ , as shown in step I. b of Figure 2. These graphs represent the joint distributions of Z t with the current states S t and subsequent states S t +1 . In step I. c of Figure 2, we generate 2 -layer approximate binary trees for G zs and G zs ′ and calculate the mutual information I SI ( Z t ; S t ) and I SI ( Z t ; S t +1 ) using Equation 5. Building upon the Information Bottleneck (IB) [Tishby et al., 2000], we present an embedding principle that aims to minimize I SI ( Z t ; S t ) while maximizing I SI ( Z t ; S t +1 ) , as demonstrated in step I. d of Figure 2. When the joint distribution between variables Z t and S t +1 shows a one-to-one correspondence-meaning for each z t ∈ Z t value, there is a unique s t +1 ∈ S t +1 corresponding to it, and vice versa-their mutual information takes its maximum value. We introduce a theorem to elucidate the equivalence between I SI ( Z t ; S t +1 ) and I ( Z t ; S t +1 ) under this condition.

Theorem 4.1. For a joint distribution of variables X and Y that shows a one-to-one correspondence, I SI ( X ; Y ) equals I ( X ; Y ) .

A detailed proof is provided in Appendix B.3. When Z t and S t are mutually independent, the mutual information I ( Z t ; S t ) attains its minimum value. Our I SI ( Z t ; S t ) goes beyond this, incorporating the joint entropy H ( Z t , S t ) according to Theorem 3.4. This integration effectively eliminates the irrelevant information embedded in the representation variable Z t , a significant step in our research. Consequently, structural mutual information can be considered a reasonable and desirable learning objective for acquiring dynamics-relevant state-action representations.

Representation Learning Objective. Due to the computational challenges of directly minimizing I SI ( Z t ; S t ) , we formulate a variational upper bound I ( Z t ; S t ) + H ( Z t | S t ) + H ( S t ) (see Appendix C.2). Noting that the term H ( S t ) is extraneous to our model, we equate the minimization of I SI ( Z t ; S t ) to the minimization of I ( Z t ; S t ) and H ( Z t | S t ) .

By employing a feasible decoder to approximate the marginal distribution of Z t , we derive an upper bound of I ( Z t ; S t ) (See Appendix C.3) as follows:

To concurrently decrease the conditional entropy H ( Z t | S t ) , we introduce a predictive objective (See Appendix C.4) through a tractable decoder q z | s for the conditional probability p ( z t | s t ) as follows:

where L z | s represents the log-likelihood of Z t given S t .

To efficiently optimize I SI ( Z t ; S t +1 ) , we maximize its lower bound, I ( Z t ; S t +1 ) , as detailed in Theorem 3.4. By utilizing an alternative decoder q s | z for the conditional probability p ( s t +1 | z t ) , we obtain a lower bound of I ( Z t ; S t +1 ) (See Appendix C.5) as follows:

where L s | z denotes the log-likelihood of S t +1 conditioned on Z t .

Within our SI2E framework, the definitive loss for representation learning is a combination of the above bounds, L = L up + L z | s + η · L s | z , where η is a Lagrange multiplier used to maintain equilibrium among the specified terms.

4.2 Maximum Structural Entropy Exploration

We have designed a unique intrinsic reward mechanism to address the challenge of imbalance exploration towards low-value states in traditional entropy strategies, as discussed by [Kim et al., 2023]. Specifically, we generate a hierarchical state-action structure based on the agent's policy and define value-conditional structural entropy as an intrinsic reward for effective exploration.

Hierarchical State-action Structure. Derived from the history of agent-environment interactions, we extract state-action pairs (step II. a in Figure 2) to form a complete graph G sa (step II. b in Figure 2) that encapsulates the value relationships caused by the agent's policy. Within this graph, any two vertices v i and v j is connected by an undirected edge whose weight w ij is determined as: w ij = || π ( s i t , a i t ) -π ( s j t , a j t ) || 2 . The state-action pairs ( s i t , a i t ) and ( s j t , a j t ) are associated with vertices v i and v j , respectively. We minimize the 2 -dimensional structural entropy of this graph G sa to generate its 2 -layer optimal encoding tree, denoted as T ∗ sa (step II. c in Figure 2). This tree T ∗ sa delineates a hierarchical community structure among the state-action vertices, with the root node corresponding to a community encompassing all vertices. Each intermediate node in T ∗ sa corresponds to a sub-community, including vertices that share similar π values.

Value-conditional Structural Entropy. To measure the extent of the policy's coverage across the state-action space, we construct an additional distribution graph G ′ sa (step II. d in Figure 2). The graph G ′ sa shares the same vertex set as G sa . The following proposition confirms the existence of such a graph, with a detailed proof provided in Appendix B.4.

Proposition 4.2. Given positive visitation probabilities p ( s 0 t , a 0 t ) , . . . , p ( s n -1 t , a n -1 t ) for all stateaction pairs, there exists a weighted, undirected, and connected graph G ′ sa , where each vertex's degree d i equals its visitation probability p ( s i t , a i t ) .

In the graph G ′ sa , the set of all state-action vertices is denoted as V 0 , and the set of all stateaction sub-communities is denoted as V 1 . The Shannon entropies associated with the distribution of visitation probabilities for these sets are represented as H ( V 0 ) and H ( V 1 ) , respectively, where H ( V 0 ) = H ( S t , A t ) . Within the 2 -layer state-action community represented by T ∗ sa , we define the structural entropy of G ′ sa using Equation 2, denoted as H T ∗ sa ( G ′ sa ) (step II. e in Figure 2). The following theorem delineates the relationship between the value-conditional entropy H T ∗ sa ( G ′ sa ) with the state-action Shannon entropy H ( S t , A t ) . A detailed proof is provided in Appendix B.5.

Theorem 4.3. For a tuning parameter 0 ≤ ζ ≤ 1 , it holds for the structural entropy H T ∗ sa ( G ′ sa ) and the Shannon entropy H ( S t , A t ) that:

where H ( V 0 ) -H ( V 1 ) is a variational lower bound of H T ∗ sa ( G ′ sa ) . On the one hand, the term H ( V 0 ) ensures maximal coverage of the entire state-action space, analogous to the traditional Shannon entropy. On the other hand, the term H ( V 1 ) mitigates uniform coverage among state-action subcommunities with diverse π values, thus addressing the challenge of imbalance exploration. By identifying the hierarchical state-action structure caused by the agent's policy, the SI2E achieves enhanced maximum coverage exploration, thereby guaranteeing its exploration advantage.

Estimation and Intrinsic Reward. Considering the impracticality of directly acquiring visitation probabilities, we employ the k -NN entropy estimator in Equation 1 to estimate the lower bound:

where d z is the dimension of state-action embedding, n 0 and n 1 are the vertex numbers in V 0 and V 1 , and d ( v ) is twice the distance from vertex v to its k -th nearest neighbor. By ignoring the constant term in Equation 12, we define the intrinsic reward r i t and train RL agents to address the target task using a combined reward r t = r e t + β · r i t (step II. f in Figure 2), where β is a positive hyperparameter that modulates the trade-off between exploration and exploitation. The pseudocode, complexity analysis, and limitations of our framework are provided in Appendix A.

5 Experiments

In this section, we present a comprehensive suite of comparative experiments on MiniGrid [ChevalierBoisvert et al., 2018], MetaWorld [Yu et al., 2020], and the DeepMind Control Suite (DMControl) [Tunyasuvunakool et al., 2020] to evaluate the effectiveness of SI2E in terms of both final performance and sample efficiency. Consistent with previous work [Zeng et al., 2023c], we measure the required steps to attain specified rewards ( 0 . 9 times SI2E's convergence reward) as a benchmark for assessing sample efficiency. For the SI2E implementation, we employ a randomly initialized encoder optimized to minimize the combined loss L . All experiments are conducted with 10 different random seeds, and the learning curves are delineated in Appendix E.

5.1 MiniGrid Evaluation

Initially, we assess our framework on navigation tasks using the MiniGrid benchmark, which includes goal-reaching tasks in sparse-reward environments. This setting is partially observable: the agent receives a 7 × 7 × 3 embedding of the immediate surrounding grid rather than the entire grid. For comparative purposes, we employ the A2C agent [Mnih et al., 2016] with Shannon entropy (SE) [Seo et al., 2021] and value-based state entropy (VCSE) [Kim et al., 2023] as our baselines. Table 1 (upper) displays the average values and standard deviations of success rates and required steps for various navigation tasks. The tasks encompass navigation with obstacles (SimpleCrossingS 9 N 1 ), long-horizon navigation (RedBlueDoors, DoorKey, and Unlock), and long-horizon navigation with obstacles (KeyCorridorS 3 R 1 ). The SI2E consistently exhibits enhanced final performance and sample efficiency across tasks, with an average success rate increase of 4 . 92% , from 89 . 97% to 94 . 40% , and an average decrease in required steps of 38 . 10% , from 635 . 65 K to 393 . 47 K . In the RedBlueDoors task, where baseline performances are inadequate, our SI2E significantly improves the success rate from 79 . 82% to 85 . 80% and reduces the required steps from 1161 . 90 K to 461 . 90 K .

5.2 MetaWorld Evaluation

We further evaluate the SI2E framework on visual manipulation tasks from the MetaWorld benchmark, which presents exploration challenges due to its large state space. We select the model-free DrQv2 algorithm as the underlying RL methodology. Adhering to the setup of [Seo et al., 2023], we employ the same camera configuration and normalize the reward with a scale of 1 . We summarize the success rates and required steps for all exploration methods across six MetaWorld tasks in Table 1 (lower).

Our SI2E framework enables the DrQv2 agent to solve all tasks with an average success rate of 97 . 87 after an average of 64 . 79 K environmental steps, significantly outperforming other baselines. Specifically, in the Door Open task, all baselines struggle to achieve a meaningful success rate with a satisfactory number of environmental steps. This result demonstrates the SI2E's effectiveness in improving and accelerating agent exploration in challenging tasks with expansive state-action spaces.

5.3 DMControl Evaluation

Subsequently, we evaluate our framework across various continuous control tasks within the DMControl suite. As the foundational agent, we choose the same DrQv2 algorithm, which operates on pixel-based observations. We incorporate a state-action exploration baseline, MADE [Zhang et al., 2021b], for a more comprehensive comparison. We evaluate all exploration methods across six continuous control tasks, documenting the episode rewards in Table 2. Observations reveal that SI2E remarkably increases the mean episode reward in each DMControl task. Specifically, in the Cartpole Swingup task characterized by sparse rewards, our framework boosts the average reward from 707 . 76 to 795 . 09 , resulting in a 12 . 34% improvement in the final performance. Moreover, we compare the sample efficiency of SI2E and the best-performing baseline in Appendix E.3.

These results not only demonstrate the effectiveness of SI2E in acquiring dynamics-relevant representations for state-action pairs but also highlight its potential to motivate agents to explore the state-action space. To better understand the rationality and advantage of the SI2E framework, we provide visualization experiments in Appendix E.4.

5.4 Ablation Studies

To further investigate the impact of two critical components within the SI2E framework, embedding principle (Section 4.1) and intrinsic reward mechanism (4.2), we perform ablation studies on MetaWorld and DMControl tasks, focusing on two distinct variants: (i) SI2E-DB, which utilizes the DB bottleneck [Bai et al., 2021] for learning state-action representations, and (ii) SI2E-VCSE, employing the state-of-the-art VCSE approach [Kim et al., 2023] for calculating intrinsic rewards. As depicted in Figure 3, SI2E surpasses all variants regarding final performance and sample efficiency. This outcome underscores the essential role of these critical components in conferring SI2E's superior capabilities. Additional ablation studies for the parameters β and n are available in Appendix E.5.





5



5





























4



5





























5



5





























5



5























6 Related Work

6.1 Maximum Entropy Exploration

Maximum entropy exploration has evolved from focusing initially on unsupervised methods to incorporating task rewards in more advanced supervised models. In the unsupervised paradigm,

agents autonomously acquire behaviors by using state entropy as an intrinsic reward for exploration [Liu and Abbeel, 2021, Mutti et al., 2022, Yang and Spaan, 2023]. In contrast, in the supervised paradigm, agents aim to maximize state entropy in conjunction with task rewards [Seo et al., 2021, Yuan et al., 2022].

However, these methods face challenges due to imbalances in the distributions of states with differing policy values. To address this issue, a value-based approach [Kim et al., 2023] has been proposed, which integrates value estimates into the entropy calculation to ensure balanced exploration. Nevertheless, the effectiveness of this approach heavily depends on the partitioning structure of states according to policy values, requiring prior knowledge about downstream tasks.

In this work, we leverage structural information principles to derive the hierarchical state-action structure in an unsupervised manner. We further define the value-conditional structural entropy as an intrinsic reward to achieve more effective agent exploration. Compared to current maximum entropy explorations, SI2E introduces an additional sub-community entropy and minimizes this entropy to motivate the agent to explore specific sub-communities with high policy values. This approach helps avoid redundant explorations within low-value sub-communities and achieves enhanced maximum coverage exploration. Our method allows for balanced exploration without requiring prior knowledge of downstream tasks, effectively addressing the limitations of previous approaches.

6.2 Representation Learning

Novelty Search [Tao et al., 2020] and Curiosity Bottleneck [Kim et al., 2019b] leverage the Information Bottleneck principle for effective representation learning. Additionally, the EMI method [Kim et al., 2019a] maximizes mutual information in both forward and inverse dynamics to develop desirable representations. However, these methods are limited by the lack of an explicit mechanism to address the white noise issue in the state space. To overcome this challenge, the Dynamic Bottleneck model [Bai et al., 2021] is introduced for robust exploration in complex environments.

Our work defines structural mutual information to measure the structural similarity between two variables for the first time. Additionally, we present an innovative embedding principle that incorporates the entropy of the representation variable. This approach more effectively eliminates irrelevant information than the traditional information bottleneck principle.

6.3 Structural Information Principles

Since the introduction of structural information principles [Li and Pan, 2016], these principles have significantly transformed the analysis of network complexities, employing metrics such as structural entropy and partitioning trees. This innovative approach has not only deepened the understanding of network dynamics-even in the context of multi-relational graphs [Cao et al., 2024a]-but has also led to a wide array of applications across different domains. The application of structural information principles has extended to various fields, including graph learning [Wu et al., 2022], skin segmentation [Zeng et al., 2023a], and the analysis of social networks [Peng et al., Zeng et al., 2024, Cao et al., 2024b]. In the domain of reinforcement learning, these principles have been instrumental in defining hierarchical action and state abstractions through encoding trees [Zeng et al., 2023b,c], marking a significant advancement in robust decision-making frameworks.

7 Conclusion

We propose SI2E, a novel exploration framework based on structural information principles. This framework defines structural mutual information to effectively capture state-action representations relevant to environmental dynamics. It maximizes the value-conditional structural entropy to enhance coverage across the state-action space. We have established theoretical connections between SI2E and traditional information-theoretic methodologies, underscoring the framework's rationality and advantages. Through extensive and comparative evaluations, SI2E significantly improves final performance and sample efficiency over state-of-the-art exploration methods. Our future work includes expanding the height of encoding trees and the range of experimental environments. Our goal is for SI2E to remain a robust and adaptable tool in reinforcement learning, particularly suited to high-dimensional and sparse-reward contexts.

Acknowledgments

The corresponding authors are Hao Peng and Angsheng Li. This work is supported by the National Key R&D Program of China through grant 2021YFB1714800, NSFC through grants 61932002, 62322202, and 62432006, Beijing Natural Science Foundation through grant 4222030, Local Science and Technology Development Fund of Hebei Province Guided by the Central Government of China through grant 246Z0102G, Hebei Natural Science Foundation through grant F2024210008, and Guangdong Basic and Applied Basic Research Foundation through grant 2023B1515120020.

References

Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations , 2018.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning , pages 507-517. PMLR, 2020.

Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems , 34:17007-17020, 2021.

Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics , pages 679-684, 1957.

Yuwei Cao, Hao Peng, Angsheng Li, Chenyu You, Zhifeng Hao, and Philip S. Yu. Multi-relational structural entropy. In The 40th Conference on Uncertainty in Artificial Intelligence , pages 42894298, 2024a.

Yuwei Cao, Hao Peng, Zhengtao Yu, and Philip S. Yu. Hierarchical and incremental structural entropy minimization for unsupervised social event detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8255-8264, 2024b.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. 2018.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Riashat Islam, Raihan Seraj, Pierre-Luc Bacon, and Doina Precup. Entropy regularization with discounted future state distribution in policy gradient methods. ArXiv , 2019.

Daejin Jo, Sungwoong Kim, Daniel Nam, Taehwan Kwon, Seungeun Rho, Jongmin Kim, and Donghoon Lee. Leco: Learnable episodic count for task-specific intrinsic reward. Advances in Neural Information Processing Systems , 35:30432-30445, 2022.

Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. arXiv preprint arXiv:2305.19476 , 2023.

Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference on Machine Learning , pages 3360-3369. PMLR, 2019a.

Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity-bottleneck: Exploration by distilling task-specific novelty. In International conference on machine learning , pages 3379-3388. PMLR, 2019b.

Angsheng Li and Yicheng Pan. Structural information and dynamical complexity of networks. IEEE Transactions on Information Theory , 62:3290-3339, 2016.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems , 34:18459-18473, 2021.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Mirco Mutti, Riccardo De Santi, and Marcello Restelli. The importance of non-markovianity in maximum state entropy exploration. In International Conference on Machine Learning , pages 16223-16239. PMLR, 2022.

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems , 30, 2017.

Yicheng Pan, Feng Zheng, and Bingchen Fan. An information-theoretic perspective of hierarchical clustering. arXiv preprint arXiv:2108.06036 , 2021.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778-2787. PMLR, 2017.

Hao Peng, Jingyun Zhang, Xiang Huang, Zhifeng Hao, Angsheng Li, Zhengtao Yu, and Philip S Yu. Unsupervised social bot detection via structural information theory. ACM Transactions on Information Systems , pages 1-42.

Óscar Pérez-Gil, Rafael Barea, Elena López-Guillén, Luis M Bergasa, Carlos Gomez-Huelamo, Rodrigo Gutiérrez, and Alejandro Diaz-Diaz. Deep reinforcement learning based control for autonomous vehicles in carla. Multimedia Tools and Applications , 81(3):3553-3576, 2022.

Sahaya Beni Prathiba, Gunasekaran Raja, Kapal Dev, Neeraj Kumar, and Mohsen Guizani. A hybrid deep reinforcement learning for autonomous vehicles smart-platooning. IEEE Transactions on Vehicular Technology , 70(12):13340-13350, 2021.

Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning , pages 9443-9454. PMLR, 2021.

Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning , pages 1332-1344. PMLR, 2023.

Claude Shannon. The lattice theory of information. Transactions of the IRE professional Group on Information Theory , 1(1):105-107, 1953.

Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences , 23 (3-4):301-321, 2003.

Ruo Yu Tao, Vincent François-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems , 33:8114-8126, 2020.

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020.

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, and etc. Alphastar: Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Shanchuan Wan, Yujin Tang, Yingtao Tian, and Tomoyuki Kaneko. Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 4289-4298, 2023.

Junran Wu, Xueyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In ICML , pages 24017-24030. PMLR, 2022.

Qisong Yang and Matthijs TJ Spaan. Cem: Constrained entropy maximization for task-agnostic safe exploration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10798-10806, 2023.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations , 2021.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Mingqi Yuan, Man-On Pun, and Dong Wang. Rényi state entropy maximization for exploration acceleration in reinforcement learning. IEEE Transactions on Artificial Intelligence , 2022.

Guangjie Zeng, Hao Peng, Angsheng Li, Zhiwei Liu, Chunyang Liu, Philip S Yu, and Lifang He. Unsupervised skin lesion segmentation via structural entropy minimization on multi-scale superpixel graphs. In IEEE ICDM , pages 768-777, 2023a.

Xianghua Zeng, Hao Peng, and Angsheng Li. Effective and stable role-based multi-agent collaboration by structural information principles. Proceedings of the AAAI Conference on Artificial Intelligence , (10):11772-11780, Jun. 2023b.

Xianghua Zeng, Hao Peng, Angsheng Li, Chunyang Liu, Lifang He, and Philip S Yu. Hierarchical state abstraction based on structural information principles. In IJCAI , pages 4549-4557, 2023c.

Xianghua Zeng, Hao Peng, and Angsheng Li. Adversarial socialbots modeling based on structural information principles. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 392-400, 2024.

Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by maximizing rényi entropy for reward-free rl framework. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10859-10867, 2021a.

Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell. Made: Exploration via maximizing deviation from explored regions. Advances in Neural Information Processing Systems , 34:9663-9680, 2021b.

Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

A Framework Details

A.1 Notations

A.2 Tree Optimization on T 2

In this subsection, we have provided additional explanations and illustrative examples for the encoding tree optimization on T 2 . As shown in Figure 4, the stretch operator is executed over sibling nodes α i and α j that share the same parent node, λ . The detailed steps of this operation are as follows:

where α ′ is the added tree node via the stretch operation.

𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

…

…

𝜆𝜆

𝛼𝛼

𝑖𝑖

𝛼𝛼

𝑗𝑗

…

…

𝜆𝜆

𝛼𝛼

𝛼𝛼

𝛼𝛼

′

𝑖𝑖

𝑗𝑗

Figure 4: Stretch operation on sibling nodes within the encoding tree.

The corresponding variation in structural entropy, ∆ H , due to the stretch operation is calculated as:

As shown in Algorithm 1, the HCSE algorithm iteratively and greedily selects the pair of sibling nodes that cause the maximum entropy variation, ∆ H to execute one stretch optimization.

Algorithm 1 The Encoding Tree Optimization on T 2

1: Input: one-layer initial encoding tree T

2: Output: the optimal encoding tree T ∗ ∈ T 2

3: while True do

4: ( α i , α j ) ← maximize the entropy reduction ∆ H caused by one stretch operation

5: if ∆ H = 0 then

6: Break

7: end if

8: Create a new tree node α ′

9: ( α ′ ) -← λ, ( α i ) -← α ′ , ( α j ) -← α ′

10: end while

A.3 Intuitive Example of Optimal Encoding Tree

𝑥𝑥0

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑦𝑦0

𝑦𝑦1

𝑦𝑦3

0.2

0.3

0.2

0.2

0.05

0.03

0.02

a) Bipartite Graph

𝐺𝐺

𝑥𝑥𝑥𝑥

{

𝑥𝑥0

,

𝑦𝑦0

}{

𝑥𝑥1

,

𝑦𝑦1

}{

𝑥𝑥2

,

}{

𝑥𝑥3

,

𝑦𝑦3

}

b) Optimal Encoding Tree

𝑇𝑇

𝑥𝑥𝑥𝑥

∗

{

𝑥𝑥1

,

𝑦𝑦0

}{

𝑥𝑥2

,

𝑦𝑦1

}{

𝑥𝑥3

,

}{

𝑥𝑥0

,

𝑦𝑦3

}

c) Resulting Encoding Tree

𝑇𝑇

𝑥𝑥𝑥𝑥

1

A.4 The Pseudocode of SI2E

A.5 Complexity Analysis of SI2E

Within the SI2E framework, we analyze the time complexities of critical components independent of the underlying RL algorithm. During the state-action representation phase, the construction of bipartite graphs takes O ( n 2 ) time complexity, the generation of 2 -layer approximate binary trees requires O ( n · log 2 n ) time complexity, and the calculation of mutual information involves a time complexity of O ( n 2 ) . During the effective exploration phase, the generation of hierarchical community structure incurs a O ( n · log 2 n ) complexity, the construction of the distribution graph leads to a complexity of O ( n 2 ) , and value-conditional structural entropy is calculated with O ( n ) time complexity.

A.6 Limitations

Our work, which is a result of thorough research, aims to address the limitations of information theory methods and structural information theory research in reinforcement learning. Therefore, we have selected the state-of-the-art information theory exploration method as the baseline in our evaluation. Despite the current limitations in height due to complexity and cost issues, the encoding tree structure in the SI2E framework holds immense potential. In our future research, we are optimistic about expanding its height further and conducting more research on the advantages and restrictions brought by this expansion.

B Theorem Proofs

B.1 Proof of Proposition 3.1

Proof. For any two vertices v i ∈ V and v j ∈ V without any edge connecting them, their corresponding tree nodes are denoted as α i and α j . These nodes' parents are initially assigned as the root node λ . Before executing one stretch operation on vertices α i and α j , their structural entropies are calculated as follows:

where d i and d j are the degrees of vertices v i and v j . Post-stretch operation, their structural entropies are given by:

where α ′ are their new common parent node. The absence of an edge between v i and v j ensures that:

The structural entropy of α ′ can be determined as:

The entropy reduction ∆ H , consequent to the stretch operation on vertices v i and v j , is calculated as:

(22)

Given the zero reduction in entropy, as per lines 5 and 6 of the optimization algorithm for T 2 (See Appendix A.2), the stretch operation involving v i and v j is omitted from the optimization process.

B.2 Proof of Theorem 3.4

Proof. The difference between the mutual information I SI ( X ; Y ) and I ( X ; Y ) is expressed as:

Given the conditions p ( x i , y j ) ≤ p ( x i ) ≤ 1 and p ( x i , y j ) ≤ p ( y j ) ≤ 1 , the following inequalities are satisfied:

By defining ϵ i = log p ( x i ,y j ) p ( x i ) and ϵ j = log p ( x i ,y j ) p ( y j ) , we obtain the following results:

Therefore,

B.3 Proof of Theorem 4.1

Proof. For a one-to-one correspondence of variables X and Y , the joint probability of a tuple ( x i , y j ) is as follows:

The calculation for I SI ( X ; Y ) is carried out in the following manner:

Similarly, the calculation for I ( X ; Y ) proceeds as follows:

Hence, I SI ( X ; Y ) equals I ( X ; Y ) when the joint distribution between X and Y shows a one-to-one correspondence.

B.4 Proof of Proposition 4.2

Proof. Now, we employ mathematical induction to demonstrate the existence of the graph G ′ sa . Base Case ( n = 2 ): Suppose the degree distribution of two vertices is given by ( p 0 , p 1 ) with p 0 ≤ p 1 . We construct the graph G ′ sa as follows:

Create an edge with weight p 0 between v 0 and v 1 .

Add a self-connected edge at vertex v 1 with weight p 1 -p 0 .

Inductive Case ( n = k +1 ): Consider the addition of a new vertex v k to construct G ′ sa with k +1

Inductive Step ( n = k ): Assume that, the graph G ′ sa with k vertices exists and satisfies Proposition 4.2.

vertices using the following steps:

Start with a subgraph that includes the first k vertices, yielding a distribution of ( p ′ 0 , . . . , p ′ k -1 ) with ∑ k -1 i =0 p ′ i = 1 .

Modify the weight of all edges connected to each vertex v i ( 0 ≤ i ≤ k ) by a factor (1 -p n ) p i p ′ .

For each vertex v i , create an edge with weight p i p n connecting it to the new vertex v k .

Add a self-connected edge at vertex v k with weight p 2 k .

These modifications ensure that the degree distribution of the graph remains consistent with the addition of the new vertex, thus completing the inductive step and proving the existence of G ′ sa for any n .

B.5 Proof of Theorem 4.3

In the tree T ∗ sa , we denote the i -th intermediate node as α i and its j -th child node as α ij . The single state-action vertex in the corresponding subset of α ij is assumed as ( s ij , a ij ) . In the graph G ′ sa , the degree of any state-action vertex ( s ij , a ij ) is equated to its visitation probability, thereby:

The 2 -dimensional value-conditional structural entropy H T ∗ sa ( G ′ sa ) is calculated through the following expressions:

Given that p ( s ij , a ij ) · vol( α i ) ≤ p ( s ij , a ij ) ≤ vol( α i ) , this inequalities hold that:

By defining ζ ij = log p ( s ij ,a ij ) p ( s ij ,a ij ) vol( α i ) , we obtain that:

Selecting the minimal ζ -value as ζ ∗ allow us to reformulate Equation 33 as follows:

i

C Detailed Derivations

C.1 Derivation of I SI ( X ; Y )

For each intermediate node α i ∈ T ∗ xy with vertex subset { x i , y i } , the entropy sum of this node and its children is calculated as follows:

The H SI ( X ) + H SI ( Y ) -H T ∗ xy ( X,Y ) term in Equation 5 is calculated as follows:

Proposition 3.1 ensures that:

Consequently, we can reformulate Equation 40 as follows:

(42)

For any integer l > 0 , the H SI ( X ) + H SI ( Y ) -H T l xy ( X,Y ) term in Equation 5 is given by:

Consequently,

C.2 Upper Bound of I SI ( Z t ; S t )

Theorem 3.4 assures the following inequality:

Given the relationship between joint entropy and conditional entropy in traditional information theory, this inequality can be reformulated as follows:

C.3 Upper Bound of I ( Z t ; S t )

Through the non-negativity of KL-divergence, the following upper bound of I ( Z t ; S t ) holds that:

C.4 Upper Bound of H ( Z t | S t )

Through the non-negativity of KL-divergence, the following upper bound of H ( Z t | S t ) holds that:

C.5 Lower Bound of I ( Z t ; S t +1 )

Leveraging the non-negative Shannon entropy and KL-divergence, we obtain the lower bound of I ( Z t ; S t +1 ) :

D Experimental Details

In the experiments conducted for this work, we utilize a single NVIDIA RTX A1000 GPU and eight Intel Core i9 CPU cores clocked at 3.00GHz for each training run. The total number of environmental steps was set to 3000 K/ 1000 K for the MiniGrid benchmark, 200 K/ 100 K for the MetaWorld benchmark, and 250 K for the DeepMind Control Suite (DMControl).

D.1 Implementation Details

A2C implementation details. In our implementation of the A2C algorithm, we utilize the official RE3 implementation 2 , adhering to the pre-established hyperparameters set, except where explicitly noted. State representations in baselines are derived using a fixed encoder that is randomly initialized, and intrinsic rewards are normalized based on the standard deviation computed from sample data, consistent with the original methodology. However, this normalization process is omitted in the intrinsic reward calculations for VCSE and SI2E implementations. Across all exploration methods, we maintain fixed scale parameters β = 0 . 005 and k = 5 , in line with the original framework. The comprehensive hyperparameters for the A2C algorithm are detailed in Table 4.

DrQv2 implementation details. For the DrQv2 algorithm, we employ its official implementation 3 [Yarats et al., 2021], maintaining the original hyperparameter settings unless specified otherwise. A fixed noise level of 0 . 2 and k = 12 are used for all exploration methods, including SE, VCSE, MADE, and SI2E. In the calculation of intrinsic rewards in baselines, we train the Intrinsic Curiosity Module [Pathak et al., 2017] using representations from the visual encoder to measure vertex distance in the estimation of value-conditional structural entropy. Specific hyperparameters in the DrQv2 are summarized in Table 5.

D.2 Environment Details

MiniGrid Experiments. In our MiniGrid benchmark experiments, we encompass six navigation tasks, including RedBlueDoors, SimpleCorssingS 9 N 1 , KeyCorridor, DoorKey6 x 6 , DoorKey8 x 8 , and Unlock, with visual representations provided in Figure 6. Notably, all tasks are employed in their original forms without any modifications.

Fo

Fo

MetaWorld Experiments. In our evaluation using the MetaWorld benchmark, we conduct experiments on six manipulation tasks: Door Open, Drawer Open, Faucet Open, Window Open, Button Press, and Faucet Close. These tasks are visualized in Figure 7.

江

DMControl Experiments. Our research in DMControl suite focues on six continuous control tasks, specifically Hopper Stand, Cheetah Run, Quadruped Walk, Pendulum Swingup, Cartpole Balance Sparse, and Cartpole Swingup Sparse. And visualizations of thes tasks are provided in Figure 8.

E Additional Experiments

E.1 Experiments on MiniGrid Benchmark

Figure 9 illustrates the learning curves for the A2C algorithm integrated with our SI2E framework, as well as with other exploration baselines, SE and VCSE. The corresponding variants are labeled as A2C, A2C+SE, A2C+VCSE, and A2C+SI2E. These results demonstrate that SI2E consistently outperforms other baselines across various navigation tasks. In terms of final performance, A2C+SI2E achieves an average success rate of 94 . 40% across all navigation tasks, significantly outperforming the second-best A2C+VCSE, which has an average success rate of 89 . 78% . Regarding sample efficiency, SI2E converges in fewer than 50% of the total environmental steps in almost all tasks, except for DoorKey8 x 8 . This indicates SI2E's effectiveness in enhancing the agent's exploration of the state-action space, surpassing the baseline methods.





5



6



6



6



6



6



















5



5



5



5



6



















5



6



6



6



6



6



















5



5



5



5



6























5



6



6



6



6



6





















5



5



5



5



6

























To further substantiate our results on the Minigrid environment, we have introduced Leco[Jo et al., 2022] and DEIR[Wan et al., 2023], two additional state-of-the-art baselines representing advanced mechanisms for episodic intrinsic rewards. These results in Table 6 demonstrate that our method consistently maintains a performance advantage in terms of effectiveness and efficiency, even when compared to advanced episodic intrinsic reward mechanisms.

E.2 Experiments on MetaWorld Benchmark

Figure 10 summarizes the learning curves for the DrQv2 algorithm with different exploration methods, including SI2E (ours), SE, and VCSE. These variants are denoted as DrQv2, DrQv2+SE, DrQv2+VCSE, and DrQv2+SI2E, respectively. As shown in Figure 10, SI2E consistently and significantly achieves higher success rates with fewer environmental steps across all manipulation tasks. On average, SI2E improves the final performance by 10 . 21% and sample efficiency by 45 . 06% .





4



5



5



5























4



5



5



5



















4



5



5



5



















4



4



4



4



5























4



4



4



4



5





















4



5



5



5

























To provide additional validation, we conducted further experiments in the MetaWorld environment using the advanced TACO [Zheng et al., 2024] as the underlying agent. The new experimental results, as documented in Table 7, demonstrate the robustness and superior performance of our method with different underlying agents.

E.3 Experiments on DMControl Suite

For each task, we benchmark the convergence reward of the best-performing baseline as the target and track the environmental steps required by both SI2E and this baseline to reach the target. As illustrated in Figure 11, SISA demonstrates an average improvement of 35 . 60% in sample efficiency. This improvement is reflected in a reduction in the required environmental steps, decreasing from 31 . 83% to 20 . 5% of the total steps required to achieve the reward target.





































































Figure 12 shows the learning curves for the DrQv2 algorithm when integrated with our SI2E framework and other exploration baselines, SE, VCSE, and MADE. The variants are identified as DrQv2, DrQv2+SE, DrQv2+VCSE, DrQv2+MADE, and DrQv2+SI2E. These results reveal that SI2E exploration significantly improves the sample efficiency of DrQv2 in both sparse reward and dense reward tasks, outperforming all other baselines. Particular in sparse reward tasks (Cartpole Balance and Cartpole Swingup), our framework successfully accelerates training and achieves higher episode rewards. This suggests that SI2E avoids exploring states that may not contribute to task resolution, thereby enhancing performance with a 5 . 17% improvement in final performance and a 15 . 28% increase in sample efficiency.

5











































4



5



5



5



5





















4



5



5



5



5





















4



5



5



5



























E.4 Visualization Experiment.

To understand the learned representation through our structural mutual information principle (Section 4.1), we use t-SNE to project the original observation O and state-action embedding Z into 2 -dimensional vectors. Figure 13a visualizes these 2 -dimensional vectors and measures the Euclidean distance between temporally consecutive vectors. Our presented principle aligns all movements on the same curve and reduces their distance by an average of 36 . 41 , demonstrating its effectiveness for dynamics-relevant state-action embedding.

To analyze the benefits of our intrinsic reward mechanism (Section 4.2), we keep the map configuration constant in the SimpleCrossing task and display the exploration paths of various methods in Figure 13b. Our SI2E, in comparison to the SE and VCSE baselines, prevents biased exploration towards low-value areas and effectively explores high-value crucial areas, like the crossing point, with fewer environmental steps, ensuring its performance superiority.

20

0

20

20

0

20

2-dimensional Observations

40

20

0

20

20

0

20

2-dimensional Representations

0

100

200

300

400

500

Environmental Step

0

20

40

Euclidean Distance

Distances between Consecutive Vectors

Original Observation

State-action Embedding

(a) Representation visualization

SimpleCrossing

25K Steps

50K Steps

75K Steps

100K Steps

A2C+SE

A2C+VCSE

A2C+SI2E

(b) Exploration visualization

Moreover, we have provided the visualizations of state coverage of our SI2E framework and other baselines (SE and VCSE) in the continuous Cartpole Balance task in Figure 14. Compared with the SE and VCSE baselines, our SI2E framework achieves a more efficient exploration strategy by minimizing occurrences in extreme positions and angles, resulting in a higher density in key areas.



























E.5 Ablation Studies

To investigate the influence of parameters β and n on our framework's performance, we incrementally adjust these parameters across two distinct tasks: DMControl tasks Hopper Stand and Pendulum Swingup. We meticulously document the resulting learning curves to assess the outcomes. Figure 15a illustrates that an increase in parameter β consistently enhances performance across both tasks, substantiating the effectiveness of our exploration method. Conversely, Figure 15b demonstrates that variations in batch size n yield comparable performance outcomes, particularly notable in Pendulum Swingup task, thereby confirming the SI2E's stability amidst variations in batch size.





5



5



























5



5



























5



5



























5



5























(a) Effect of scale parameter β

(b) Effect of batch size n

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly outlined the contributions and scope of our paper in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have outlined the limitations of our paper in the conclusion and the details are provided in our appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theoretical result, we have included the complete set of assumptions and a correct proof in our appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The source codes are accessible via an anonymous link in the introduction, and specific experimental setups are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The main experimental results can be replicated using the source codes accessible via an anonymous link provided in the introduction.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed experimental settings (hyperparameter and optimizer) in our evaluation and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In our evaluation section, we have conducted multiple trials with different random seeds, and the charts reflect the average performance and standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided sufficient information on the computer resources in our appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper strictly maintains anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is focused on fundamental research in reinforcement learning and is not specifically tied to any particular applications. Furthermore, there is no direct path to negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We appropriately cited the original papers for our experimental datasets, including MiniGrid, MetaWorld, and the DeepMind Control Suite.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to tackle the issue in existing RL exploration methods - the ignorance of inherent structure within the state and action space. Therefore the authors propose a new intrinsic reward mechanism that maximizes value-conditional structural entropy and it is adaptable to high-dimensional RL environments with sparse rewards.\n\nSTRENGTHS:\n- The paper is well-motivated. \n- The presentation is clear and easy to follow, with enough proofs to support the method design. \n- The authors present comprehensive experimental evaluations on both procedurally-generated environments and high-dimensional continuous control problems.\n\nWEAKNESSES:\n- In L78-80, the authors claim that SI2E significantly improves final performance compared to SOTA. However for experiments with on Minigrid are not compared to sota (for example, some episodic intrinsic reward mechanisms), which seems to be over-claimed. The authors should better add more sota baselines or change the claims in the introduction.\n- The authors do not mention problems regarding potential asymptotic inconsistency. Is the proposed method asymptotically consistent? How will the proposed intrinsic reward get rid of potential influence on the final performance?\n\nQUESTIONS:\n- For Figure 3, how many random seeds are used? The authors only plot a single seed (or average performance) without variance for this experiment, unlike other experiments. Is there a reason for it? \n- For Section 5.4 Ablation Studies, the difference among SI2E, SI2E-DB, and SI2E-VCSE, doesn't seem to be very noticeable, especially for final performance. Can the authors show more results on more tasks with clearer difference?\n\nLIMITATIONS:\n- I don't find any big limitation remains undiscussed for now. However I do have some concerns and questions mentioned in the weaknesses and questions sections.\n\n(Post Rebuttal score 5->6)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new exploration scheme for RL agents by using structural mutual information for dynamics-aware state-action representation and an intrinsic reward to enhance state-action coverage by maximizing value-conditional structural entropy\n\nSTRENGTHS:\n- constructing intrinsic reward using graph-based information for RL agent exploration is somewhat novel.\n- extensive comparison with various baselines across a set of tasks demonstrates superior task performance and sample efficiency.\n\nWEAKNESSES:\n### Unclear presentation\n\n- lines 31-38: the statements don't clarify that entropy maximization strategies use state entropy as an intrinsic reward to encourage exploration, leading to confusion about the connection between computing intrinsic rewards and entropy maximization strategies (lines 37-38). Additionally, there's no explanation for why intrinsic rewards based on state values can mitigate the mentioned issue.\n- line 144: the stretch operator and the HCSE algorithm are not clearly explained, making it hard to understand their functions. Simply adding a reference is not sufficient.\n\n- some claims, such as those in lines 35-36 and 200-201, lack evidence or references\n\n### Figures\n\n- Figure 1: It lacks details, for example, how structural information selectively removes redundant transitions in the example. It does not effectively illustrate the benefit of using structural information\n- Figure 2: the presentation is cluttered\n   - the layout is extremely tight with a disordered flow of annotations (vertical and horizontal), and the caption lacks clarification and is not self-contained.\n   - the colour scheme (of nodes and vertices) in the two steps is unspecified\n   - the space between Figure 2 and line 158 is too narrow\n\n### Unclear and non-detailed method presentation\n\n- the use of $T$ to represent both the tree and subsets of graph vertices (lines 117-120) is confusing\n- no discussion about why 2-layer binary trees are used (lines 139-140), which is necessary to understand the approach's limitations\n- only a minimal description of how an optimal encoding tree is generated from $G_{sa}$ (lines 230-231) in sec. 4.2\n\n### Related work\nthe related work section is short and unstructured. It lists related works without clearly stating how this work differs from others, its unique aspects, and the issues it addresses. It also does not show how learning dynamics-relevant representations for state-action pairs differs from other recent representation learning approaches\n\n### Minor points\n\n- line 34, Renyi entropy needs a citation\n- eq 2 misses explanation of $\\alpha^-$\n- line 139, structur -> structure\n- lines 193-194, should use double dashes\n- SI2E in CartPole Swingup Sparse doesn't show superiority in sample efficiency (Figure 3)\n- as to line 321, better to briefly summarize the ablation findings on finetuning $\\beta$ and $n$ before referring to the appendix\n- the best-performing baseline is not specified in sec. 5.3 and appendix E.3\n\nQUESTIONS:\n### Structural entropy and graph\n- what is the intuition or rationale behind 1) using structural information for encouraging agent exploration and 2) why/how SI boosts efficient exploration in RL?\n- can a vertex contain joint variables or only a single variable? What is a single-variable graph?\n- why is structural information limited to a single variable? Can't the underlying graph be expanded to cover multiple variables (lines 128-147)?\n\n### Method's design and choice\n\n- why some tree nodes in step II.c (Figure 2) have more than two children? isn't the encoding tree assumed to be binary?\n- how do the max and min of structural MI capture dynamics-relevant information (lines 58-60 and 176-178), and why is this beneficial for agent exploration?\n- why do we need $l$-transformation for defining structured MI (lines 155-157)?\n- what is $V_1$ exactly, and how are sub-communities obtained? why does $-H(V_1)$ mitigate uniform coverage?\n### Step II in Figure 2 (sec. 4.2)\n\n- how many state-action pairs are used to form the $G_{sa}$ graph for different tasks in the experiment? how many pairs are sufficient for building a *good* $G_{sa}$? how is a bipartite graph constructed for continuous state and action spaces? can you provide a simple example of such a construction?\n\n- why $\\pi$ is used for a value function when it represents a policy?\n\n- why is step d necessary after $T^\\star_{sa}$ is obtained, can't $G_{sa}$ perform the required?\n\n## Figure 1\n\n- what do dotted blue lines of varying intensity represent? why transitions between s2 and s5 are redundant and how *a policy maximizing structural entropy would selectively focus on crucial transitions*\n- which dotted line corresponds to $a_0$? note that the outgoing arrow of state $s_0$ is solid\n- in step I. state-action representation, which (top) boxes correspond to $s_t$, $s_{t+1}$, and $z_t$ respectively?\n\n### Experiments\n\n- is it possible to visualize state coverage in other tasks like meta-world, which has continuous states?\n- why are required steps or success rates not reported for some tasks in Table 1, such as RedBlueDoors and Faucet tasks?\n\n### Other questions\n\n- where is equation 1 used?\n- lines 38-41: What is meant by 'their' in *due to their instability*? What do these lines aim to emphasize?\n- what is the entropy reduction mentioned in supplement A.2?\n\n\nI am willing to increase my score if the clarity questions are addressed.\n\nLIMITATIONS:\nThe authors acknowledge the limitations of the work. It might be useful to discuss how the height of the encoding tree would affect the exploration.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the current field of reinforcement learning, when using representation learning and entropy maximization for exploration, analyses often focus on single variables, making it difficult to capture potential relationships between two variables. Therefore, this paper proposes the SI2E framework. This framework defines the structural mutual information between two variables, based on which state-action representations are obtained. These representations effectively capture the intrinsic relationships between states and actions. Subsequently, an encoding tree is constructed using these representations, and the value- conditional structural entropy is defined. By maximizing this value, the coverage of the state-action space is expanded, enabling the agent to better perform exploration tasks. Extensive experiments on SI2E have been conducted, demonstrating the framework's effectiveness.\n\nSTRENGTHS:\n1. The abstract part of the paper clearly expresses the paper's innovative aspects and the problems it aims to address.\n2. The paper is well-organized and clearly presented. The model introduction is progressive, and the structure is reasonable.\n3. Experiments conducted on multiple datasets have demonstrated the effectiveness of the model.\n\nWEAKNESSES:\n1. The baselines are not strong enough, and more state-of-the-art models should be used for comparison.\n2. The experimental results in Appendix E.5 are more suitable for inclusion in the Parameter Analysis section rather than Ablation Studies.\n3. The comparison of SI2E, SI2E-DB, and SI2E-VCSE is only conducted in Cartpole Balance Sparse and Cartpole Swingup Sparse, making it difficult to demonstrate the importance of the two key components of SI2E in other scenarios.\n4. The Related Work section does not clarify the relationship between SI2E and related works, particularly how SI2E improves upon or innovates based on previous research.\n\nQUESTIONS:\n1.Why was the A2C agent chosen for experiments on MiniGrid instead of DrQv2 or other more recent models?\n2.Why was MADE not included in the comparison for experiments on MiniGrid and MetaWorld?\n\nLIMITATIONS:\nThe authors have further discussed the encoding tree involved in the SI2E framework", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SI2E framework based on structural information principles to overcome inherent limitations in traditional information theory as applied to Reinforcement Learning (RL). The SI2E framework innovatively quantifies the dynamic uncertainties inherent in state-action transitions through a metric termed as 'structural entropy'. The authors define a novel concept, 'structural mutual information', to derive state-action representations that are particularly relevant to the dynamics of the environment. Additionally, they introduce a value-based approach to structural entropy, which is designed to enhance the agent's exploratory capabilities within the RL landscape. The paper establishes robust theoretical connections with established information-theoretic methodologies, thereby substantiating the rationality of the SI2E framework. Empirical validations through comparative experiments in the MiniGrid and DeepMind Control Suite environments underscore the framework's superiority over existing exploration benchmarks, showcasing its practical efficacy.\n\nSTRENGTHS:\n1. Innovative Contribution: The paper presents a novel framework, SI2E, which innovatively applies structural information principles to enhance reinforcement learning. This approach addresses key challenges in traditional information theory within the RL domain, offering a fresh perspective for exploration.\n2. Clarity and Organization: The manuscript is exceptionally well-organized and written with exceptional clarity, making complex concepts easily comprehensible to readers.\n3. Theoretical Rigor: The authors provide a solid theoretical foundation for their method, supported by comprehensive proofs that validate the framework's rationality and effectiveness.\n4. Empirical Validation: The paper includes detailed experimental verifications across a variety of scenarios, which not only demonstrate the effectiveness of the SI2E framework but also highlight its robustness and superiority over current exploration methods.\n5. Comprehensive Appendices: The paper's appendix is thorough, offering additional theoretical proofs, detailed model descriptions, and supplementary experimental data, which further substantiate the paper's claims.\n6. Overall Impact: The paper demonstrates significant strength in both theoretical development and empirical validation, contributing valuable insights to the field of reinforcement learning and warranting acceptance for publication.\n\nWEAKNESSES:\n1. Although the paper includes a time complexity analysis, there is a gap in understanding the actual wall time required for training. Providing insights into the practical training duration would offer a more tangible assessment of the method's efficiency.\n2. The current related work section provides a succinct overview. To strengthen the paper's context and background, a more detailed discussion of classical information-theoretic methodologies and structural information principles in the appendix would be beneficial.\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}