Dynamic Bottleneck for Robust Self-Supervised Exploration
Year: 2021

Abstract:
Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamicsirrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.

heading: 1 Introduction
text: The tradeoff between exploration and exploitation has long been a major challenge in reinforcement learning (RL) [35, 50, 58]. Generally, excessive exploitation of the experience suffers from the potential risk of being suboptimal, whereas excessive exploration of novel states hinders the improvement of the policy. A straightforward way to tackle the exploration-exploitation dilemma is to enhance exploration efficiency while keeping exploitation in pace. When the extrinsic rewards are dense, reward shaping is commonly adopted for efficient exploration. However, in many real-world applications such as autonomous driving [34], the extrinsic rewards are sparse, making efficient exploration a challenging task in developing practical RL algorithms. The situations become even worse when the extrinsic rewards are entirely unavailable. In such a scenario, the task of collecting informative trajectories from exploration is known as the self-supervised exploration [11].
An effective approach to self-supervised exploration is to design a dense intrinsic reward that motivates the agent to explore novel transitions. Previous attempts include count-based [9] and curiosity-driven [39] explorations. The count-based exploration builds a density model to measure the pseudo-count of state visitation and assign high intrinsic rewards to less frequently visited states. In contrast, the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
curiosity-driven methods maintain a predictive model of the transitions and encourage the agent to visit transitions with high prediction errors. However, all these methods becomes unstable when the states are noisy, e.g., containing dynamics-irrelevant information. For example, in autonomous driving tasks, the states captured by the camera may contain irrelevant objects, such as clouds that behave similar to Brownian movement. Hence, if we measure the novelty of states or the curiosity of transitions through raw observed pixels, exploration are likely to be affected by the dynamics of these irrelevant objects.
To encourage the agent to explore the most informative transitions of dynamics, we propose a Dynamic Bottleneck (DB) model, which generates a dynamics-relevant representation Zt of the current state-action pair (St, At) through the Information-Bottleneck (IB) principle [55]. The goal of training DB model is to acquire dynamics-relevant information and discard dynamics-irrelevant features simultaneously. To this end, we maximize the mutual-information I(Zt;St+1) between a latent representation Zt and the next state St+1 through maximizing its lower bound and using contrastive learning. Meanwhile, we minimize the mutual-information I([St, At];Zt) between the state-action pair and the corresponding representation to compress dynamics-irrelevant information. Based on our proposed DB model, we further construct a DB-bonus for exploration. DB-bonus measures the novelty of state-action pairs by their information gain with respect to the representation computed from the DB model. We show that the DB-bonus are closely related to the provably efficient UCB-bonus in linear Markov Decision Processes (MDPs) [1] and the visiting count in tabular MDPs [3, 22]. We further estimate the DB-bonus by the learned dynamics-relevant representation from the DB model. We highlight that exploration based on DB-bonus directly utilize the information gain of the transitions, which filters out dynamics-irrelevant noise. We conduct experiments on the Atari suit with dynamics-irrelevant noise injected. Results demonstrate that our proposed selfsupervised exploration with DB-bonus is robust to dynamics-irrelevant noise and outperforms several state-of-the-art exploration methods.

heading: 2 Related Work
text: Our work is closely related to previous exploration algorithms that construct intrinsic rewards to quantify the novelty of states and transitions. Several early approaches directly define the pseudocount by certain statistics to measure the novelty of states [41, 30]; more recent methods utilize density model [9, 38] or hash map [52, 42] for state statistics. Nevertheless, these approaches are easily affected by dynamics-irrelevant information such as white-noise. The contingency awareness method [16] addresses such an issue by using an attentive model to locate the agent and computes the pseudocount based on regions around the agent. However, such an approach could ignore features that are distant from the agent but relevant to the transition dynamics. Another line of research measures the novelty through learning a dynamics model and then use the prediction error to generate an intrinsic reward. These methods are known as the curiosity-driven exploration algorithms. Similar to the pseudo-count based methods, curiosity-driven methods become unstable in the presence of noises, because the prediction model is likely to yield high error for stochastic inputs or targets. Some recent attempts improve the curiosity-driven approach by learning the inverse dynamics [39] and variational dynamics [7] to define curiosity, or utilizes the prediction error of a random network to construct intrinsic rewards [12]. However, without explicitly removing dynamics-irrelevant information, these methods are still vulnerable to noises in practice [11].
The entropy-based exploration uses state entropy as the intrinsic reward. VISR [19], APT [29] and APS [28] use unsupervised skill discovery for fast task adaptation. In the unsupervised stage, they use k-nearest-neighbor entropy estimator to measure the entropy of state, and then use it as the intrinsic reward. RE3 [47] and ProtoRL [59] use random encoder and prototypes to learn the representation and use state-entropy as bonuses in exploration. Nevertheless, the state entropy will increase significantly if we inject noises in the state space. The entropy-based exploration will be misled by the noises. Previous approaches also quantify the epistemic uncertainty of dynamics through Bayesian network [21], bootstrapped Q-functions [37, 8], ensemble dynamics [40], and Stein variational inference [43] to tackle noisy environments. However, they typically require either complicated optimization methods or large networks. In contrast, DB learns a dynamics-relevant representation and encourages exploration by directly accessing the information gain of new transitions via DB-bonus.
Another closely related line of studies uses the mutual information to promote exploration in RL. Novelty Search (NS) [53] proposes to learn a representation through IB. Curiosity Bottleneck
(CB) [26] also performs exploration based on IB by measuring the task-relevant novelty. However, both NS and CB require extrinsic rewards to learn a value function and are not applicable for selfsupervised exploration. Moreover, NS contains additional k-nearest-neighbor to generate intrinsic reward and representation loss to constrain the distance of consecutive states, which are costly for computation. In contrast, our DB model handles self-supervised exploration without accessing extrinsic rewards. EMI [25] learns a representation by maximizing the mutual information in the forward dynamics and the inverse dynamics , which is different from the IB principle used in our method. In addition, we aim to perform robust exploration to overcome the white-noise problem, while EMI does not have an explicit mechanism to address the noise.
Our work is also related to representation learning in RL. DrQ [60], RAD [27] and CURL [49] learn the state representation by data augmentation and contrastive learning [14, 20, 36] to improve the data-efficiency of DRL. Deep InfoMax [33] and Self-Predictive Representation (SPR) [46] learn the contrastive and predictive representations of dynamics, respectively, and utilize such representations as auxiliary losses for policy optimization. However, none of these existing approaches extracts information that benefits exploration. In contrast, we show that the dynamics-relevant representation learned by DB can be utilized for efficient exploration.

heading: 3 The Dynamic Bottleneck
text: In this section, we introduce the objective function and architecture of the DB model. We consider an MDP that can be described by a tuple (O,A,P, r, γ), which consists of the observation space O, the action space A, the transition dynamics P, the reward function r, and the discount factor γ ∈ (0, 1). At each time step, an agent decides to perform an action at ∈ A after observing ot ∈ O, and then the observation transits to ot+1 with a reward rt received. In this paper, we use upper letters, such as Ot, to denote random variables and the corresponding lower case letter, such as ot, to represent their corresponding realizations.
We first briefly introduce the IB principle [55]. In supervised setting that aims to learn a representation Z of a given input source X with the target source Y , IB maximizes the mutual information between Z and Y (i.e. max I(Z;Y )) and restricts the complexity of Z by using the constrain as I(Z;X) < Ic. Combining the two terms, the objective of IB is equal to max I(Z;Y )− αI(Z;X) with the introduction of a Lagrange multiplier.
DB follows the IB principle [55] to learn dynamics-relevant representation. The input variable of the DB model is a tuple (Ot, At) that contains the current observation and action, and the target is the next observation Ot+1. We denote by St and St+1 the encoding of observations Ot and Ot+1. The goal of the DB model is to obtain a compressed latent representation Zt of (St, At), that preserves the information that is relevant to St+1 only. Specifically, we use fSo and f S m as the encoders of two consecutive observations ot and ot+1, respectively. We parameterize the dynamics-relevant representation zt by a Gaussian distribution with parameter φ, and it takes (st, at) as input. We summarize the DB model as follows,
st = f S o (ot; θo), st+1 = f S m(ot+1; θm), zt ∼ gZ(st, at;φ). (1)
Following the IB principle, the objective of the DB model seeks to maximize the mutual information I(Zt;St+1) while minimizing the mutual information I([St, At];Zt). To this end, we propose the DB objective by following the IB Lagrangian [55], which takes the form of
min−I(Zt;St+1) + α1I([St, At];Zt). (2)
Here α1 is a Lagrange multiplier that quantifies the amount of information about the next state preserved in Zt. Fig. 1 illustrates the DB objective. We minimize I([St, At];Zt) and consider it as a regularizer in the representation learning. Then the representation learning is done by maximizing the mutual information I(Zt, St+1). Maximizing I(Zt, St+1) ensures that we do not discard useful information from (St, At). In DB, the mutual information is estimated by several variational bounds parameterized by neural networks to enable differentiable and tractable computations. In what follows, we propose a lower bound of (2), which we optimize to train the DB model.
3.1 Maximizing the lower bound of I(Zt;St+1)
As directly maximizing I(Zt;St+1) is intractable, we propose to optimize a predictive objective, which is a lower bound of I(Zt;St+1) [2]. It holds that
I(Zt;St+1) = Ep(zt,st+1) [
log p(st+1|zt) p(st+1) ] = E [ log
q(st+1|zt;ψ) p(st+1)
] +DKL[p(st+1|zt)‖q(st+1|zt;ψ)],
(3)
where p(st+1|zt) is an intractable conditional distribution and q(st+1|zt;ψ) is a tractable variational decoder with parameter ψ. By the non-negativity of the KL-divergence, we obtain the following lower bound,
I(Zt;St+1) ≥ Ep(zt,st+1)[log q(st+1|zt;ψ)] +H(St+1),
whereH(·) is the entropy. SinceH(St+1) is irrelevant to the parameter ψ, maximizing I(Zt;St+1) is equivalent to maximizing the following lower bound,
Ipred , Ep(zt,st+1)[log q(st+1|zt;ψ)]. (4)
Ipred can be interpreted as the log-likelihood of the next-state encoding st+1 given the dynamicsrelevant representation zt. In practice, we parameterize the prediction head q(st+1|zt;ψ) by a neural network that outputs a diagonal Gaussian random variable. Since st, st+1 and zt are all low-dimensional vectors instead of raw image pixels, optimizing Ipred is computationally efficient.
Momentum Encoder To encode the consecutive observations ot and ot+1, we adopt the Siamese architecture [10] that uses the same neural network structures for the two encoders. Nevertheless, we observe that if we train both the encoders by directly maximizing Ipred, the Siamese architecture tends to converge to a collapsed solution. That is, the generated encodings appears to be uninformative constants. A simple fact is that if both the encoders generate zero vectors, predicting zeros conditioning on zt (or any variables) is a trivial solution. To address such issue, we update the parameter θo of fSo in (1) by directly optimizing Ipred. Meanwhile, we update the parameter θm of fSm by a momentum moving average of θo, which takes the form of θm ← τθm + (1 − τ)θo. In the sequel, we call fSo the online encoder and f S m the momentum encoder, respectively. Similar techniques is also adopted in previous study [18, 46] to avoid the mode collapse.
3.2 Contrastive Objective for Maximizing I(Zt;St+1)
In addition to the lower bound of I(Zt;St+1) in §3.1, we also investigate the approach of maximizing the mutual information by contrastive learning (CL) [36]. CL classifies positive samples and negative samples in the learned representation space. An advantage of adopting CL is that training with negative samples plays the role of regularizer, which avoids collapsed solutions. Moreover, the contrastive objective yields a variational lower bound of the mutual information I(Z;St+1). To see such a fact, note that by the Bayes rule, we have
I(Zt;St+1) ≥ Ep(zt,st+1)ES− [ log exp(h(zt, st+1))∑
sj∈S−∪st+1 exp(h(zt, sj))
] , Ince. (5)
Here h is a score function which assigns high scores to positive pairs and low score to negative pairs. We refer to Appendix A for a detailed proof of (5). The right-hand side of (5) is known as
the InfoNCE objective [36]. The positive samples are obtained by directly sampling the transitions (s, a, s′). In contrast, the negative samples are obtained by first sampling a state-action pair (s, a), and then sampling a state s̃ independently. Then a negative sample is obtained by concatenating them together to form a tuple (s, a, s̃). The negative samples do not follow the transition dynamics. In practice, we collect the negative sample by sampling observation encodings randomly from the batch. We remark that comparing with methods that require data augmentation to construct negative samples [20, 49], DB utilizes a simple scheme to obtain positive and negative samples from on-policy experiences.
In (5), we adopt the standard bilinear function as the score function h, which is defined as follows,
h(zt, st+1) = f P o (q̄(zt;ψ)) >WfPm(st+1), (6)
where fPo (·;ϕo) and fPm(·;ϕm) project st+1 and the mean value of next-state prediction q(st+1|zt;ϕ), i.e., q̄(·;ψ), to a latent space to apply the contrastive loss Ince in (5), andW is the parameter of the score function. Similar to the observation encoder and MoCo-based architectures [49, 20, 15], we also adopt an online projector fPo and a momentum projector f P m for zt and st+1, respectively. The momentum projector is updated by ϕm ← τϕm + (1− τ)ϕo.
3.3 Minimizing the Upper Bound of I([St, At];Zt)
We minimize the mutual information I([St, At];Zt) through minimizing a tractable upper bound of the mutual information. To this end, we introduce a variational approximation q(zt) to the intractable marginal p(zt) = ∫ p(st, at)p(zt|st, at)dstat. Specifically, the following upper-bound of I([St, At];Zt) holds,
I([St, At];Zt) = Ep(st,at) [p(zt|st, at)
p(zt)
] = Ep(st,at) [p(zt|st, at) q(zt) ] −DKL [ p(zt)‖q(zt) ] ≤ Ep(st,at) [ DKL[p(zt|st, at)‖q(zt)] ] , Iupper,
(7)
where the inequality follows from the non-negativity of the KL divergence, and q(zt) is an approximation of the marginal distribution of Zt. We follow Alemi et al. [2] and use a standard spherical Gaussian distribution q(zt) = N (0, I) as the approximation. The expectation of Iupper is estimated by sampling from on-policy experiences.

heading: 3.4 The Loss Function and Architecture
text: The final loss for training the DB model is a combination of the upper and lower bounds established in previous sections,
min θo,φ,ψ,ϕo,W
LDB = α1Iupper − α2Ipred − α3Ince, (8)
where α1, α2 and α3 are hyper-parameters. As we show in the ablation study (§5), all of the three components in the loss plays an important role in learning dynamics-relevant representations. We illustrate the architecture of the DB model in Fig. 2. In practice, we minimize LDB in (8) by gradient descent, which iteratively updates the parameters of fSo , g
Z , qψ,W and fPo . Meanwhile, we adopt exponential moving average to update the parameters of fSm and f P m to avoid collapsed solutions. We refer to Appendix B for the pseudocode of training DB model.

heading: 4 Exploration with DB-Bonus
text: We are now ready to introduce the DB-bonus rdb for exploration. In this section, we first present the DB-bonus for self-supervised exploration. We establish the theoretical connections between the DB-bonus and provably efficient bonus functions. We further present the empirical estimation of DB-bonus and the policy optimization algorithm that utilizes the DB-bonus.
In the sequel, we assume that the learned parameter Θ of the DB model follows a Bayesian posterior distribution given the training dataset Dm = {(sit, ait, sit+1)}i∈[0,m], which is a collection of past experiences from m episodes performed by the agent to train the DB model. We aim to estimate the following conceptual reward, which is defined by the mutual information between the parameter of the DB model and the transition dynamics given the training dataset,
rdb(st, at) , I ( Θ; (st, at, St+1)|Dm )1/2 = [ H ( (st, at, St+1)|Dm ) −H ( (st, at, St+1)|Θ,Dm )]1/2 .
(9)
Intuitively, DB-bonus defined in (9) encourages the agent to explore transitions that are maximally informative to the improvement of the DB model.

heading: 4.1 Theoretical Analysis
text: We show that the DB-bonus defined in (9) enjoys well theoretical properties, and establish theoretical connections between rdb and bonuses based on the optimism in the face of uncertainty [4, 23], which incorporates UCB into value functions in both tabular [5, 22, 17] and linear MDPs [24, 13].
Connection to UCB-bonus in linear MDPs In linear MDPs, the transition kernel and reward function are assumed to be linear. In such a setting, LSVI-UCB [24] provably attains a near-optimal worst-case regret, and we refer to Appendix C.1 for the details. The idea of LSVI-UCB is using an optimistic Q-value, which is obtained by adding an UCB-bonus rucb [1] to the estimation of the Q-value. The UCB-bonus is defined as rucbt = β · [ η(st, at) >Λ−1t η(st, at) ]1/2
, where β is a constant, Λt = ∑m i=0 η(x i t, a i t)η(x i t, a i t) > + λ · I is the Gram matrix, and m is the index of the current episode. The UCB-bonus measures the epistemic uncertainty of the state-action and is provably efficient [24].
For linear MDPs, we consider representation z ∈ Rc as the mean of the posterior gZ from the DB model, and set zt to be a linear function of the state-action encoding, i.e., zt = Wtη(st, at) parameterized by Wt ∈ Rc×d. Then, the following theorem establishes a connection between the DB-bonus rdb and the UCB-bonus rucb.
Theorem 1. In linear MDPs, for tuning parameter β0 > 0, it holds that
β0/ √
2 · rucbt ≤ I(Wt; (st, at, St+1)|Dm) 1/2 ≤ β0 · rucbt , (10)
where I(Wt; (st, at, St+1)|Dm)1/2 is the DB-bonus rdb(st, at) under the linear MDP setting.
In addition, using rdb as bonus leads to the same regret as LSVI-UCB by following a similar proof to Jin et al. [24]. We refer to Appendix C.2 for the problem setup and the detailed proofs. We remark that Theorem 1 is an approximate derivation because we only consider the predictive objective Ipred in (8) in Theorem 1. Nevertheless, introducing the contrastive objective Ince is important in the training of the DB model as it prevents the mode collapse issue. Theorem 1 shows that the DB-bonus provides an instantiation of the UCB-bonus in DRL, which enables us to measure the epistemic uncertainty of high-dimensional states and actions without the linear MDP assumption.
Connection to visiting count in tabular MDP The following theorem establishes connections between DB-bonus and the count-based bonus rcount(st, at) = β√
Nst,at+λ in tabular MDPs.
Theorem 2. In tabular MDPs, it holds for the DB-bonus rdb(st, at) and the count-based intrinsic reward rcount(st, at) that,
rdb(st, at) ≈ √ |S|/2√
Nst,at + λ = β0 · rcount(st, at), (11)
when Nst,at is large, where λ > 0 is a tuning parameter, |S| is the number of states in tabular setting.
We refer to Appendix C.3 for a detailed proofs. As a result, DB-bonus can also be considered as a count-based intrinsic reward in the space of dynamics-relevant representations.

heading: 4.2 Empirical Estimation
text: To estimate such a bonus under our DB model, we face several challenges. (i) Firstly, estimating the bonus defined in (9) requires us to parameterize representation under a Bayesian learning framework, whereas our DB model is parameterized by non-Bayesian neural networks. (ii) Secondly, estimating the DB-bonus defined in (9) requires us to compute the mutual information between the unknown transitions and the estimated model, which is in general hard as we do not have access to such transitions in general. To address such challenges, we estimate a lower bound of the DB-bonus, which is easily implementable and achieves reasonable performance empirically. Specifically, we consider to use rdbl (st, at) as the lower bound of the information gain in (9),
rdb(st, at) ≥ [ H ( g(st, at, St+1)|Dm ) −H ( g(st, at, St+1)|Θ,Dm )]1/2 , rdbl (st, at), (12)
which holds for any mapping g according to Data Processing Inequality (DPI). DPI is an information theoretic concept that can be understood as ‘post-processing’ cannot increase information. Since g(st, at, St+1) is a post-processing of (st, at, St+1), we have I(Θ; (st, at, St+1)) > I(Θ; g(st, at, St+1)), where g is a neural network in practice. In our model, we adopt the following mapping, g(st, at, St+1)|Θ,Dm = gZ(st, at;φ), (13) where gZ is the representation distribution of DB, and φ constitutes a part of parameters of the total parameters Θ. Intuitively, since gZ is trained by IB principle to capture information of transitions, adopting the mapping gZ to (12) yields a reasonable approximation of the DB-bonus. It further holds
rdbl (st, at) = [ H ( gmargin ) −H ( gZ(st, at;φ) )]1/2 = EΘDKL [ gZ(zt|st, at;φ)‖gmargin ]1/2 , (14)
where we define gmargin = g(st, at, St+1)|Dm as the marginal of the encodings over the posterior of the parameters Θ of the DB model. In practice, since gmargin is intractable, we approximate gmargin with standard Gaussian distribution. We remark that such approximation is motivated by the training of DB model, which drives the marginal of representation gZ toward N (0, I) through minimizing Iupper in (7). Such approximation leads to a tractable estimation and stable empirical performances.
In addition, since we do not train the DB model with Bayesian approach, we replace the expectation over posterior Θ in (14) by the corresponding point estimation, namely the parameter Θ of the neural networks trained with DB model on the dataset Dm. To summarize, we utilize the following approximation of the DB-bonus rdb proposed in (9),
r̂dbl (st, at) = DKL [ gZ(·|st, at;φ) ‖ N (0, I) ]1/2 ≈ rdbl (st, at). (15) Since DB is trained by IB principle, which filters out the dynamics-irrelevant information, utilizing the bonus defined in (15) allows the agent to conduct robust exploration in noisy environments.
We summarize the the overall RL algorithm with self-supervised exploration induced by the DB-bonus in Algorithm 1, which we refer to as Self-Supervised Exploration with DB-bonus (SSE-DB). For the RL implementation, we adopt Proximal Policy Optimization (PPO) [45] with generalized advantage estimation [44] and the normalization schemes from Burda et al. [11]. We refer to Appendix D for the implementation details. The codes are available at https://github.com/Baichenjia/DB.
Algorithm 1 SSE-DB 1: Initialize: The DB model and the actor-critic network 2: for episode i = 1 to M do 3: for timestep i = 0 to T − 1 do 4: Obtain action from the actor at = π(st), then execute at and observe the state st+1; 5: Add (st, at, st+1) into the on-policy experiences; 6: Obtain the DB-bonus r̂dbl of (st, at) by (15); 7: end for 8: Update the actor and critic by PPO with the collected on-policy experiences as the input; 9: Update DB by gradient descent based on (8) with the collected on-policy experiences; 10: end for
100
200
300
400
500 Alien
200
400
600
800
Asteroids
0
200
400
600
800
BankHeist
−60
−40
−20
0
20
40
60
Boxing
0
50
100
150
200
250
300
Breakout
1000
1500
2000
2500
3000
3500
Centipede
5000
10000
15000
20000
25000
30000
35000 CrazyClimber
2000
4000
6000
8000
Gopher
0
50
100
150
200
250
300
350 Gravitar
100
200
300
400
500
600
700 Kangaroo
0
5000
10000
15000
20000
KungFuMaster
200
300
400
500
600 MsPacman
0 100 200 300 400 0
100
200
300
400
500
600
700 Seaquest
0 100 200 300 400
250
500
750
1000
1250
1500
1750
Solaris
0 100 200 300 400 −25
−20
−15
−10
−5
0
Tennis
0 100 200 300 400 1000
1500
2000
2500
3000
3500
4000 TimePilot
0 100 200 300 400
2500
5000
7500
10000
12500
15000
17500 UpNDown
0 100 200 300 400 400
600
800
1000
1200
1400
1600
1800 WizardOfWor
Frames (millions)
Ex tri
ns ic
Re wa
rd p
er E
pi so
de
SSE-DB(ours) ICM Disagreement CB random
Figure 3: The evaluation curve in Atari games. The different methods are trained with different intrinsic rewards. The extrinsic rewards are only used to measure the performance. Each method was run with three random seeds.

heading: 5 Experiments
text: We evaluate SSE-DB on Atari games. We conduct experiments to compare the following methods. (i) SSE-DB. The proposed method in Alg. 1. (ii) Intrinsic Curiosity Model (ICM) [39]. ICM uses an inverse dynamics model to extract features related to the actions. ICM further adopts the prediction error of dynamics as the intrinsic reward for exploration. (iii) Disagreement [40]. This method captures epistemic uncertainty by the disagreement among predictions from an ensemble of dynamics models. Disagreement performs competitive to ICM and RND [12]. Also, this method is robust to white-noise. (iv) Curiosity Bottleneck (CB) [26]. CB quantifies the compressiveness of observation with respect to the representation as the bonus. CB is originally proposed for exploration with extrinsic rewards. We adapt CB for self-supervised exploration by setting the extrinsic reward zero. We compare the model complexity of all the methods in Appendix D. Other methods including Novelty Search [53] and Contingency-aware exploration [16] are also deserve to compare. However, we find Novelty Search ineffective in our implementation since the detailed hyper-parameters and empirical results in Atari are not available. Contingency-aware exploration is related to DB while the attention module is relatively complicated and the code is not achievable.

heading: 5.1 The Main Results
text: We evaluate all methods on Atari games with high-dimensional observations. The selected 18 games are frequently used in previous approaches for efficient exploration. The overall results are provided in Fig. 3. We highlight that in our experiments, the agents are trained without accessing the extrinsic
rewards. The extrinsic rewards are ONLY utilized to evaluate the performance of the policies obtained from self-supervised exploration. Our experiments show that SSE-DB performs the best in 15 of 18 tasks, suggesting that dynamics-relevant feature together with DB-bonus helps the exploration of states with high extrinsic rewards.
In addition, since pure exploration without extrinsic rewards is very difficult in most tasks, a random baseline is required to show whether the exploration methods learn meaningful behaviors. We adopt the random score from DQN [35] and show the comparison in the figure. In Solaris, Centipede and TimePilot, our method obtains similar scores to random policy, which suggests that relying solely on intrinsic rewards is insufficient to solve these tasks. We also observe that SSE-DB is suboptimal in Tennis. A possible explanation is that for Tennis, the prediction error based methods, such as ICM, could capture additional information in the intrinsic rewards. For example, in Tennis, the prediction error becomes higher when the ball moves faster or when the agent hits the ball towards a tricky direction. The prediction-error based methods can naturally benefit from such nature of the game. In contrast, SSE-DB encourages exploration based on the information gain from learning the dynamics-relevant representation, which may not capture such critical events in Tennis.

heading: 5.2 Robustness in the Presence of Noises
text: Observation Noises. To analyze the robustness of SSE-DB to observation noises, an important evaluation metric is the performance of SSE-DB in the presence of dynamics-irrelevant information. A particularly challenging distractor is the white-noise [11, 26], which incorporates random taskirrelevant patterns to the observations. In such a scenario, a frequently visited state by injecting an unseen noise pattern may be mistakenly assigned with a high intrinsic reward by curiosity or pseudo-count based methods.
We use two types of distractors for the observations of Atari games, namely, (1) the random-box noise distractor, which places boxes filled with random Gaussian noise over the raw pixels, and (2) the pixellevel noise distractor, which adds pixel-wise Gaussian noise to the observations. Fig. 4 shows examples of the two types of distractors. In the sequel, we discuss results for the random-box noise distractor on selected Atari games, which we find sufficiently representative, and defer the complete report to Appendix E.1 and E.2.
Fig. 5(a) shows the performance of the compared methods on Alien, Breakout and TimePilot with and without noises. We observe that SSE-DB outperforms ICM on Alien and TimePilot with random-box noises. Nevertheless, in Breakout, we observe that both the methods fail to learn informative policies. A possible explanation is that, in Breakout, the ball is easily masked by the box-shaped noise (i.e., middle of Fig. 4). The random-box noise therefore buries critical transition information of the ball, which hinders all the baselines to extract dynamics-relevant information and leads to failures on Breakout with random-box noise as shown in Fig. 5(a).
Action Noise. In addition to observation noises, noises in actions also raise challenges for learning the transition dynamics. To further study the robustness of SSE-DB, we conduct experiments on Atari games with sticky actions [32, 40]. At each time step, the agent may execute the previous action instead of the output action of current policy with a probability of 0.25. We illustrate results on three selected Atari games, i.e., Boxing, Gravitar and MsPacman, in Fig. 5(b) and defer the complete results to Appendix E.3. Our experiments show that SSE-DB is robust to action noises, whereas ICM suffers significant performance drop from the action noise.

heading: 5.3 Visualization and Ablation Study
text: Visualization of the Learned Representations. To understand the latent representation learned by the DB model, we visualize the learned Z with t-SNE [31] plots, which projects a 128d z-vector to a 2d one through dimensionality reduction. We compare the representations learned by SSEDB and ICM with random-box noise. We illustrate the learned representations of MsPacman in Fig. 6. According to the visualization, representations learned by DB tends to align temporallyconsecutive movements on the same curve. Moreover, each segment of a curve corresponds to a semantic component of the trajectory, such as eating pellets aligned together or avoiding ghosts. The segments of a curve end up with critical states, including death and reborn of the agent and ghosts. The visualization indicates that DB well captures the dynamics-relevant information in the learned representations. In contrast, such temporally-consecutive patterns are missing in the learned representation of ICM.
Visualization of the DB bonus. We provide visualization of the DB-bonus in Appendix E.5. The results show the DB-bonus effectively encourages the agent to explore the informative transitions.
Ablation Study. The training of DB loss consists of multiple components, including Ipred, Ince, Iupper, and the momentum observation encoder. To analyze the importance of the components, we conduct an ablation study by removing each of them respectively and evaluate the DB model correspondingly. The ablation study suggests that the all the components are crucial for learning effective dynamics-relevant representations. In addition, we observe that Iupper is particularly important in the environment with dynamics-irrelevant noise. Please refer to Appendix E.4 for details.

heading: 6 Conclusion
text: In this paper, we introduce Dynamic Bottleneck model that learns dynamics-relevant representations based on the IB principle. Based on the DB model, we further propose DB-bonus based on the DB model for efficient exploration. We establish theoretical connections between the proposed DB-bonus and provably efficient bonuses. Our experiments show that SSE-DB outperforms several strong baselines in stochastic environments for self-supervised exploration. Moreover, we observe that DB learns well-structured representations and the DB-bonus characterizes informative transitions for exploration. For our future work, we wish to combine the DB representation to effective exploration methods including BeBold [61] and NGU [6] to enhance their robustness in stochastic environments.

heading: Acknowledgements
text: The authors thank Tencent Robotics X and Vector Institute for the computation resources supported. Part of the work was done during internship at Tencent Robotics X lab. The authors also thank the anonymous reviewers, whose invaluable suggestions have helped us to improve the paper.

Review summaries:
This paper propose an exploration method for reinforcement learning by introducing a dynamics bottleneck (DB) model that captures dynamics-revelant information from the states. To this end, they propose to maximize the mutual information between $Z_{t}$ and $S_{t+1}$ but minimize the mutual information between $Z_{t}$ and $(S_{t}, a_{t})$ to compress representation. Intrinsic reward based on learned DB-model. The proposed method is evaluated on tasks from Atari Learning Environment and corrupted Atari tasks.

In this paper, the authors introduced Dynamic Bottleneck (DB) model that learns dynamics-relevant representations based on the Information Bottleneck principle. They further proposed DB-bonus for efficient exploration and established theoretical connections between the proposed DB-bonus and provably efficient bonuses. The experiments show that the proposed method outperforms several strong baselines in stochastic environments for self-supervised exploration.

This paper proposes a new self-supervised exploration method for RL that encourages an agent to explore states which are more relevant in terms of information gain. To this aim, the Information-Bottleneck (IB) principle has been utilized to learn a model that can obtain dynamics-relevant information and discard dynamics-irrelevant features together. The paper evaluates the proposed technique in multiple Atari environments without accessing the extrinsic rewards.