Dynamic Bottleneck for Robust Self-Supervised Exploration
Authors: Chenjia Bai
Lingxiao Wang
Lei Han
Animesh Garg
Jianye HAO
Peng Liu
Zhaoran Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Dynamic Bottleneck
Information Bottleneck
Exploration
Reinforcement Learning

Abstract: Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics-irrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.

Dynamic Bottleneck for Robust Self-Supervised Exploration
Chenjia Bai Harbin Institute of Technology cii_cheejiasutu hit. edu. en
Lingxiao Wang
Northwestern University
Lei Han
Animesh Garg
Tencent Robotics X Ixhan@tencent.. com
University of Toronto, Vector Institute, NVIDIA gargScs toronto.
Jianye Hao
Peng Liu
Zhaoran Wang Northwestern University zhaoranvangogmail.c com
Tianjin University jianye. hao@tju edu cn
Harbin Institute of Technology pengliuthin edu. cn
Abstract
Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics- irrelevant information, e.g., white-noise. To handle such dynamics-irrelevan information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based the nntrmation-bottlenne principle. Based on the DB model, we further propose DB-bonus, which encourages the agent explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. evaluate the proposed method Atari suits with dynamics-inelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.
Introduction
The tradeoff between exploration and exploitation has long been a major challenge reinforce- ment learning (RI.) 35 50 58
Generally, excessive exploitation of the experience suffers from the potential risk of being suboptimal, whereas excessive exploration of novel states hinders the improvement of the policy. A straightforward way tackle the exploration-exploitaaion dilemma is to enhance exploration efficiency while keeping exploitation in pace. When the extrinsic rewards are dense, reward shaping is commonly adopted for efficient exploration. However, in many real-wort world applications such as autonomous driving (34), the extrinsic rewards are sparse, making efficient exploration challenging task in developing practical RL algorithms. The situations become even worse when the extrinsic rewards are entirely unavailable. In such scenario, the task of collecting informative trajectories from exploration is known the self-supervised exploration [[]]
An effective approach to self-supervised exploration is to design dense intrinsic reward that motivates the agent to explore novel transitions. Previous attempts include count-based [9] and curiosity-driven 39 explorations. The count-based exploration builds a density model to measure the pseudo-count of state visitation and assign high intrinsic rewards less frequently visited states. In contrast, the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

currosity- driven methods maintain a predictive model of the transitions and encourage the agent to visit transitions with high prediction errors. However, all these methods becomes unstable when the states are noisy, e.g., containing dynamics-irelleand information. For example, in autonomous driving tasks, the states captured by the camera may contain irrelevant objects, such as clouds that behave similar to Brownian movement. Hence, if we measure the novelty of states or the curiosity of transitions through raw observed pixels, exploration are likely to be affected by the dynamics of these irrelevant objects.
To encourage the agent to explore the most informative transitions of dynamics, we propose a Dynamic Bottleneck (DB) model, which generates a dynamics- representation Z, of the current action pair (S,, through the Information -ittaation-ootteneckk. (IB) principle [55]. The goal of training DB model is to acquire dynamics-relevan information and discard dynamics-irrelevat features simultaneously. To this end, we maximize the intormiinformationn T T( St-1) between a latent representation Ze and the next state St+1 through maximizing its lower bound and using contrastive learning. Meanwhile, we minimize the nutual-informattin T([S,, A,]; Z+) between the state-action pair and the corresponding representation to compress dynamics-irrelevan information. Based on our proposed DB model, we further construct DB-bonus for exploration. DB-bonus measures the novelty of state-action pairs by their information gain with respect to the representation computed from the DB model. We show that the DB-bonus are closely related to the provably efficient UCB-bonus in linear Markov Decision Processes (MDPs) [ and the visiting count in tabular MDPs 22]. We further estimate the DB-bonus by the learned dynamics-relevam representation from the DB model. We highlight that exploration based on DB DB-bonus directly utilize the information gain of the transitions, which filters out dlyamicc-irrelevve the Atari suit with dynamics-irrelevant noise injected. Results demonstrate that our proposed self- supervised exploration with DB-bonus is robust to dynamics-irrelevan noise and outperforms several state-of-the-art exploration methods.
noise. We conduct experiments on
2 Related Work
Our work is closely related to previous exploration algorithms that construct intrinsic rewards to quantify the novelty of states and transitions. Several early approaches directly define the pseudo- count by certain statistics to measure the novelty of states [0] more recent methods utilize density model 9 38 or hash map [52 42 for state statistics. Nevertheless, these approaches are easily affected by dynamics-irrelevan irrelevant information such as white-noise. The contingency awareness method [6] addresses such an issue by using an attentive model locate the agent and computes the pseudo- count based on regions around the agent. However, such an approach could ignore features that are distant from the agent but relevant to the transition dynamics. Another line of research measures the novelty through learning dynamics model and then use the prediction error to generate an intrinsic reward. These methods are known as the curiosity-driven exploration algorithms. Similar to the pscudo-count based methods, cifossitydrrvven methods become unstable in the presence of noises, because the prediction model is likely to yield high error for stochastic inputs or targets. Some recent attempts improve the curiosity-driven approach by learning the inverse dynamics [39] and variational dynamics 7 to define curiosity, or utilizes the prediction error of a random network to construct intrinsic rewards [12. However, without explicitly removing dynamics- inneevantn information, these methods are still vulnerable to noises in practice 11
The entropy-based exploration uses state entropy as the intrinsic reward. VISR 19. APT 22 and APS (2 use unsupervised skill discovery for fast task adaptation. In the unsupervised stage, they use d-nearest-neighher entropy estimator to measure the entropy of state, and then use it as the intrinsic reward. RE3 [47] and ProtoRL [59] use random encoder and prototypes to Iearn the representation and use state-entropy as bonuses in exploration. Nevertheless, the state entropy will increase significantly we inject noises in the state space. The entropy-basee exploration will be misled by the noises. Previous approaches also quantify the epistemic uncertainty of dynamics through Bayesian network bootstrapped Q-functions 37 g), ensemble dynamics [40] and Stein variational inference to tackle noisy environments. However, they typically require either complicated optimization methods or large networks. In contrast, DB learns a dynamics-relbe rellvvant representation and encourages exploration by directly accessing the information gain of new transitions via DB-bonus.
21,
43
Another closely related line of studies uses the mutual information to promote exploration in RI.. Novelty Search (NS) (53) proposes to learn a representation through IB. Curiosity Bottleneck

---

(CB) [66] also performs exploration based on IB by measuring the rele-aelevant novelty. However, both NS and CB require extrinsic rewards learn value function and are not applicable for self- supervised exploration. Moreover, NS contains additional k-nnarest-neighhor to generate intrinsic reward and representation loss to constrain the distance of consecutive states, which are costly for computation. In contrast, our DB model handles self-supervised exploration without accessing extrinsic rewards. EMI [25] learns a representation by maximizing the mutual information in the forward dynamics and the inverse dynamics, which is different from the IB principle used in our method. In addition, we aim to perform robust exploration to overcome the white-noise problem. while EMI does not have an explicit mechanism to address the noise.
Our work also related to representation learning RL. DrQ [60], RAD 12 and CURL 49 learn the state representation by data augmentation and contrastive learning [4 20 36 to improve the data-efficiency DRI. Deep InfoMax 33 and Self-Predictive Representation (SPR) [46] learn the
contrastive and predictive representations of dynamics, respectively, and utilize such representations
as auxiliary losses for policy optimization. However, none of these existing approaches extracts information that benefits exploration. In contrast, we show that the dynamics-relevant representation learned by DB can be utilized for efficient exploration.
3 The Dynamic Bottleneck
In this section, we introduce the objective function and architecture of the DB model. We consider an MDP that can be described by tuple (O,A, P, r). which consists of the observation space O, the action space A, the transition dynamics P., the reward function r, and the discount factor 7 E (0, 1). At each time step, agent decides to perform an action a, E A after observing oz E O, and then the observation transits to 0++1 with a reward r; received. In this paper, we use upper letters, such as O,, to denote random variables and the corresponding lower case letter, such as oz, to represent their corresponding realizations.
A..
We first briefly introduce the IB principle [55] In supervised setting that aims to learn a representation Z of a given input source X with the target source Y, IB maximizes the mutual information between Z and Y (i.e. max I (Z; Y)) and restricts the complexity of Z by using the constrain as I X) < I.. Combining the two terms, the objective of IB is equal to max I (2; - a (Z( with the introduction of Lagrange multiplier.
DB follows the IB principle [55) to learn dynamics-relevant representation. The input variable the DB model is tuple (Oz, A,) that contains the current observation and action, and the target is the next observation O++1- We denote by S, and St+1 the encoding of observations O and O.+1. The goal of the DB model is to obtain compressed latent representation Z, of (St. At). that preserves the information that is relevant to S only. Specifically, we use f and f as the encoders of two consecutive observations o and O++1, respectively. We parameterize the synamics-relevant representation 24 by a Gaussian distribution with parameter ø, and it takes (8t, a2) as input. We summarize the DB model as follows,
(or; Oo), S141 f f (0)41:0m)) ~ ( (st, Ø).
(1)
Following the IB principle, the objective of the DB model seeks to maximize the mutual information T(Z;: St+1) while minimizing the mutual information T[[S, Ar); 7,). To this end, we propose the DB objective by following the IB Lagrangian (55), which takes the form of
min - -I(e St-1) (I),
(2)
Here is Lagrange multiplier that quantifies the amount of information about the next state preserved in Z,. Fig. iillustrates the DB objective. We minimize I[[S,, A,]: Zz) and consider it as a regularizer in the representation learning. Then the representation learning is done by maximizing the mutual information / (Zz, 11 Maximizing [(zr, S.+1) ensures that we do not discard useful information from (St, At). In DB, the mutual information is estimated by several variational bounds parameterized by neural networks to enable differentiable and tractable computations. what follows, propose a lower bound of 2 which we optimize to train the DB model.
3

---

min [[[S,, Ar]; Z.)
St A
S
max [(25 S1+1)
Figure 1: Illustration of DB objective. DB minimizes the mutual information T[[S, Ad) Z,) to obtain compressive representation, and maximize [( S141) preserve the information.
3.1 Maximizing the lower bound of [Z;; S1+1)
As directly maximizing I[Z;; S(41) is intractable, we propose to optimize predictive objective, which is lower bound of (Z;; S(+1) 2 holds that
T(Z;; St-1) 1p( log p($1+1/2()
p(s1t1)
= E log g(sc+1|ze: [b]
+ D
p(s(+1)
where p($++1 (2e) is an intractable conditional distribution and g( v is tractable variational decoder with parameter v.. By the non-negativity of the KL-divergence, we obtain the following lower bound,
1z, S1+1) 2 (p(ee.ee))) [log ((14 |e; vb] H(S++));
where H(-) is the entropy. Since H((++1) irrelevant to the parameter vb, maximizing [(Z;; St+1) is equivalent to maximizing the following lower bound,
Ipred Ep(zase- ((+++)<);)))))
(4)
Ipred can be interpreted as the liceliidl of the next-state encoding 81+1 given the dynamics- relevant representation In practice, we parameterize the prediction head g($t+1 s/) by neural network that outputs a diagonal Gaussian random variable. Since 8,, 81411 and 2, are all low-dimensional vectors instead of raw image pixels, optimizing Ipred is computationally efficient.
Momentum Encoder To encode the consecutive observations o, and 0++1, we adopt the Siamese architecture [10] that uses the same neural network structures for the two encoders. Nevertheless, we observe that if we train both the encoders by directly maximizing pred, the Siamese architecture tends to converge to collapsed solution. That is, the generated encodings appears to be uninfor- mative constants. A simple fact that both the encoders generate zero vectors, predicting zeros conditioning
(or any variables) is trivial solution. To address such issue, update the parameter O,, of fe in (T) by directly optimizing /prea- Meanwhile, we update the parameter On of f by a momentum moving average of 0., which takes the form of 6.. < T0m + (1 - T)0.- In the sequel, we call fō the online encoder and f. the momentum encoder, respectively. Similar techniques is also adopted in previous study 18 46) to avoid the mode collapse.
3.2 Contrastive Objective for Maximizing I(Z,; S(+1)
In addition to the lower bound of (Zy; S2+1) in $3.1 we also investigate the approach of maximizing the mutual information by contrastive learning (CL) [3]. CL classifie positive samples and negative samples in the learned representation space. An advantage of adopting CL is that training with negative samples plays the role of regularizer, which avoids collapsed solutions. Moreover, the contrastive objective yields a variational lower bound of the mutual information ((2;8+11). To see such fact, that by the Bayes rule, we have
(xp(h(z,,8441)
I [Zz;S441 E Ep( log Σ u exp(h(2z,,s))
I Ince.
(5)
Here h is score function which assigns high scores to positive pairs and score to negative pairs. We refer to Appendix for detailed proof of (5). The right-hand side of (5) is known as

---

10+1
S1+1
Pm
Oboorvat trakee (CNN, SID0
Raproceiiicn Postnerr (PCN, Oaannar, 1260)
W
Dt
8t
24
00 (
Yo
fo
Upparticaal
Prodetko Haad (PCN, S000)
Pogacien Head (FO.
Caleaiizen
Figure 2: The network architecture of DB. The architecture contains several convolution neural networks (CNNs) and fully-connected networks (FCNs). It consists of four components, including (i) the observation encoder (fo f.m which consists of an online network for and momentum network fm (ii) the representation posterior g (s,, dzi Φ), which generates a Gaussian distribution for the input state action pair (St. 0z); (iii) the prediction head To ($1+1 24), which predicts the next state st+-1 based on sampling representation
from the representation posterior; and (iv) the projection heads (f% f.n), which maps the next observation encoding 8t+1 and its prediction from the prediction head to low-dimensional space to perform contrastive estimation.
the InfoNCE objective 336 The positive samples obtained by directly sampling the transitions (s, a, 8'). In contrast, the negative samples are obtained by first sampling state-action pair (8, a), and then sampling state & independently. Then negative sample is obtained by concatenating them together to form a tuple (s, o, 3). The negative samples do not follow the transition dynamics. In practice, we collect the negative sample by sampling observation encodings randomly from the batch. We remark that comparing with methods that require data augmentation to construct negative samples 20 49
DB utilizes a simple scheme to obtain positive and negative samples from on on-policy
experiences.
In 55) adopt the standard bilinear function the score function h, which is defined follows,
l12,,, 8141) 1e (g(2)) +) W (WFF((++)) (s++1),
where (-((::))) and (i (m)) project 9141 and the mean value of next-state prediction g(5++1 1 = sp). ie., [(: v)) to latent space to apply the contrastive loss Jnce in (5). and W is the parameter of the score function. Similar the observation encoder and MoCo-based architectures [49 20 [5]. we also adopt an online projector fo and momentum projector f for 2 and $1+1, respectively. The momentum projector updated by fm + Tm + - T)PP.
3.3 Minimizing the Upper Bound of /(S,, Ar]; Z.)
We minimize the mutual information Ar]; Z.) through minimizing tractable upper bound of the mutual information. To this end, we introduce a variational approximation ((<x) to the intractable marginal p(z+) p(s,, at (()pp(((e.... a de)ds, Specifically, the following upper-bound of I([S,, Z,) holds,
(p(zelse at) = Ep(erae) p(2))
9(2)) (( ( DKL [p(=))la(<z)]
I[[S,, Zt) = Ep(eraa)
Bp(er ac)
a:) ax)]]((z)))] ( I Jupper:
where the inequality follows from the non-negativity of the KL divergence, and ((xx) is approxi-
mation of the marginal distribution of Z.. We follow Alemi et al. [2] and use a standard spherical Gaussian distribution v(=)) (0, I) as the approximation. The expectation of Tupper is estimated by sampling from on-policy experiences.
3.4 The Loss Function and Architecture
The final loss for training the DB model combination of the upper and lower bounds established in previous sections,
min
LDB a Lupper a2 4pedd Cr3 Ince:
(8)
0 O.W.V.P...
5

---

where a1, d2 and ag are components in the loss plays an important role in learning dynamics- rellvant representations. We illustrate the architecture of the DB model in Fig- 2 In practice, we minimize LD in (8) by gradient descent, which iteratively updates the parameters of fo .g Ie: W and fo Meanwhile, we adopt exponential moving average to update the parameters of fm and fm. avoid collapsed solutions. We refer to Appendix B for the pseudocode of training DB model.
As we show in the ablation study ($5), all of the three
Exploration with DB-Bonus
We are now ready to introduce the DB-bonus ydb for exploration. In this section, we first present the DB-bonus for self-supervvsed exploration. We establish the theoretical connections between the DB-bonus and provably efficient bonus functions. We further present the empirical estimation of DB-bomus and the policy optimization algorithm that utilizes the DB-bonus.
In the sequel, assume that the learned parameter @ of the DB model follows Bayesian posterior distribution given the training dataset Dm = {(), at, 1)+1) ]ie which is a collection of past experiences from m episodes performed by the agent train the DB model. We aim to estimate the following conceptual reward, which is defined by the mutual information between the parameter of the DB model and the transition dynamics given the training dataset,
db (St, as) & (O; (st, az, S(+))lDm)) 1/2
H((5) o2 (Pm))(Pm) H (((s, at S4+1)19, Dm.) )mm))
1/2
Intuitively, DB-bonus defined in encourages the agent to explore transitions that are maximally informative the improvement the DB model.
4.1 Theoretical Analysis
We show that the DB-bonus defined in (9) enjoys well theoretical properties, and establish theoretical connections between dh and bonuses based on the optimism in the face of uncertainty 4 233. which
incorporates UCB into value functions in both tabular [5] 22 and linear MDPs 24 13
Connection UCB-bonus in linear MDPs In linear MDPs, the transition kernel and reward function are assumed to be linear. In such setting, LSVI-UCB (24) provably attains mear-optimal worst-case regret, and we refer Appendix C.. for the details. The idea of LSVI-UCB using an optimistic Q-value, which is obtained by adding an JCB-bonus y. uub [1] to the estimation of the where / a constant,
Q-value. The UCB-bonus is defined as ucb = 3- [n(s;, A, (n'nss, az)] 1/2
A, = > n at (1) a;) + A is the Gram matrix, and m the index of the current episode. The UCB-bonus measures the epistemic uncertainty of the state-action and is provably efficient [24).
For linear MDPs, we consider representation E IR as the mean of the posterior 92 from the DB model, and set 24 be linear function of the state-action encoding, i.e., 2 Win(s, az) parameterized by W. E Rexd Then, the following theorem establishes connection between the DB-bonus d and the UCB-bonus ueb.
Theorem 1. In linear MDPs, for tuning parameter 30 O, it holds that
Po/V2 r
rpc < I(W;; (st, ar S++1 (Dmm)) )/ < Bo ruub
(10)
where T(W; (St, 0e, (D,.) 2/2 is the DB-bonus (st, a.) under the linear MDP setting.
In addition, using db as bonus leads to the same regret as LSVI-UCB by following similar proof to Jin al. (24]. We refer Appendix C.2 the problem setup and the detailed proofs. We remark that Theorem is an approximate derivation because we only consider the predictive objective Ired (8) in Theorem
Nevertheless, introducing the contrastive objective Ince is important in the training of the DB model as it prevents the mode collapse issue. Theorem [hows that the DB-bonus provides an instantiation of the UCB-bonus in DRI., which enables us measure the epistemic uncertainty of nigh-dimmnsional states and actions without the linear MDP assumption.

---

Connection to visiting count in tabular MDP The following theorem establishes connections between DB-bonus and the count-based bonus yount (st, at)
a in tabular MDPs.
8
Theorem 2. In tabular MDPs, it holds for the DB-bonus r (st ar) and the count-based intrinsic reward coonnt (St, at) that,
VSS/22
p (st,
St. ai) VN +
Bu count (st, as),
(11)
when No,,a, is large, where À > 0 is a tuning parameter, (S) is the number of states in tabular setting.
We refer Appendix C.3 for detailed proofs. As result, DB-bonus can also be considered as
count-based intrinsic reward in the space of dynamics-relevant representations.
4.2 Empirical Estimation
To estimate such bonus under our DB model, we face several challenges. (i) Firstly, estimating the bonus defined in 9 requires us to parameterize representation under a Bayesian learning framework, whereas our DB model is parameterized by non-Bayesian neural networks. (ii) Secondly, estimating the DB-bonus defined in (9) requires us to compute the mutual information between the unknown transitions and the estimated model, which is in general hard as do not have access to such transitions in general. To address such challenges, we estimate a lower bound of the DB-bonus, which is easily implementable and achieves reasonable performance empirically. Specifically, we consider to use ri (St, as) as the lower bound of the information gain in (9)
(st, a.) 2 (o(ss, ae (.m)) - Hg(s,, ae 0041)10, 1/2 & r( (st, oz).
(12)
which holds for any mapping g according to Data Processing Inequality (DPI). DPI is an infor- mation theoretic concept that be understood "post-processing" cannot increase informa- tion. Since g8t, 0e, St+1) is post-processing of (s,,a, at, St+1), we have I(@; (st, Gt, (r+1)) > I(@: g(s,, at. Se+1)), where is neural network in practice. In our model, we adopt the following mapping.
(13) where g² is the representation distribution of DB, and á constitutes a part of parameters of the total parameters O. Intuitively, since g is trained by IB principle to capture information of transitions, adopting the mapping 9² to (1) yields reasonable approximation of the DB-bonus. It further holds
g(st, a, 11+1) (, Dm = g (s;
rd (Se, at) ( Hg ar: o))
1/2 Ee [9 (2e Se; oei margiin 1/2
(14)
where we define marginn = =g( a,, (Dm as the marginal of the encodings over the posterior of the parameters θ of the DB model. In practice, since mmargin is intractable, approximate margin with standard Gaussian distribution. We remark that such approximation is motivated by the training of DB model, which drives the marginal of representation g toward (0 1) through minimizing Tupper in 77. Such approximation leads to a tractable estimation and stable empirical performances.
In addition, since we do not train the DB model with Bayesian approach, we replace the expectation over posterior in (14) by the corresponding point estimation, namely the parameter θ of the neural networks trained with DB model on the dataset Dm. To summarize, we utilize the following approximation the DB-bonus db proposed (
pdl(ss, (st, ar) = DKL [g (+|s,, ax; Φ) || Il) = rgl (st, at).
(15)
Since DB is trained by IB principle, which filters out the dynamics- eynleicc-irrelevntt information, utilizing the bonus defined in (15) allows the agent to conduct robust exploration in noisy environments.
We summarize the the overall RL algorithm with self-supervise exploration induced by the DB-bonus Algorithm which we refer to as Self-Supervised Exploration with DB-bonus (SSE-DB). For the RL implementation, we adopt Proximal Policy Optimization (PPO) 45 with generalized advantage estimation and the normalization schemes from Burda et al. [1]. We refer to Appendix for the implementation details. The codes are available htttt:://ggthhteee com www.eaaaeeaeeeeeoooaa jaa/B
7

---

Algorithm SSE-DB
1: Initialize: The DB model and the actor-critic network
2: for episode i = to M do
3: for timestep = 0 to T 1 do
4: Obtain action from the actor O₃ = (st), then execute as and observe the state $1415
5: Add (St, de. $1+11 into the on-policy experiences;
6:
Obtain the DB-bonus f/th of (st. ae) by (15):
7:
end for
8:
Update the actor and critic by PPO with the collected on-policy experiences as the input;
9: Update DB by gradient descent based on ( with the collected on-policy experiences;
10: end for
Ossggeemmnt
Cord
3,00
Extri Rovard por Eptcods
Congrne
1000-
152
3423
1000
1000
300
Frames Omiillonss
Figure 3: The evaluation curve in Atari games. The different methods are trained with different intrinsic rewards. The extrinsic rewards only used measure the performance. Each method was run with three random seeds.
5 Experiments
We evaluate SSE-DB on Atari games. We conduct experiments to compare the following methods. (i) SSE-DB. The proposed method in Alg. [ (ii) Intrinsic Curiosity Model (ICM) [39]. ICM uses an inverse dynamics model to extract features related to the actions. ICM further adopts the prediction error of dynamics the intrinsic reward for exploration. (iii) Disagreement [40] This method captures epistemic uncertainty by the disagreement among predictions from an ensemble of dynamics models. Disagreement performs competitive ICM and RND [12] Also, this method is robust to white-noise. (iv) Curiosity Bottleneck (CB) (26). CB quantifies the compressiveness of observation with respect to the representation as the bonus. CB originally proposed for exploration with extrinsic rewards. We adapt CB for self-supervised exploration by setting the extrinsic reward zero. the methods Appendix [ Other methods including exploration [16] also deserve to compare. However, and awarre aware exploration is related to DB while the attention module is relatively complicated and the code not achievable.
compare the model complexity Novelty Search [53] and Contingency-awer
we find Novelty Search ineffective in our implementation since the detailed
empirical results in Atari not available. Contingeney-
5.1 The Main Results
We evaluate all methods on Atari games with high-dimensional observations. The selected 18 games are frequently used in previous approaches for efficient exploration. The overall results are provided in Fig. 3 We highlight that in our experiments, the agents are trained without accessing the extrinsic

---

(a) Atari with random- boxx noise
(b) Atari with sticky actions
Figure 5: A comparison selected Atari games with and without noises. (a) Random-boo noise. In Alien and TimePilot, SSE-DB is barely affected by random-box noise. Both methods fail Breakout. (b) Sticky actions. Illustration shows SSE-DB ís barely affected by the sticky actions.
rewards. The extrinsic rewards ONLY utilized to evaluate the performance of the policies obtained from self--uuprvvsed exploration. Our experiments show that SSE-DB performs the best in 15 of 18 tasks, suggesting that dynamies-relevant feature together with DB-bonus helps the exploration of states with high extrinsic rewards.
In addition, since pure exploration without extrinsic rewards is very difficult in most tasks, random baseline is required to show whether the exploration methods learn meaningful behaviors. We adopt the random score from DQN [35] and show the comparison in the figure. In Solaris, Centipede and TimePilot, our method obtains similar scores to random policy, which suggests that relying solely on intrinsic rewards is insufficient to solve these tasks. We also observe that SSE-DB is suboptimal in Tennis. A possible explanation is that for Tennis, the prediction error based methods, such as ICM, could capture additional information in the intrinsic rewards. For example, in Tennis, the prediction error becomes higher when the ball moves faster or when the agent hits the ball towards a tricky direction. The prediction-error based methods game. In contrast, SSE-DB encourages exploration based on the information gain from learning the dynamics-relevant representation, which may not capture such critical events Tennis.
naturally benefit from such nature of the
5.2 Robustness in the Presence of Noises
Observation Noises. To analyze the robustness of SSE-DB to observation noises, an important evaluation metric the performance SSE-DB the presence of dynamics-irreleann information. A particularly challenging distractor is the white-noise [6]. which incorporates random task- irrelevant patterns to the observations. In such a scenario, frequently visited state by injecting an unseen noise pattern may be mistakenly assigned with high intrinsic reward by curiosity or pseudo-count based methods.
We use two types of distractors for the observations of Atari games, namely, (1) the random-box noise distractor, which places boxes filled with random Gaussian noise over the raw pixels, and (2) the pixel- level noise distractor, which adds pixel-wise Gaus- sian noise to the observations. Fig. 4 shows examples of the two types of distractors. In the sequel, we discuss results for the random-box noise distractor on selected Atari games, which we find sufficiently representative, and defer the complete report to Ap-
B.D
D.D
Normal Observation
Random Bos Noise
Pixel Noise
Figure Observation of Breakout with no distractor (left), random- box noise distractor (middle), and pixel noise distractor (right).
pendix E and E.2
Fig. ((a) shows the performance of the compared methods on Alien, Breakout and TimePilot with and without noises. We observe that SSE-DB outperforms ICM on Alien and TimePilot with random- box noises. Nevertheless, in Breakout, we observe that both the methods fail to learn informative policies. possible explanation is that, in Breakout, the ball is easily masked by the box-shaped noise (i.e., middle of Fig- 4. The random-box noise therefore buries critical transition information of the ball, which hinders the baselines extract renemics-relevant information and leads to failures on Breakout with random-bor noise as shown in Fig. 5(a)

---

unn
ax
(a) DB representations with Random- Box
(b) ICM representations with Random- Box
Figure 6: Visualization of latent representations of MsPacman with random-box noise. The figures plot the representation of the same trajectory by DB (left) and ICM (right). Consecutive states are connected by the shaded lines. Numbers on the representations steps. Representation learned by DB tends to align consecutive states on the same curve.
the corresponding number of time
Action Noise. In addition to observation noises, noises in actions also raise challenges for learning the transition dynamics. To further study the robustness of SSE-DB, we conduct experiments on Atari games with sticky actions [32 [0]. At each time step, the agent may execute the previous action instead of the output action of current policy with probability of 0.25. We illustrate results on three selected Atari games, i... Boxing, Gravitar and MsPacman, Fig. 5(b) and defer the complete results to Appendix E.3 Our experiments show that SSE-DB is robust action noises, whereas ICM suffers significant performance drop from the action noise.
5.3 Visualization and Ablation Study
Visualization of the Learned Representations. To understand the latent representation learned by the DB model, we visualize the learned Z with t-SNE (31 plots, which projects a 128d z-vector to a 2d one through dimensionality reduction. We compare the representations learned by SSE- DB and ICM with random-box noise. We illustrate the learned representations of MsPacman in Fig. 6 According to the visualization, representations learned by DB tends to align temporally- consecutive movements on the same curve. Moreover, each segment of a curve corresponds to semantic component of the trajectory, such as eating pellets aligned together or avoiding ghosts. The segments of a curve end up with critical states, including death and reborn of the agent and ghosts. The visualization indicates that DB well captures the eyvamics-relevant information in the learned representations. In contrast, such representation of ICM.
consecutive patterns are missing in the learned
Visualization of the DB bonus. We provide visualization of the DB-bonus in Appendix E.5 The results show the DB-bonus effectively encourages the agent to explore the informative transitions.
Ablation Study. The training of DB loss consists of multiple components, including 4pred Iuce- Lupper. and the momentum observation encoder. To analyze the importance of the components, we conduct an ablation study by removing each of them respectively and evaluate the DB model correspondingly. The ablation study suggests that the all the components are crucial for learning effective dynamics-relevant representations. addition, we observe that Jupper is particularly important in the environment with canaamccs-irreeevee noise. Please refer to Appendix E.4 for details.
6 Conclusion
In this paper, we introduce Dynamic Bottleneck model that learns dynamics-relevan representations based on the IB principle. Based on the DB model, we further propose DB-bonus based on the DB model for efficient exploration. We establish theoretical connections between the proposed DB-bonus and provably efficient bonuses. Our experiments show that SSE-DB outperforms several strong baselines in stochastic environments for self-supervised exploration. Moreover, we observe that DB learns sell-structured representations and the DB-bonus characterizes informative transitions for exploration. For our future work, we wish to combine the DB representation to effective exploration methods including BeBold [6]] and NGU 6 to enhance their robustness in stochastic environments.
10

---

Acknowledgements
The authors thank Tencent Robotics and Vector Institute for the computation resources supported. Part of the work was done during internship at Tencent Robotics X lab. The authors also thank the anonymous reviewers, whose invaluable suggestions have helped us to improve the paper.
References
[1] Yasin Abbasi-Yadkonl Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in neural information processing systems, volume 24, pages 2312-2320, 2011.
[2] Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[3] Peter Auer. Using confidence bounds for exploitation-exxllraaion trade-offs. Journal of Machine Learning Research, 3(Nov).397-422, 2002.
[4] Peter Auer and Ronald Ortner. Logarithmic online regret bounds for undiscounted reinforcement learning. In Advances in Neural Information Processing Systems, pages 49 56, 2007.
[5] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272,
2017.
[6] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
[7] Chenjia Bai, Peng Liu, Zhaoran Wang, Kaiyu Liu, Lingxiao Wang, and Yingnan Zhao. Varia- tional dynamic for self-supervised exploration in deep reinforcement learning. arXiv preprint arXiv:2010.08755, 2020.
[8] Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference Machine Learning, pages 577-587. PMLR, 2021.
[9] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
[10] Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669-6) 688, 1993.
[11] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations, 2019.
[12] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019.
[13] Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint 27X1 2020.

---

[16] Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, aware exploration in reinforcement learning. In International
and Honglak Lee. Contingency- aware Conference on Learning Representations, 2019.
[17] Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforce- ment learning. In Advances in Neural Information Processing Systems, pages 2818-2826, 2015.
[18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
[19] Steven Hansen, Will Dabney, Andre Barreto, David Warle-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- tional Conference on Learning Representations, 2019.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[21] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1 1109-1111 2016.
[22] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal bounds for reinforcement learning. Journal of Machine Learning Research, 114A 2010.
[23] Chi Jin, Zeyuan Allen-Zhu Sebastien Bubeck, and Michael Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems, pages 4663,477, 2018.
[24] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137- 2143, 2020.
[25] Hyoungseok Kim, Jackyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference on Machine Learning, pages 3360-3368 2019.
[26] Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity- bottleneck: Exploration by distilling task-specific novelty. In International Conference Machine Learning, pages 2379-3388, 2019.
[27] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, In Advances in Neural Information Processing Systems, 2020.
[28] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736-6747. PMLR, 2021.
[29] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551 2021.
[30] Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in Neural Information Processing Systems, pages 206-214, 2012.
[31] Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, ((No)):2579 2008
[32] Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
12

---

[33] Bogdan Mazoure, Remi Tachet des Combes, Thang Long DOAN, Philip Bachman, and R De- von Hjelm. Deep reinforcement and infomax learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
[34] Rhiannon Michelmore, Marta Kwiatkowska, and Yarin Gal. Evaluating uncertainty quantifica- tion end-to-end autonomous driving control. arXiv preprint arXiv: 100::81110000 1.06817, 2018.
[35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 11:529 503,-33333000 2015.
[36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0000000748 2018.
[37] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pages 4026-4034, 2016.
[38] Georg Ostrovski, Mare G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pages
2017.
[39] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction. In International Conference on Machine Learning, pages 2017.
[40] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International Conference Machine Learning. pages 5062-5071, 2019
[41] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. In International conference on Machine learning, pages 697-704, 2006.
[42] Tabish Rashid, Bei Peng. Wendelin Bochmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations, 2020.
[43] Neale Ratzlaff, Qinxun Bai, Li Fuxin, and Wei Xu. Implicit generative modeling for efficient exploration. In International Conference on Machine Learning, pages 7985-7995. PMLR, 2020.
[44] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
2016.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[46] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In Interna- lional Conference on Learning Representations, 2021.
[47] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 9443-9454, 2021.
[48] Ravid Shwartz- and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1703.0000,, 2017.
[49] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised rep- resentations for reinforcement learning. In International Conference on Machine Learning, 2020.
13

---

[50] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[51] Adrien Ali Taiga, William Fedus, Marlos Machado, Aaron Courville, and Mare G Bellemare. On bonus based exploration methods in the arcade learning environment. In International Conference on Learning Representations, 2020.
[52] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurrk, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 2753-2762, 2017.
[53] Ruo Yu Tao, Vincent François-Lavel, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. In Advances in Neural Information Processing Systems, volume 33, 2020.
[54] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
[55] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[56] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. In Advances in Neural Information Processing Systems, 2021.
[57] Mike West. Outlier models and prior distributions in bayesian linear regression. Journal of the Royal Statistical Society: Series (Methodological), 46(3):431 439, 1984.
[58] Tianpei Yang, Hongyao Tang. Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, and Peng Liu. Exploration in deep reinforcement learning: A comprehensive survey. arXiv preprint arXiv:2109. 06668, 2021.
[59] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In Proceedings of the 38th International Conference Machine Learning, volume 139, pages 11920-1193 11931, 2021.
[60] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[61] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Bebold: Exploration beyond the boundary of explored regions. arXiv preprint arXiv:20/2.06611, 2020.
14


Reviews:
[{"review_id": "v3gi7OnMPEc", "reviewer": "Reviewer_L2WS", "summary": "In this paper, the authors introduced Dynamic Bottleneck (DB) model that learns dynamics-relevant representations based on the Information Bottleneck principle. They further proposed DB-bonus for efficient exploration and established theoretical connections between the proposed DB-bonus and provably efficient bonuses. The experiments show that the proposed method outperforms several strong baselines in stochastic environments for self-supervised exploration.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The information-based DB method is novel in combating noise in observation. \n2. The variational methods help to scale the method up to solve high dimensional problems.\n3. The authors show the link of the DB bonus to two provably efficient cases: linear MDP and tabular MDP.\n4. The ablation studies are sufficient that unveil interesting patterns latent space.\n\nCons:\n1. I am a little bit confused about the definition of “noisy states”. In the introduction, the authors give an example: “For example, in autonomous driving tasks, the states captured by the camera may contain irrelevant objects, such as clouds, birds, and aircraft.” However, most of these objectives may would have their own dynamics, it is just those dynamics are irrelevant to rewards. Could the proposed method be able to handle disturbance with dynamics? If not, I suggest change this example and make it clear in the paper. If so, then maybe add experiments in this case.\n\nTypos:\n1. The “log” operators are missing in Eq. 7. \n2. There are some obvious grammar errors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "j4Km1ltV9Ty", "reviewer": "Reviewer_LMHD", "summary": "This paper propose an exploration method for reinforcement learning by introducing a dynamics bottleneck (DB) model that captures dynamics-revelant information from the states. To this end, they propose to maximize the mutual information between $Z_{t}$ and $S_{t+1}$ but minimize the mutual information between $Z_{t}$ and $(S_{t}, a_{t})$ to compress representation. Intrinsic reward based on learned DB-model. The proposed method is evaluated on tasks from Atari Learning Environment and corrupted Atari tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Proposed method is not surprising, but seems novel and reasonable, writing is clear, and its effectiveness is supported by extensive experimental results. I like the paper in general, but I have some minor concerns that could be addressed easily, and i'm willing to increase the score when the concerns below are resolved:\n\n- Positioning of the proposed method in the paragraph 2 of introduction is a bit questionable. For example, representations learned by inverse dynamics model you mentioned as 'curiosity-driven explorations' should capture controllable features (relevant discussions are in [Pathak'17; Badia'20]), which in principle should not be affected by dynamics irrelevant objects like birds. While this aspect ​of capturing dynamics-relevant information is the very important characteristic of the proposed method and the method could be very good at doing it, it could not be its uniqueness compared to other related works. Could you elaborate on this and provide more intuitive explanation and supporting evidence why the proposed method should be better than the previous works?\n\n- While I appreciate that experimental results are extensive, but important and more recent baselines for self-supervised exploration are missing, like VISR [Hansen'20], APT [Liu'21a], Plan2Explore [Sekar'20], which were all available at the time of submission (via conference, arXiv, OpenReview). As a note, much more recent works like APS [Liu'21b], RE3 [Seo'21], ProtoRL [Denis'21] are also relevant.\n\n- How are hyperparameters ($\\alpha_1, \\alpha_2, \\alpha_3$) are selected? Is the proposed method sensitive to the choice of these hyperparameters?\n\nMinor\n- line 38: discards -> discard\n- line 137: plays\n- line 292: observer - > observe\n\n[Pathak'17] Curiosity-driven Exploration by Self-supervised Prediction\n\n[Sekar'20] Planning to Explore via Self-Supervised World Models\n\n[Hansen'20]  Fast Task Inference with Variational Intrinsic Successor Features\n\n[Liu'21a] Behavior From the Void: Unsupervised Active Pre-Training\n\n[Liu'21b] APS: Active Pretraining with Successor Features\n\n[Seo'21] State Entropy Maximization with Random Encoders for Efficient Exploration\n\n[Yarats'21] Reinforcement Learning with Prototypical Representations", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The draft does not address limitations and potential negative societal impact in the main text. They should include the relevant discussion in the main text following the suggestion of NeurIPS conference.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "YYI28_bgXx", "reviewer": "Reviewer_1fqw", "summary": "This paper proposes a new self-supervised exploration method for RL that encourages an agent to explore states which are more relevant in terms of information gain. To this aim, the Information-Bottleneck (IB) principle has been utilized to learn a model that can obtain dynamics-relevant information and discard dynamics-irrelevant features together. The paper evaluates the proposed technique in multiple Atari environments without accessing the extrinsic rewards. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the main idea of the paper is interesting and it is well explained and well written. In addition, the experiments in this paper support the main claim of the paper and show better performance than some of the curiosity-driven methods. \n\n1) While the idea of the paper seems promising, utilizing information-bottleneck for exploration has been well-studied and it seems this paper has a good overlap with previous works such as [1] and [2]. The main difference that I see between this work and previous works is using contrastive learning. Is it fair? or are there many more differences between them?  \n\n2) It is not clear to me why I([St, At]; Zt) should be minimized rather than maximized? Since the main claim of the paper is to acquire dynamics-relevant information and discard dynamics-irrelevant, why does minimizing I([St, At]; Zt) not result in discarding \"relevant\" information? I might miss something here, but can the authors elaborate on this?\n\n3) There'd have been interesting if the authors had included experiments in which reward = DB-Bonus + extrinsic-reward for a game like Montezuma’s Revenge. The reason I mention this is because if combining these two works, it further supports the claim of this paper (i.e. acquiring dynamics-relevant information and discard irrelevant ones).\n\n4) It is not clear how negative samples are collected for Eq 5? What does \"sampling observation encodings randomly from the batch\" [line 143] mean? Don't we do the same thing for regular samples ( i.e. positive ones)? \n\n5) Is it fair to say that Theory 1 is not that useful and relevant as it only holds without the contrastive objective? as such it shouldn't be claimed as a contribution in this paper.\n\n6) SSE-DB seems only better than 12-13 tasks, not 15 tasks (Not better in Tennis, Crazy Climber, Centipede, Time Plot, MasPAcman). Refer to results in Figure 3. Is that right?\n\nMinor comments:\n- Explain the information-bottleneck principle in the paper or at least in the appendix. The paper should be self-contained.\n\n- Explain what is \"data processing inequality\" in line 222.\n\n- What is \\mu in line 191?\n\n- Use a different notation for variational approximation r(z_t), it can be mistaken with reward function.\n\n\n[1] Curiosity-Bottleneck: Exploration By Distilling Task-Specific Novelty, Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, Gunhee Kim\n\n [2] EMI: Exploration with Mutual Information \nHyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, Hyun Oh Song", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper discusses a new method to encourage an RL agent to explore state-action pairs with high information gain which doesn't have a social impact. As such, I don't think this work has any negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "T7xOpfstyhn", "reviewer": "Reviewer_sHR8", "summary": "The authors introduce the dynamics bottleneck which uses the information bottleneck to discover the novelty of state-action pairs to introduce a reward bonus, used for exploration in RL tasks. The authors evaluate their method on top of PPO in the Atari domain. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Overall, the empirical results look good compared to related approaches.\n- Theoretical results drawing the connection between the proposed approach and different bonuses in the exploration literature shows this approach is reasonable. \n- I thought the supplementary was very complete, reproducibility of the paper is very high. Supplementary includes an ablation study. \n\nWeaknesses:\n\n- For a paper that aims to tackle the task of exploration, I thought the choice of environments was a bit lacking in specifically hard exploration tasks such as Montezuma's revenge or modified sparse reward environments. For example [1] lists (Gravitar, Montezuma’s Revenge, Pitfall!, Private Eye, Solaris, and\nVenture) as hard exploration tasks. Out of these 6, the 2 in tested in this paper (Solaris and Gravitar) have mostly flat learning curves. Compared to the learning curves from Dopamine [2], the performance of SSE-DB doesn't seem to be much better than the randomly initialized policy on these tasks. ~~According to the PPO paper [3], on Gravitar (Solaris results were not included in this paper), vanilla PPO can achieve a performance of 500-750+ after 40M frames, which is than SSE-DB in fewer frames.~~ (Edit: not a valid concern)\n- Clarity on novelty. One thing that wasn't too clear to me after the reading the paper is what of the approach was novel and what was based on prior work. For example 3.1 and 3.3 are also described in the CB paper [4] (with references to prior work), so it isn't clear to me if there is novelty here, or if the novelty was limited to viewing the information bottleneck over the dynamics. A similar concern for 3.2 as (as mentioned in the related work) there has been literature on contrastive learning in RL. \n\nMinor:\n- In the section 5.2 on the experiment in the presence of noise, is SSE-DB was only using the learned reward bonus, or is the learned representation used in some more meaningful way? \n\nReferences:\n- [1] Burda, Yuri, et al. \"Exploration by random network distillation.\" 2018.\n- [2] Castro, Pablo Samuel, et al. \"Dopamine: A research framework for deep reinforcement learning.\" 2018.\n- [3] Schulman, John, et al. \"Proximal policy optimization algorithms.\" 2017.\n- [4] Kim, Youngjin, et al. \"Curiosity-bottleneck: Exploration by distilling task-specific novelty.\" 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is addressed. Although I would encourage the authors to move their comments from the checklist to either the main paper or the supplementary material. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper presents a method for intrinsic exploration based on learning a representation of observation that can ignore distractors using the dynamics bottleneck principle. Optimality of the exploration bonus in the case of bandits is established. Results on standard benchmarks used in the exploration literature are convincing. Reviewers unanimously vote to accept the paper, which I agree with. 
Decision: Accept (Poster)