DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
Authors: Jiameng Fan
Wenchao Li
Venue: ICLR 2022 Submitted (2022)
Keywords: Representation Learning
Deep Reinforcement Learning
Information Bottleneck

Abstract: Deep reinforcement learning (DRL) agents are often sensitive to visual changes that were unseen in their training environments. To address this problem, we leverage the sequential nature of RL to learn robust representations that encode only task-relevant information from observations based on the unsupervised multi-view setting. Specifically, we introduce a novel contrastive version of Multi-View Information Bottleneck (MIB) objective for temporal data. We train RL agents from pixels with this auxiliary objective to learn robust representations that can compress away task-irrelevant information and are predictive of task-relevant dynamics. This approach enables us to train high-performance policies that are robust to visual distractions and can generalize well to unseen environments. We demonstrate that our approach can achieve SOTA performance on diverse visual control tasks on the DeepMind Control Suite when the background is replaced with natural videos. In addition, we show that our approach outperforms well-established baselines for generalization to unseen environments on the Procgen benchmark.

Under review as a conference paper at ICLR 2022
DRIBO: ROBUST DEEP REINEORCEMENT LEARNING VIA MULTI-VILE INFORMATION BOTTLENECK
Anonymous authors Paper under double-bling review
ABSTRACT
Deep reinforcement learning (DRL) agents are often sensitive to visual changes that were unseen in their training environments. To address this problem, we leverage the sequential nature of RL to learn robust representations that encode only task-relevant information from observations based on the unsupervised multi- view setting. Specifically, we introduce a novel contrastive version of Multi- Information Bottleneck (MIB) objective for temporal data. We train RL agents from pixels with this auxiliary objective to learn robust representations that can compress away irreeirrelevnnt information and are predictive of task-relevan dynamics. This approach enables to train nigh-performance policies that robust to visual distractions and can generalize well to unseen environments. We demonstrate that our approach can achieve SOTA performance diverse visual control tasks the DeepMind Control Suite when the background is replaced with natural videos. In addition, we show that our approach outperforms well-established baselines for generalization to unseen environments on the Procgen benchmark.
INTRODUCTION
Deep reinforcement learning (DRL) methods have been shown to be successful in learning high- quality controllers directly from raw images in an end-to-end fashion (Mnih et al., 2015; Levine et al., 2016; Bojarski et al., 2016). However, it has been observed that DRL agents perform poorly in environments different from those where the agents were trained on, even when these environments contain semantically equivalent information relevant to the control task (Farebrother et al., 2018; Cobbe et al., 2019; Zhang et al., 2018b;a; Yu et al., 2019). By contrast, humans routinely adapt to new, unseen environments. For example, while visual scenes can be drastically different when driving in different cities, human drivers can quickly adjust to driving in a new city which they have never visited. We argue that this ability to adapt stems from the fact that driving skills are invariant to many visual details that are actually not relevant to driving. Conversely, DRL agents without this ability are hindered from understanding the temporal structure of task- relevant dynamics without being distracted by lask-irrelevant visual details (Jonschkowski & Brock, 2015; Zhang et al., 2021; Agarwal al., 2021; Lee et al., 2020b).
Viewing from a representation learning perspective, a desired representation for RL should facilitate the prediction of future states (beyond expected rewards) on potential actions and discard excessive, lask-irrelevant information from visual observations, An RL agent that learns from such represen- tations has the advantage of learning an optimal policy more easily upon the prediction and being more robust to visual changes. In addition, the resulting policy is more likely to generalize to unseen environments the task-relevan information in the new environment remains similar to that in the training environments. Prior works (Hafner et al., 2019; Lee et al., 2020a) that encode images into low-dimensional latent space for RL typically rely on a reconstruction loss to learn representations that are sufficient to reconstruct the input images and predict ahead in the latent space While these approaches can learn representations that retain information in the visual observations, they do nothing to discard the irrelevant information.
We tackle this problem by considering state representations for RL that are robust under the multi-view setting (Li et al., 2018; Federici et al., 2020; Fischer, 2020), where each view is assumed provide the same amount of task-relevant information while all the information not shared by them is deemed task-irrelevant. Data augmentation can be easily leveraged generate such multi-view observations

---

Under review as a conference paper at ICLR 2022
Sequential Obvervations
Reinforcement Learning Objective
Viner by
MultiView Information
Bottleneck
Figure 1: Robust Deep Reinforcement Learning via View Infomration BOuteenek (DRIBO) incorporates the inherent temporal structure of RL and unsupervised multi- settings into robust representation learning in RL. We consider sequential multi-view observations, o 01:T
(1) and o of of original observations 01.7 sharing the same task-relevant information while any information not shared by them are lask-irrelevant. For example, natural video backgrounds of sequential observations are task- iiss-irrelevant and can be drastically different between training and testing environments. DRIBO
uses multi-view information bottleneck loss to ensure that $1:T
(2) (1) and $1: the representations of
multi-view observations, shares maximal relevatl information while reducing the task-irrelevant
information. DRIBO trains the RL policy and (or) value function top of the encoder.
without requiring additional new data. Data augmentation in RL has delivered promising results for visual control tasks (Laskin et al., 2020b; Lange et al., 2012; Laskin et al., 2020a). However, these methods rarely exploit the sequential aspect of RL which requires an ideal representation to be predictive of future states given actions. In fact, the sequential nature of RL provides an additional temporal dimension for identifying fask-irrelevant information when it is independent of actions. Instead of learning representations from each visual observation (Laskin et al., 2020a), we propose to learn a predictive model that captures the temporal evolution of representations from a sequence of observations and actions. Concretely, we introduce new multi-view information bottleneck (MIB) objective that maximizes the mutual information between sequences of observations and representations while reducing the lusk-irrelevant information identified from the multi-view ob- servations. We incorporate this MIB objective into RL by using it as auxiliary learning objective. We illustrate our approach in Figure 1. Our contributions are summarized below.
We propose DRIBO, a novel technique that learns robust representations in RL by identifying and discarding task-irrelevant information in the representations based on MIB.
We leverage the sequential nature of RL to learn representations better suited for RL with a non--rconsttuccion-biieee based, DRIBO loss that maximizes the mutual information between sequences observations and representations while disregarding task-irrelevan information.
Empirically, we show that our approach can (i) lead to better robustness against ieskvitreeevant distractors the DeepMind Control Suite and (ii) significantly improve generalization the Proegen benchmarks compared to current state-of-theearts.
2 RELATED WORK
Reconstruction-batt based Representation Learning. Early works first trained autoencoders to learn representations reconstruct observations. Then, the RI. agent was trained from the learned representations (Lange & Riedmiller, 2010; Lange et al., 2012). However, there is no guarantee that the agent will capture useful information for control. To address this problem, learning encoder and dynamics jointly has been proved effective in learning task-oriented and predictive representa- tions (Wahlström et al., 2015; Watter al., 2015). More recently, Hafner al. (2019; 2020; 2021) and Lee et al. (2020a) learn latent dynamics model and train RL agents with predictive latent representations. However, these approaches suffer from the problem of embedding all details into representations even when they are task-irrelevant. The reason is that improving reconstruction quality from representations to visual observations forces the representations to retain more details. Despite success on many benchmarks, task-irrelevan visual changes affect performance signifi- cantly (Zhang et al., 2018a). Experimentally, we show that our non-rcconstructive approach, DRIBO, is substantially more robust against visual changes than prior works. We also compare DRIBO with another non rononsscnsre method, DBC (Zhang et al., 2021), which uses bisimulation metrics to leam representations in RL that contain only task-relevant information.
Contrastive Representations Learning. Contrastive representation learning methods train an en- coder that obeys similarity constraints in dataset typically organized by similar and dissimilar
2

---

Under review as a conference paper at ICLR 2022
pairs. The similar examples are typically obtained from nearby image patches (Oord et al., 2018; Hénaff et al., 2020) or through data augmentation (Chen et al., 2020). A scoring function that dower-bounds mutual information is one of the typical objects to be maximized (Belghazi et al., 2018; Oord et al., 2018; Hjelm et al., 2019; Poole et al., 2019). A number of works have applied the above ideas to RI. settings to extract predictive signals. HMI (Kim et al., 2019) applies Lensen-Shannon divergence-base baseed lower bound on mutual information across subsequent frames as an exploration bonus. DRIML (Mazoure et al., 2020) uses an auxiliary contrastive objective to maximize concor- dance between representations to increase predictive properties of the representations conditioned on actions. CURL (Laskin et al., 2020a) incorporates contrastive learning into RL algorithms to maximize similarity between augmented versions of the same observation. However, solely maxi- mizing the lower-bound of mutual information retains all the information including those that are task-irrelevant (Federici et al., 2020; Fischer, 2020).
Multi- View Information Bottleneck (MIB). MVRL (Li et al., 2019) uses the multi-view setting to tackle partially observable Markov decision processes with more than one observation model. For classification tasks, Federici et al. (2020) uses MIB by maximizing the mutual information between the representations of the two views while at the same time eliminating the tabel-irrelevant information identified by multi-view observations. Fischer (2020) describes a variant of the Conditional Entropy Bottleneck (CEB) which is mathematically equivalent to MIB. However, MIB/CEB cannot be directly used in RI. settings due to the sequential nature of decision making problems. PI-SAC (Leee et al., 2020b) uses contrastive version of CEB to model Predictive Information (Bialek & Tishby, 1999) which is the mutual information between the past and the future to solve RL problems. However, this approach does not scale long sequential data in RL and in practice only models short-term Predictive Information. Task-relevant information in RL is relevant because they influence not only current control decision and reward but also states and rewards well into the future. Our work, DRIBO, learns robust representations with predictive model to maximize the mutual information between sequences of representations and observations, while eliminating task-irrelevant information based on the information bottleneck principle. Learning a predictive model also adopts richer learning signals than those provided by individual observation and reward alone. Philosophically and technically, our approach is different from PI-SAC which does not quantify task-irrelevant information from multi-view observations and cannot capture long-term dependencies. Another line of work, IDAAC (Raileanu & Fergus, 2021), leverages an adversarial framework that the learned representations yield features that are instance-independent and invariant to task-irrelevant changes.
3 PRELIMINARIES
We denote Markov decision process (MDP) as M, with state action and reward r. S and A stand for the corresponding random variables. We denote a policy on M as æ. The agent's goal is to learn policy that maximizes the cumulative rewards. We define SCR" as the staterrppresentation space. The visual observations are oEO, where we denote multi-view observations from the viewpoint i as ()) stands for the random variable of the observation. We introduce a multi-view trajectory (i)
(i) - oi .a,,..., ST, of ar) where T is the length. Knowing that the trajectory density is defined over joint observations, states, and actions, we write:
Pr (TM)) P
obs (OT ST) P ST (T ar ... (s1 obs (o (1 Po((s)) (I)
with Po being the initial state distribution, P being the transition model and P being the unknown observation model for view i. DRL agents learn from visual observations by treating consecutive ob- servations as states to implicitly capture the predictive property. However, rich details in observations can easily distract the agent. An ideal representation should contain no task-irrelevant information and satisfy some underlying MDP which determines the distribution of the multi-view trajectory in Eq. (I). Thus, instead of mapping single-step observation to a representation, we consider learning a predictive model that correlates sequential observations and representations.
Let aj be the optimal action sequence for some O1.7 which is obtained by executing the action sequence a1:7. We assume that our contains enough information to obtain ai:r which maximizes the cumulative rewards. With this assumption, we define task-relevant information if it is necessary for deriving a... By contrast, task-irrelevant information does not contribute to the choice of aig- We first consider sufficient representations that are discriminative enough to obtain A each
3

---

Under review as a conference paper at ICLR 2022
timestep. This property can be quantified by the amount of mutual information between O1.7 and Air and mutual information between ST:T and Air
Definition 1. Representations S1:1 of O1:r are sufficient for RL iff I(0 -
RL agents that have access to a sufficient representation S, at timestep must be able to generate A as if it has access to the original observations. This can be better understood by subdividing I(OuT) SuT) into two components using the chain rule of mutual information:
I(Or:: SE.) I(S::r; O1:1(4i,)) I(S1:: Ai.r)
(2)
Conditional mutual information I(S O.TlAAir)) quantifies the information in SI:T that is task- irrelevant. I(S1:;; Air) quantifies task-relevart information that is accessible from S1.r. The last term is independent of the representation as long as St is sufficient for A (see Definition 1). Thus, representation contains minimal task-irrelevant information whenever I(O 1:r; b minimized. Maximizing I(O1.T; SI.)) learns sufficient representation. With the information bottleneck principle (Tishby et al., 2000), we can construct an objective to maximize I(O S1r) while minimizing Z (S1::; O1.7 (Ai,r) to compress away
information.
However, estimating the mutual information between long sequences is difficult due to the high dimensionality of the problem. In addition, the minimization of I(S O2:r (Ai,r) can only be done directly in supervised settings where Ai. observed. One option is to use MIB which can compress away task-irrelevant information the representations in unsupervised settings (Federici et al., 2020). The problem, however, is that MIB in its original form only considers single observation and its representation and thus does not guarantee that the learned representations retain the important temporal structure of RL. In the next section, we describe how we extend MIB RL settings.
4 DRIBO
DRIBO learns robust representations that are predictive of future representations while discarding task-irrelevant information. To learn such representations, we construct a new MIB objective that (i) reduces the problem of maximizing the mutual information between sequences of observations and representations to maximizing the mutual information between them at each timestep and (ii) quantifies the amount of task-irrelevan information in the representations in the multi- setting.
4.1 MUTUAL INFORMATION MAXIMIZATION
To capture the temporal evolution of observations and representations given any action sequence, consider maximizing the conditional mutual information I(S1:r; O1rr(ArT)) which is lower bound of I(O S1.)) (see Appendix 1) The observations OuT are generated sequentially in the environment by executing the actions ALTT The conditional mutual information not only estimates the sufficiency of the representations but also maintains the temporal structure of RL problems.
To tackle the challenges of estimating the mutual information between sequences, first factorize the mutual information between two sequential data to the mutual information at each timestep.
Theorem 1. Let Ovr be the sequential observations obtained by executing action sequence A If Si. is a sequence of sufficient representations for Ov.r, we have:
T
I(SI.;: OIT(AITT)))
I,S A(-1)
(3)
The proof included in Appendix A.I. Theorem shows that the sum of mutual informa- tion Z(S; 0, A, over multiple timesteps is a lower bound of I(S 01:rl A defined with conditional probabilities, p(s)|St-1, a:-1) and ((-, 0;-1) This lower bound models the dynamics and tempo- ral structure of RL since the first conditional probability is the composition of transition probability and observation model and the second conditional probability is the transition probability of the underlying MDP. Thus, the factorized mutual information explicitly retains the predictive property of representations. Even when the representations Sir are not sufficient, maximizing the mutual information between S, and O, (I(S; O, S,-1, A,-1) encourages encoding more detailed features from O into S, and makes S, sufficient.
I(S,; O,,,,,,,,,,,44 4,-1)
0( at-1), 1),
4

---

Under review as a conference paper at ICLR 2022
4.2 MULTI-VIEW SETTING
To learn sufficient representations with minimal task-irrelevant information, we consider a two-view setting to identify the task-irrelevant information without supervision. Consider of and of to be two visual images of the control scenario from two different viewpoints. Under the multi-view assumption, any representation 8 containing all information accessible from both views and is predictive of future representations would contain sufficient task-relevant information. Furthermore, if 8, captures only the details that visible from both observations, it would eliminate the view- specific details and reduce the sensitivity of the representation to view-changes.
sufficient representation in RL retains all the information that is shared by mutually redundant observations 0 and o We refer to Appendix A for the sufficiency condition of representations and mutually redundancy condition (Federici et al., 2020) between O!") and o,2² Intuitively, with the mutual redundancy condition, any representation that contains all the information shared by both views is task-relevant as the joint observation. By factorizing the mutual information between s(")
and 2 as Eq. (2), we can identify two components:
I o0 S 411) = I(S!") 11 A,-1, 0,-,,,,, :5 A2-1) (4)
Here, 1 is a representation of visual observation 0 Since we assume mutual redundancy between the two views, the information shared between 0( and 5(") conditioned of²) must be irrelevant to the task, which can be quantified as I(S{ : (((")s(()) Ac-1, )! (first term in Eq. (4)). Then, I(O, s(") - At-1) has to be maximal if the representation sufficient. formal description of the above statement can be found in Appendix A.
The less the two views have in common, the less task-irrelevant information can be encoded into the representations without violating sufficiency, and consequently, the less sensitive the resulting representation is task-irrelevant nuisances. In the extreme, s! is the underlying states of MDP if of and of share only task-relevant information. With Eq. (3) and 4, we have c(B to maximize the mutual information between representations and observations and compress away task-irrelevant in- formation [(O(2) ( 14-1) based on the information bottleneck principle. xı is a Lagrange multiplier. This loss also retains the temporally evolving information of the underlying dynamics.
c - :0 10. A-1, A,I(O SS A.-))
(5)
Symmetrically, we define a loss
((2) in for representations and observations from view 2:
c/ - - - "
(2) A.
(6)
The above losses extend MIB RL and minimizing learns representations that robust to task-irrelevant distractors and predictive of the future. The multi-view observations can be easily obtained with random data augmentation techniques so that each view augmented differently.
4.3 DRIBO LOss FUNCTION
By re- peraametterizing the Lagrangian multipliers (details in Appendix B), the average of two loss functions c( and 222) from two views at timestep & can be upper bounded as follows:
(S 1) (.
of " (11))()))lb0s (2) o! 3D (po (5)
(1)
a.1)))
where / denotes the parameters of an encoder p (1) 1o 81-1, a.-1) (details in Section 4.4),
8:-1 is sufficient representation, DSKL represents the symmetrized KL divergence obtained a.-1)),
by averaging the expected values of DKL "0o"", ,;"",, a (p
DRI |o{ [ a. (s) (o (1) 8 , a,-1)) 1)). and the coefficient 3 represents the trade- off between sufficiency and sensitivity to task-inrelevatt information. 33 is papereparameter.
5

---

Under review as a conference paper at ICLR 2022
Algorithm DRIBO Loss
1: Input: Batch B storing N sequential observations and actions with length T from replay buffer.
2: Apply random augmentation transformations on B to obtain multi-view batches ( and B
3: for i, (o1r 1.T 01.1) in enumerate (B(1) (B))) do
(2)
4: for L 1 to T do
po (8 (1) (1) (1)
0f 88- ar- 8 (2) po
(2) 88-11, at-1)
5:
8t
(8(1),44((111 (()))11)))) - (8) (1) ((())
6:
7: end for
8: C'KLL
$ ET= DoKL (s! 10!1) - a ap-1))lpo (s) (2)
(1)
- ai-1))
9:
end for
return -Tp(((s(W)) (tlos)),,,ss.))))3 s(O)))) NN Σ fisKL
10:
To generalize the above loss to sequential data in RL, we apply Theorem I to obtain the DRIBO
loss: -DRIBO+ Ei= (0; 33. We summarize the batch-based computation of the loss function in Algorithm 1. We sample sf and e( from (s (1) oo!")
(1) ax-1) and pa (s{ Oo (2) _ az-1) respectively. Though the first term in Eq. (7) is conditioned on st-1. we prove that the sampling process does not affect its effectiveness based on the multi-view assumption in Appendix B. The sym- metrized KL divergence term can computed from the probability density of s()) and s(a) using the encoder. The mutual information between the two representations : ( 1,
can be
maximized by using any sample-hased differentiable mutual information lower bound fz.(s!)) ,,(()) where de represents the learnable parameters. We use InfoNCE (Oord et al., 2018 8) to estimate mutual information since the multi-view setting provides a large number negative examples. The positive {
x(()),,
pairs are the representations (s (1) st ) of the multi-view observations generated from the same observation. The remaining pairs of representations within the same batch are used as negative pairs. The full derivation of the DRIBO loss function can be found in Appendix B.
4.4 ENCODER ARCHITECTURE AND INCORPORATING DRIBO IN RI.
The encoder po(8; Oz; 8t-1: 1, at-1) approximates the posterior representation given the current obser- vation, and representation and action from the previous timestep. The posteriors can also be viewed as a reparameterization of po( (11T 01:T, 01.r) II, Pe s Or. $;-1, at-1), which reflects the inherent temporal structure of RL. We implement the encoder as a recurrent space model (RSSM (Hafner et al., 2019)) which leverages recurrent neural networks to perform accurate long-term predictions. More details can be found in Appendix D. 1. Training an RSSM encoder with DRIBO enables the representations to predictive of future states.
We simultaneously train our representation learning models and the RL agent by adding Ldribo (Algorithm 1) as an auxiliary objective during training. The multi-view observations can be easily obtained using the same experience replay of RL agents through data augmentation. We demonstrate the effectiveness of DRIBO by building the agents on top of SAC (Haamoja et al., 2018), an off off-cy RL algorithm, and PPO (Schulman et al., 2017), an on on-ooiic RL algorithm, in Section 5.1 and Section 5.3 respectively. More details can be found Appendix D.
EXPERIMENTS
We experimentally evaluate DRIBO on a variety of visual control tasks. We designed the experiments compare DRIBO to the current best methods in the literature on: (i) the effectiveness of solving visual control tasks, (ii) their robustness against task-irrelevant distractors, and (iii) the ability to generalize to unseen environments. For effectiveness and robustness, we demonstrate DRIBO's perfor- mance on DeepMind Control Suite (DMC (Tassa et al., 2018)) with task-irrelevant visual distractors in backgrounds. The backgrounds are replaced with natural videos from the Kinetics dataset (Kay et al., 2017) (middle column in Figure 2). In Appedix C.I, we also show comparison between DRIBO and SOTAs on DMC environments without the background distractors (left column Figure 2).

---

Under review as a conference paper at ICLR 2022
For generalization, present results on Procgen (Cobbe et al., 2020) which provides different levels of the same game to test how well agents generalize unseen levels. We use single-step observations to train DRIBO without assuming that observation at each timestep provides full observability of the underlying dynamics. By contrast, current SOTA approaches require the use of consecutive observations to capture predictive properties of the underlying states.
We also conduct careful ablations analysis to show that the benefit of DRIBO is due primarily to learning from the tem- poral structure of RL and the DRIBO loss.
Figure 2: Left: DMC observations without visual distractors. Middle: ob- servations with natural videos as back- grounds. Right: spatial attention maps of encoders for the middle images.
For the DMC suite, all agents are built on top of SAC. For the Procgen suite, we augment PPO, a RL baseline for Procgen, with DRIBO. Implementation details are in Appendix D.
5.1 EFFECTIVENESS AND ROBUSTNESS
We compare DRIBO against several SOTA methods. The first is RAD (Laskin al., 2020b), recent method that uses augmented data to train pixel-based policies on DMC benchmarks. The second SLAAC (Leee et al., 2020a), a SOTA representation learning method for RI, that learns a dynamic model using a reconstruction loss. The third is CURL. (Laskin et al., 2020a), an approach that leverages contrastive learning to maximize the mutual information between representations of augmented versions of the same observation but does not distinguish between relevant and irrelevant features. The fourth is DBC (Zhang et al., 2021) which shares a similar goal with DRIBO. DBC learns invariant representation based on bisimulation metries without requiring reconstruction. Finally, we compare with PI-SAC (Lee et al., 2020b) which leverages the Predictive Information to compress away lask-irreeevant information with CEB. We apply random crop*grayscale to obtain the augmented data for RAD which achieves the best performance in (Laskin et al., 2020b). For CURL and DRIBO, apply random crop obtain augmented data and multi-view observations.
cartpole/swingup
cheetah/run
finger/spin
Average Returns
800
800
eno
300
600
400
200
400
200
100
hopper/hop
walker/walk
walker/run
Average Returns
400
150-
300
100
600
400
200
200
100
Environment Steps
165
Environment Steps
Environment Steps
105
DRIBO (Ours)
RAD
SLAC
CURL
DBC
PI-SAC
Figure 3: Results for DMC over seeds with one standard error shaded in the natural video setting.
Natural Video Setting. To investigate the effectiveness and robustness of RI. agents in DMC environments, we introduce high-dimensiooal visual distractors by using natural videos from the Kinetics dataset (Kay et al., 2017) backgrounds (Zhang et al., 2018a) (Figure 2: middle column). We use the class "arranging flowers" videos to replace the background in training. During testing, use the test set from the Kinetics dataset to replace the background, which contains videos from various different classes. Note that a single run of the DMC task may have multiple videos playing sequentially in the background. See Appendix C.4 for snapshots under the natural video setting.
'll other methods compared in this paper stack frames of consecutive observations.
7

---

Under review as a conference paper at ICLR 2022
In Figure 2, spatial attention maps (Zagoruyko & Komodakis, 2017) of the trained DRIBO encoder demonstrate that DRIBO trains agents to focus on the robot body while ignoring irrelevant scene details in the background. Figure shows that DRIBO performs substantially better than RAD, SLAC and CURL which do not discard task-irrelevant information explicitly. Compared to PI-SAC and DBC which are recent state-of-art methods aimed at learning representations invariant task-irrelevant information, DRIBO outperforms them consistently DRIBO achieves on average 30% and at the maximum 77% higher returns at 88e4 steps compared to the second best performing method.
Figure 4: t-SNE of latent spaces learned with DRIBO (left t-SNE) and CURL (right t-SNE). We color-code the embedded points with reward values (higher value yellow, lower value green). Each pair of solid lines indicates the corresponding embedded points for observations with identical foreground but different backgrounds. DRIBO learns representations that are neighboring in the embedding space with similar reward values. This property holds even if the backgrounds drastically different (see middle images). By contrast, CURL. maps the same image pairs to points far away from each other in the embedding space.
Visualization. We visualize the representations learned by DRIBO and CURL with t-SNE (Van der Maaten & Hinton, 2008). Figure 4 shows that even when the background looks drastically different, DRIBO learns to disregard irrelevant information and maps observations with similar robot configu- rations to the neighborhoods one another. The color code represents value the reward for each representation. The rewards can be viewed as task-relevant signals provided by the environments. DRIBO learns representations that are close in the latent space with similar reward values.
5.2 ABLATIONS
Temporal Structure of RL. To investigate whether DRIBO captures the temporal structure of RL, conducted further experiments on DRIBO agents trained using sequences of different lengths under the natural video setting. Longer sequences carry default, we train DRIBO with sequences of length 32. this ablation study, we present results of DRIBO trained using sequences of lengths and 16 respectively Figure 5. Using sequences of length 3 is similar to stacking 3 consecutive frames which is a common choice for training in DMC.
temporal information for DRIBO to learn. By
cartpole/swingup
cheetah/run
finger/spin
Average Returns
1000
759
500
200
hopper/hop
wa kerkwalk
1e5
walker/run
Average Returns
150
300
600
<20
200
5
Environment Steps
Environment Steps
Environment Steps
sea-len-3
seq-len-6
seq-len-16
seg-len-32
DRIBO-no-skl
Figure 5: DRIBO achieves better performance by capturing temporal structure RL. from longer training sequences. Compressing away task-irrelevant information using the SKL term in DRIBO loss improves performance when the architecture choice and training configurations are the same. We perform 5 runs for each method under the natural video setting. More results are in Appendix C.2.
8

---

Under review as a conference paper at ICLR 2022
Using sequences of length 6 is similar to the design choice made in PI-SAC (3 steps for the past and 3 steps for the future). We also include results on using 1 16-step sequential observations to investigate how DRIBO's performance changes as the length of the sequences increases. Theoretically, the DRIBO loss provides a lower bound on the mutual information between sequences of observations and sequences of (latent) representations with the same length. It can be observed that DRIBO performs significantly better when training with longer sequences.
Learning Objective. In the DRIBO loss, we use the SKL term (second term in Eq. (7)) compress away task-irrelevant information. Figure 5 studies the effect of removing the SKL term in the DRIBO loss (annotated as -o--O)..-skl), The objective without the SKL term is equivalent to an Info objective. We use identical training configurations (e.g. sequence length of 32) for DRIBO-no-st and DRIBO. The only difference is whether the SKI. term is included in the learning objective. Figure shows that DRIBO outperforms DRIBO-nO-ski substantially in the natural video setting.
5.3 GENERALIZATION
The natural video setting of DMC is suitable for benchmarking robustness to high-dimensional visual distractors. However, the task-relevant information and the task difficulties are unchanged. Thus, we use the ProcGen suite to investigate the generalization capabilities of DRIBO. For each game, agents are trained on the first 200 levels, and evaluated w.r.t. their zero-shot performance averaged over unseen levels during testing. Unseen levels typically have different backgrounds or layouts, which are relatively easy for humans to adapt to but challenging for RL. agents.
We compare DRIBO with recent methods that incorporate data augmentation. In addition to com- paring with RAD, we compare DRIBO with DrAC (Raileanu et al., 2020) which applies two regularization terms for policy and value function using augmented data. UCB-DrAC is built on top of DrAC, which automatically selects the best type of data augmentation for DrAC. For RAD and DrAC, we use the best reported augmentation types for different environments. DRIBO selects the same augmentation types except for a few games. The details can be found in Appendix D. We also compare the Procgen results with DAAC (Raileanu & Fergus, 2021) and IDAAC (Raileanu & Fergus, 2021), two state-of-art methods on the Procgen suite that do not apply data augmentation. DAAC decouples the learning of the policy and value function in RL improve the generalization of RL. IDAAC is built on top of DAAC by adding an auxiliary loss based on an adversarial framework.
Table shows that DRIBO achieves higher aver- aged testing returns compared to the PPO baseline and other augmentation-basse methods. The few environments, in which our approach does not out- perform the other augmentation based methods, share the commonality that task-relevant layouts remain static throughout the same run of the game,
Table I: Procgen returns on test levels after train- ing on 25M environment steps. The mean and standard deviation are computed over 10 seeds.
Ratta
57.
112.
13.44-51
54
-. 30
No2 m-4
18
383
Nep
50-007
20--1
67 62
Since the current version of DRIBO only consid-
Cantbbur
11:00 6m
51-06
634.04
45
50 0.
15 14
ers the mutual information between the complete input and the encoder output (global MI (Hjelm
Jaseer
18:0.0 05
65
6.1.04
4.5 24
63 431.02
K4
Cleeke
69:02
21
83
et al., 2019)). it may fail to capture local features. The representations for sequence of observations within the same run of the game are treated as globally negative pairs in DRIBO but they may be locally positive pairs. Thus, the performance of DRIBO can be further improved by considering local features (e.g. positions of the layouts) shared between representations positive pairs in the mutual information estimation. We leave this investigation to future work. DRIBO also outperforms DAAC and IDAAC in 9 of the 16 games.
Dagetell
28:07
48..08
17
Lege
10-07
72 1.8
8.5
Mon
41-13
43--31
54
05-00
98 -1
6 CONCLUSION
We introduce novel robust representation learning approach based on the multi-view information bottleneck principle for RL problems. Our experimental results show that (1) DRIBO learns represen- tations that are robust against task-irrelevan distractions and boosts the RL agent's 's performance even when complex visual distractors are introduced, and (2) DRIBO improves generalization performance compared to well-established baselines on the large-scale Procgen benchmarks.

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement. The implementation code can be found in Supplementary Material.zip and we will also release it on GitHub once the paper is published. All datasets we use are public. In addition, also provide detailed experiment parameters in the Appendix D.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. International Conference on Learning Representation, 2021. 1
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540. PMLR, 2018.
William Bialek and Naftali Tishby. Predictive information. arXiv preprint cond-maz/9002441, 1999.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars, arXiv preprint 160.v:1600.07316 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020. 3
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference Machine Learning, pp. 1282-1289. PMLR, 2019.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. International Conference on Machine Learning. PMLR, 2020. 7
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:/810.00//23 2018.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. International Conference on Learning Representation, 2020. 1, 5, 16, 17, 18, 27
Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020. 1,3
Tuomas Haarojj,, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 3555-2565. PMLR, 2019. 1, 6, 2,
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representation, 2020. 2
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. International Conference on Learning Representation, 2021. 2, 27
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, pp. 41 4182-4192. PMLR, 2020. 3
R Devon Hjelm, Alex Fedorov, Samuel Lavoie Maroie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference Learning Representation, 2019. 3, 9, 29
10

---

Under review as a conference paper at ICLR 2022
Rico Jonschkowski and Oliver Brock. Learning state representations with robotic priors. Autonomous Robots, 39(3):407-44) 428, 2015.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1eXiv::705.0665 2017. 6, 7
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference on Machine Learning, pp. 3360-3369, 2019. 3
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2010. 2
Sascha Lange, Martin Riedmiller, and Arne Voigtländer. Autonomous reinforcement learning on raw visual input data in real world application. In The 2012 international joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2012. 2
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020a. 2, 3,
Misha I Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020b. 2,7
Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. Advances in Neural Information Processing Systems, 33, 2020a. 1, 7
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 33:1 1890-11901, 2020b. 3,7
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1375 2016. 1
Minne Li, Lisheng Wu, WANG Jun, and Haitham Bou Ammar. Multi-view reinforcement learning. In Advances in neural information processing systems, pp. 1420-1431, 2019. 3
Yingming Li, Ming Yang. and Zhongfei Zhang. A survey of multi-view representation learning. IEEE transactions knowledge and data engineering, 31(10):1863-188,, 2018.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long DOAN, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances Neural Information Processing Systems, 33, 2020. 3
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 ((7754)) -533, 2015. 1
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10XXvv8807.0374: 2018. 3, 27
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180.
PMLR, 2019. 3
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 1334- 1373, 2021. 3,9
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning, arXiv preprint arXiv:2006.12880 2020. 9, 29
11

---

Under review as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1801v08800006699 2018. 6, 27
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicc/0004057, 2000. 4
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11) 1), 2008. 8
Niklas Wahlström, Thomas B Schön, and Mare P Desienroth. From pixels torques: Policy learning with deep dynamical models, In Deep Learning Workshop at the 32nd International Conference on Machine Learning (ICML 2015), July 10-11, Lille, France, 2015.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: a locally linear latent dynamics model for control from raw images. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume pp-
2754, 2015. 2
Wenhao Yu, C Karen Liu, and Greg Turk. Policy transfer with strategy optimization. In International Conference on Learning Representations, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representation, 2017. 8
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement 2018a. 1, 2,
learning. arXiv preprint
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. International Conference Learning Representation, 2021. 1, 2,
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint /rXiv:/804 06893, 2018b.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A THEOREMS AND PROOFS
In this section, we first list properties of the mutual information we use in our proof. For any random variables X, and Z.
(..1) Positivity:
20
(P.2) Chain rule:
I(XY: -() I(X;
(P.3) Chain rule (Multivariate Mutual Information):
(P.4) Entropy and Mutual Information:
-
(P.5) Chain rule for Entropy:
H x Σ [XX
(P.6) The mutual information among three variables is bounded by:
mmnn < (1)
A.1 THEOREM
We first show that
lower bound of I(S1:T;
11:1:;:0.TAA4 A
P.
T(O1.r: - T(S1::: O1:;; Aur)
Since S1. are representations of O1. we have 1(5):::A::100.r) 0. With (P.1) and (P.6), we have that:
1(:
Thus, we have:
I(O1:r: S1a) 2 I(S1::: 01:l
Furthermore, since I(Surr; O1er; Air) is lower bounded, maximizing I(Si:r:0 also maximizes (OI::;;1..)
Theorem A.1. Let Our be the observation sequence obtained by executing action sequence A Syr is sequence of sufficient representations for Ov:r, then we have:
T
I(S1:: 01T((A1))
(S: At-1)
(8)
13

---

Under review as a conference paper at ICLR 2022
Proof. We indicate the property use for each step of the derivation below.
I(S1:;; O1:lA1::1
P4 H(1. AIT)
(S O1:, AI.T)
rs
(1 ((S:AArT: AL:T OuT,, S1x-1))
P4
(1S,:O
P4
(II H( Aur, (ta-))
PS
1(8; 01a-
Σ I(S: 1,1,
Here, we provide a formal proof for the last step of the derivation above. Let t represent 01.7, 01 We have
11 S1 01a-1)
= - p(T)p(s;, o.lT) log
p(s 0, 01:7, $1:111 $111,,
ds:do,dr
(( -1 : a1.7, 818-1, 01a-1)
With the density of mieti-view trajectories Eq. (1), we can observe that or and 8; are generated by p(o, ((o.(ss)p(ss) 81-1, at-1) p(8;, 0: 0,(1,,,,-1)) Thus, have:
p St.1:
p(8r, 0110111;
91e = p( 8;-1, a1-1)
plo.larr. 9tm-,, p( 11-1;
This in turn implies that:
As result, we have a lower bound [(S (1) (1) Air)): (1) A,-1, A, 5 (( ( At-1))
T(S of AIT) > Eu (
With the information bottleneck principle, we have losses in Eq. (5) and Eq. (6) to compress away iask-irrelevant information in representations while maximizing the mutual information between representations and observations.
A.2 SUFFICIENT REPRESENTATIONS RL
In this section, first present the sufficiency condition for sequential data. Then, prove that the sufficiency condition on the sequential data holds, then the sufficiency condition on each corresponding individual representation and observation holds well.
14

---

Under review as a conference paper at ICLR 2022
Theorem A.2. Let O1:T and Air be random variables with joint distribution p(o1:T ai.)) Let SyT be the representation of Ou.,, then SIT is sufficient for Air if and only if [(O1::Air) I(Sr:; AiT)) Also, St is sufficient representation of O since I[O,; Ar 1,-1, =0. Hypothesis:
(H.I) Su.T is sequence of sufficient representations for Ou:r:
I(Or: Air = 0
Proof.
10
[(Or::; Air)) I(O1:;; A1:::1
[(Or:: Airr) - [(Ai:; S1.) - [(Airri S1.1(11:)
With SI.T representation of O1.T have I(S;;; A1: The reason is that Our shares the same level of information as A and S1.T. Then,
A S1.) =I 1(01::;; (A:I::SS:))) I(O1:: Ai.
(9)
So the sufficiency condition I(O1:;; Air holds if and only if I(O Aix)
(4i:;:Sr)) We factorize the mutual information between sequential observations and optimal actions
1(O Ai.)
I(OO; ((0.::
(L)) II((;; (100
I(Su:: Af.))
I(S;; ((. + 1;
Then we obtain the following relation:
I(O,; 17aa0 T(S:; Ai.
(10)
15

---

Under review as a conference paper at ICLR 2022
We also have
I(Or; A1,0:111,101..11)
I(O A1x) 1001:011:4
(22) 1(00:--1:AA,100) + A1.1;A
-1(O).-1iAii))
A(.1.) T(0r;; Ait)
-1(S1--1:Ai.)
[() ( 1i
T(0,; (1,1:-111))1:-111
T(S:: 11
1(0;
[(S; A 11 T(S, |
I(O; (44, S1:-1)
Eq. (9)
I(O,,
With the above derivation and Markov property, we have I(O,; Aj generalize A; to any A,-1 by assuming A as the optimal action for state S₁ whose last timestep state-action pair (5-11,4-1) Thus, we have S, sufficient representation for O if and only S1.7 is a sufficient representation of O1.r.
Ar-1) - O. We can
A.3 MULTI-VIEW REDUNDANCY AND SUFFICIENCY
Proposition A.1.
1.T is redundant view with respect o to obtain Air if only if Air 1 1:7 0. Any representation ( of ( that is sufficient for (2) is also sufficient
[(O 1:7H
for A 1:
Proof. See proof of Proposition B.3 in the MIB paper (Federici et al., 2020).
L
Corollary A.1. Let (1) and ( be two mutually redundant views for Ai. Let (()) be
representation of ( If ((1) is sufficient for ( s(1) can derive Až as the joint observation of the two views (T(O) (1) 0
(2) S7 A ()) 1, At-1)), where S is any sufficient
representation timestep f
Proof. For the sequential data, proof of Corollary B.2.1 in the MIB paper (Federici et al., 2020) to prove
-i Airr)
1:75
According to Theorem A.2, if ((1) is a sufficient representation of ( 1:7
1.7, s(D)) is sufficient rep- resentation of 2 Similar to proof on sequential data, we can use Corollary B.2.1 in the MIB paper (Federici et al., 2020) to show that
1100 A A A
16

---

Under review as a conference paper at ICLR 2022
Theorem A.3. Let the views Oir and 0ir of observation 01.7 are obtained by data augmenla- (2)
(1)
(2)
tion transformation sequences 11T and <1-r respectively -o (O1:)) and O1T PiT 1- (ov )) We abuse the notation for simplicity to represent (1) (0 and (0 as random variables for augmented observations. Whenever the two views (1)) and (2) must be mutually redundant for Arr- Besides, the two views Og and 0 must be mutually redundant for AZ.
AT) (O (O1:T): A AiT)
Proof. Let 81.T be sufficient representation for both original and multi-view observations. We first factorize the mutual information and refer A.2 as Theorem A.2.
(Ona) A - 1 - 0100
A 110
((0
(22 =1009
A/3 1110
12
A. I(0,; ((05:
Then, we have the following equality
1(of") :4. -1=)))= :|
Similar as derivation in Theorem A.2
Ai.
P.22 [(O! (1) A/ A 1 -
Fq. T(O ())
S1+-1) - T(S;; |
We apply the same derivation for (2) and we have the following with Markov property
(O): A.
=I(1(")(0): A,-1)
We show that the condition on sequential data can be expressed each timestep with the similar form. See proof of Proposition B.4 in the MIB paper (Federici et al., 2020) for mutual redundancy between sequential views and individual pairs of views.
Theorem A.4.
Suppose the mutually
redundant condition holds, i.e.
If s(1) is a sufficient representa- tion for (2) (Our) then 1(0r; S, Ac-1) = T(S{ 1,]S._1,
(0 (O.r)) Ai (
Proof. Since (2) (Oir) is redundant for (2) (O (Theorem A.3), any representation sS01) of (1) (Orr)) that is sufficient for (2) (O1z)) must also be sufficient for A; (Theorem A.2 and Propo- sition A.I). Using Theorem A.2 have I(S(", Aj A AT Ar-1). 1).
With (((()(()) St-1, 1--1) = I(O,; S7 At11), we conclude T(O,; S7 = T(S!)) AT A,-1).
17

---

Under review as a conference paper at ICLR 2022
We finally show the proposition for the culti-Inffrration Bottleneck principle in RL with the gener- alization of sufficiency and mutually redundancy condition from sequential data to each individual pairs of data.
Proposition A.2. Let 0 and o be mutually redundant views for A that share only optimal action information. Then sufficient representation s(") of Of for 0 that is minimal for 0.2) also minimal representation for Ai.
Proof. See proof of Proposition E.1 in the MIB paper (Federici et al., 2020).
B DRIBO Loss COMPUTATION
consider the average of the information bottleneck losses from the two views.
(1I)
:f ( A A-1))
S(2) A,-1) :0!
(12)
2
Consider s, and s on the same domain S, I(S
of (S A,-1, 0 can be expressed as:
I(S
O
A, )
A
(1) 0-11)
Po
=E
log
(o
log Pa(s)
po
a
E
Pe
DKL
DKo(po(s
(1) (2)
<Dul(po
(13)
Note that equality holds if the two distributions coineide. Analogously I(S 0 s (()))) upper bounded by DRL (po (s) (2)
A.,,,,
(2)
8 at 1)
(2)
l0 (1)
Assume S;-1 is sufficient representation of O,-1- Then, $
() provides task-relevant information
no more than the sufficient representation S.-1- [(S(1) 0 (1)) A(-1) can be thus re-expressed
2 50 5, 4,1) 1( S
5 A,
= (-
0( ( A--1A--1)
T(S)
(
21(S S A-1,)
(14)
18

---

In Algorithm 1, we use 8.0) po 10 [ a aI-1) and "e pa (2) 10,") 8 0:-1) to obtain representations for multi-view observations. argue that the substitution does not affect the effectiveness of the averaged objective. With the view assumption, b(2) and 8,2) do not share any task-irrelevant information. So, the representations at timestep conditioned on them do not share any task-irrelevant information. Maximizing the mutual information Under review as a conference paper at ICLR 2022
Where follows from ((2) being the representation of 0, The bound is tight whenever S.
sufficient from 5t (I(S) o ({ 1, A(-)==o. This happens whenever s! con-
tains all the information regarding S." Once again, can have
|S|-1, At-1) 2
T(S,", ;, |S-1, 12-1). Therefore, the averaged loss functions can be upper-bounded by
C - A A2 (( S 1S,-1,
A2
(15)
+ + (pok((pos)) (po(s)
10 ,
/ - (0, 108²) 8. ,
Lastly, by pa-ameetrizzngg the objective, we obtain:
3)= -(0:3)=-18) lo( [ (1-1,
(16)
(p
10 ,8 -- () 10," 1 a--))
(1) (1)
(2) (2)
(2)
have that representations
between s and si
(2) (first term in Eq. (16)) will encourage the representations to share maximal task-relevant information. Similar argument also works for the second term in Eq. (16). Since s
and 8 do not share any task-irrelevant information, any task-irrelevant information introduced from the conditional probability will be also identified task-irrelevant information by KI divergence, which will be reduced through minimizing the DRIBO loss.
C ADDITIONAL RESULTS
C.1 ADDITIONAL DMC RESULTS
For clean setting, the pixel observations have simple backgrounds as shown in Figure 2 (left column). Figure 6 shows that RAD, SLAC, CURL and PI-SAC generally perform the best, whereas DRIBO consistently outperforms DBC and matches SOTA.
For natural video setting, Figure 7 shows that DRIBO performs substantially better than RAD, SLAC and CURL which not discard task-irrelevant information explicitly. Compared to PI-SAC and DBC, recent state-of-art methods that aim at learning representations that are invariant to task-irrelevant information, DRIBO outperforms them consistently.
19

---

Under review as a conference paper at ICLR 2022
cartpole/swingup
cheetaf/run
finger/spin
Average Returns
800
3003
803
502
420
240
ball cup/catch
reacherleasy
hopper/hop
1000
1000
Average Returns
203
150
400
400
50
Walker/stand
walker/walk
walker/run
603
Average Returns
1000
603
600
300
440
103
2 3 4 & Environment Steps
Environment Steps
2 A
105
Environment Steps
2
les
(Ours)
SLAC
CURL@1e6
PI-SAC@Ie6
Figure 6: Average returns on DMC tasks over seeds with mean and one standard error shaded in the clean setting.
cartpole/swingup
cheetah/run
finger/spin
1000
Average Returns
000
400
800
200
cup/catch
reacherleasy
hopper/hop
1000
Average Returns
800
800
150
100
wa walker/stand
waler/walk
walker/run
1.000
Average Returns
600
200
100
3
Environment Steps
Environment Steps
Environment Steps
DRIBO (Ours)
RAD
SLAC
CURL
DBC
PI-SAC
Figure 7: Average returns on DMC tasks over seeds with mean and one standard error shaded in the natural video setting.
20

---

Under review as a conference paper at ICLR 2022
C.2
DRIBO LOss vs. INFOMAX
C.2.1
NATURAL VIDEO SETTING
cartpole/swingup
cheetah/run
finger/spin
200
300
SKL Value
150
200
200
100
50
0
7
2
3
4
5
6
7
8
hopper/hop
walker/walk
hopper/hop
250
SKL Value
150
200
400
100
150
300
200
50
50
100
1
2
5
6
7
8
1
2
3
7
8 le5
1e5
le5
Environment Steps
Environment Steps
Environment Steps
DRIBO (Ours)
DRIBO-nO-skl
Figure 8: Average SKL, values during training in DMC environments with natural videos back- ground.
To quantify the amount of task-irrelevant information retained in the representations, we compare the SKL term values between DRIBO and DRIBO-nO-skl during training in Figure 8. As described Section 5, DRIBO-nO-skl is trained with an InfoMax- objective without a SKL term. The gap between the SKL values explains the performance gap between the two approaches (as shown in Figure 5). The models trained with DRIBO take advantage of the information bottleneck to map observations from different views close to each other in the latent space. Figure 8 shows that minimizing the DRIBO loss consistently reduces the KL divergence between representations from different views. On the other hand, the models trained with DRIBO-no-skl fail to discard the task-irrelevant information contained in observations from different views even though the RSSM model helps to learn predictive representations. For DRIBO-no-skl. the KL divergence between representations from different views is consistently larger than the one learned by DRIBO.
C.2.2 CLEAN SETTING
We provide additional results on comparing DRIBO with DRIBO-no -no-skl under the clean setting Fig- 9 and Figure 10. It can be observed that DRIBO and DRIBO-no-ste perform similarly in terms of average returns. Recall our earlier plot on comparing DRIBO with DRIBO-no-ski under the natural video setting Figure 5 which shows DRIBO substantially outperforms DRIBO-nO-skl (in other words, the performance of DRIBO-nO-sk drops when the setting is changed from clean to natural vidco). Similar results can be seen in Figure and Figure where the performance of approaches like RAD, SLAC, CURL and PI-SAC significantly degrades when the backgrounds of the environments are changed different natural videos during training and testing.
21

---

Under review as a conference paper at ICLR 2022
cartpole/swingup
cheetah/run
finger/spin
Average Returns
1000
800
600
800
400
600
400
400
200
200
6
4
5
6
7
8
1
2
3
4
5
6
7
8
hopper/hop
walker/walk
walker/run
Average Returns
200
1000
800
150
400
600
100
300
400
200
50
200
100
5
1e5
1e5
1e5
Environment Steps
Environment Steps
Environment Steps
DRIBO (Ours)
DRIBO-nO-skl
Figure 9: Average returns DRIBO and DRIBO-nO-skI on DMC tasks over seeds with mean and one standard error shaded in the clean setting.
cartpole/swinguu
cheetah/run
finger/spin
500
300
800
SKL Value
400
600
300
200
400
200
100
100
0
1
4
5
6
7
8
1
2
3
4
5
6
7
8
hopper/hop
walker/walk
hopper/hop
500
400
400
SKL Value
400
300
300
300
200
100
100
100
le5
165
1e5
Environment Steps
Environment Steps
Environment Steps
DRIBO (Ours)
DRIBO-nO-skl
Figure 10: Average SKL. values during training in DMC environments in the clean setting.
22

---

Under review as a conference paper at ICLR 2022
C.3
TRAINING AND TESTING PERFORMANCE IN DMC UNDER THE NATURAL VIDEO SETTING
cortpole/swingup
cheetalvrun
fingerispin
Average Return
400
too
100
200
200
hogper/hop
walker/walk
walker/run
2000
Average Returns
500
200
400
200
6
Environment Steps
lsd
Environment Steps
1st
Environment
1s5
DRIBO (Ours) Train
DRIBO (Ours) Test
RAD Trian
RAD Test
Figure 11: Training and testing performance of RAD and DRIBO.
cartpole/swingup
cheetah/run
fingerispin
Average Retums
1400
500
2000
400
200
400
hopper/hop
2000
walker/walk
walker/run
1e5
Average Retums
200
400
150
500
100
Environment Steps
1+5
Environment Steps
1e5
Environment Steps
DRIBO (Ours) Train
DRIBO (Ours) Test
RAD Trian
RAD Test
Figure 12: Training and testing performance of CURL and DRIBO.
cartpole/swingup
cheatah/run
finger'spin
Average Returns
300
400
1036
300
630
200
hopper/hop
Walker/walk
walkeriruu
1s5
Average Returns
200
430
600
330
600
600
206
50
200
Envirenment Steps
Environment Steps
Environment
165
DRIBO (Ours) Train
DRIBO (Ours) Test
PLSAC Test
PLSAC Train
Figure Training and testing performance PI-SAC and DRIBO.
We further compare DRIBO with RAD, CURL and PI-SAC the DMC environments under the natural video setting. We observe that RAD, CURL and PI-SAC could achieve high scores during training but failed achieve the same high scores (with substantial gap) during testing.
23

---

Under review as a conference paper at ICLR 2022
C.4 ADDITIONAL VISUALIZATION
Figure 14 are the snapshots of training environments of DMC under the natural video setting. The background videos randomly sampled from the class of "arranging flower" which drastically different from backgrounds used during testing (Figure 15).
J.F VER AB AN .1 OFT
rt
01:12
Figure 14: Training environments of DMC under the natural video setting: The background videos sampled from arranging flower class in Kinetics dataset.
Figure 15: Test environments of DMC under the natural video setting: We show the sequential observations with natural videos as background in DMC and their corresponding spatial attention maps of the DRIBO trained encoder.
In Figure 15, we show sequential observations in test environments DMC under the natural video setting. The backgrounds videos randomly sampled from the test data of Kinetic dataset which contain various classes of videos. Note that a single run the DMC task may have multiple videos playing sequentially in the background.
24

---

Under review as a conference paper at ICLR 2022
Figure 15 also visualizes the corresponding spatial attention maps of DRIBO trained encoders. For different tasks, DRIBO encoders' activations concentrate on entire edges of the body, providing a more complete and robust representation of the visual observations.
In addition to Figure 2, we also show spatial attention maps for each convolutional layers in Figure 16. We can observe that DRIBO trained encoders filter the complex distractors in the backgrounds gradually layer by layer.
Figure 16: Spatial attention maps for each convolutional layers of the DRIBO trained encoders. The observations are the same as ones in Figure 2 from the snapshots during testing.
To further compare DRIBO with frame-stacked CURL, we visualize the learned representations using t-SNE plots in Figure 17. We see that even when the backgrounds are drastically different, our encoder learns to map observations with similar robot configurations near each other, whereas CURL's encoder maps similar robot configurations far away from each other. This shows that CURL does not discard as many irrelevant features in the background as DRIBO does despite leveraging data augmentations and backpropagating RL objectives the encoder.
Background 1
Background
Figure 17: I-SNE of latent spaces learned with DRIBO (left) and CURL (right). They are the same as t-SNE in Figure 4. But we color-code the embedded points corresponding to their backgrounds. The observations from the same trajectory but with different background natural videos (the same as in Figure Figure 4). Points from different backgrounds are close to each other in the embedding space learned by DRIBO, whereas no such structure seen in the embedding space learned by CURL.
25

---

Under review as a conference paper at ICLR 2022
D IMPLEMENTATION DETAILS
D.1 RSSM
We split the representation 8; into stochastic part 2, and deterministic part hes where 8; (e)) The generative and inference models of RSSM are defined as:
Deterministic state transition: h. - 1 ar-11)
Stochastic state transition: 2; = pp22(
Observation model: or = p(Or (e)
where 1 2,-1, a,-1 is implemented as a recurrent neural network (RNN) that carries the dependency the stochastic and deterministic parts at the previous timestep. Then, obtain the rep- resentation with the encoder po((81.T] OL.T, 01:7)= II, (s (r, ht), where he retains information from 81-1 (Zt-1; 11) and a;-1-
D.2 DRIBO SAC
We first show how we train SAC agent given the representations of DRIBO. Let Φ(0) ~ Po(s|o,8 s s', a') denote the encoder, where s' and a' as the representation and action last timestep.
Algorithm 2 SAC DRIBO Encoder
1: Input RL batch BRL =, (T-1)+M
with (T 1) N pairs of representation,
action, reward and next representation.
2: Get value: V= min (,(((o'))
log log (()
3: Train critics: J(Q o) = (Qi(6(o)) - -V)
4: Train actor: ((T)= log ((lo(())) 11 ((((o))
5: Train alpha: J(a) -a log
6: Update target crities: Q QQ + - -Q)
7: Update target encoder: @ Tap + TØ)Ø
Then we incorporate the above SAC algorithm into minimizing DRIBO loss in Algorithm
Algorithm DRIBO SAC
1: Input: Replay buffer DD storing sequential observations and actions with length The batch size is N. The number of total training step is K. The number of total episodes is E.
2: for .,...; do
3:
Sample sequential observations and actions from the environment and append new samples
to D.
4:
for each step k 1,.... K do Sample a sequential batch B ~ D.
5:
6:
Compute the representations batch BRI. which has the shape NT, using the encoder
pa (81:T) a1.T)
7:
Train SAC agent: EB [J Q,
Algorithm > Algorithm 1.
8:
Update and z0 to minimize LDRIBO using B
9:
end for
10:
end for
D.3 DRIBO PPO
The main difference between SAC and PPO is that PPO is on-policy RL algorithm while SAC is an off- policy RL algorithm. With the update of the encoder, representations may not be consistent within each training step which breaks the on-policy sampling assumption. To address this issue, instead obtaining 8, propagating from the initial observation the observation sequence, we store the representations sold while sampling from the on-policy batch. Then, ((()
26

---

Under review as a conference paper at ICLR 2022
Pe(s)o, snil a') to denote the representation from the encoder. Here, s and a' are the representation and action at the previous timestep. By treating the encoding process as a part of the policy and value function, the on-policy requirement satisfied since the new action/value at timestep depends only on (o,, sod a1-))
Algorithm DRIBO PPO
Input: Replay buffer D and on-policy replay buffer Pppo storing sequential observations and actions with length T. The batch size is N. The minibatch size for PPO is M. The number of total episodes E.
2: for e == 1,=1... 1..... do
3: Sample sequential
observations and
actions
from
the environment
((OuT, a1.7, rI: 817 Ji=1'
1.N
4: Append new samples to and update the on on-policy replay buffer Dpro-
5: for L..... do
6:
{(p(o,), a, r:)) Optimize PPO policy, value function and encoder using each sample (1p(0)) as r:) in
Drro
7:
the batch.
8:
Sample a sequential batch B D.
9: Update 0 and y to minimize LDRIBO using B
Algorithm 1.
10: end for
11:
end for
D.4 DMC
We use an almost identical encoder architecture as the encoder in the RSSM paper (Hafner et al., 2019), with two minor differences. Firstly, we deploy the encoder architecture in Tassa al. (2018) as the observation embedder, with two more convolutional layers to the CNN trunk. Secondly, we add layer normalization to process the output of CNN, deterministic output and stochastic output. Deterministic part of the representation is 200-dimensional vector. Stochastic part of the representation is 30-dimensional diagonal Gaussian with predicted mean and standard deviation. Thus, the representation is a 230-dimensional vector. We implement Q-network and policy in SAC as MLPs with two fully connected layers of size 1024 with ReLU activations. We estimate the mutual information using a bi-linear inner product the similarity measure (Oord et al., 2018).
Pixel Preprocessing. We construct an observational input as a single frame, where each frame is a RGB rendering of size 100 x 100 from the Oth camera. We then divide each pixel by 255 scale it down to [o, 1) range. For methods compared our experiments, an observation inputs contains an 3-stack of consecutive frames.
Augmentations of Visual Observations. For RAD, we use random crop+ grayscale to generate augmented data. For our approach DRIBO and CURL, use random crop generate multi-view observations. We apply the implementation of RAD to do the augmentation. For random crop, extracts random patch from the original observation. In DMC, we render 100 100 pixel observations and crop randomly to 84 x 84 pixels. For random grayscale, it converts RGB images to grayscale with a probability p= 0.3.
Hyperparameters. To facilitate the optimization, the hyperparameter 3 in the DRIBO loss Algo- rithm 1 slowly increased during training. / value starts from small value le and increases to le 3 with an exponential scheduler. The same procedure is also used in the MIB paper (Federici al., 2020). We show other hyperparameters for DMC experiments in Table
KL Balancingg During training, we also incorporate KL. balancing from variant method described in DreamerV2 (Hafner et al., 2021) to train RSSM. KL balancing encourages learning an accurate prior over increasing posterior entropy, so that the prior better approximates the aggregate posterion. This help us to avoid regularizing the representations toward poorly trained prior. KL balancing is orthogonal to MIB objective (DRIBO Loss). Note that DreamerV2 deploy KL balancing based on a reconstruction loss. In our case, our results show that KL balancing can improve training of RSSM with contrastive-learnng-bba
objective. We implement
27

---

Under review as a conference paper at ICLR 2022
this technique shown in Algorithm 5. 3 shares the same value the coefficient for SKL term in DRIBO Loss.
Algorithm 5 DRIBO Loss + KL Balancing
1: Compute kl balancing term:
kl balancing 0.8 compute compute_kl(stop) (ristop_gra((aprrog ( + prior)
+ 0.2 compute compute_kl(aprroxx wwlpapproxpposteeeeeee posteriorsttrroroe + stop_ (9o__gra(prrior)
return DRIBO Loss + B- kl balancing
Table 2: Hyperparameters used for DMC experiments.
Hyperparameters
Value
Observation size
(100 100)
Cropped size
(84 x 84)
Replay buffer size
1000000
Initial steps
1000
Stacked frames
No
Action repeat
2 finger, spin; walker, walk;
8 cartpole swingup
otherwise
Evaluation episodes
8 Adam
Optimizer Learning rate
encoder learning rate: 1e-5;
policy/Q network learning rate: le-5;
learning rate: le-4.
Batch size Target update
8 x 32, where = 32
network: 0.01
encoder: 0.05
Target update freq
2
Discount 7
.99
Initial temperature
0.1
Total timesteps
88e4
B scheduler start episode
10
B scheduler end episode
60
28

---

Under review as a conference paper at ICLR 2022
D.5 PROCGEN
For Procgen suite, the implementation of DRIBO is almost the same DMC experiments. Better design choice could be found by validation. We use the same as the encoder architecture used in DMC experiments, except for the observation embedder, which we use the network from IMPALA paper to take the visual observations. In addition, since the actions in Procgen suite are discrete, we use an action embedder to embed discrete actions into continuous space. The action embedder is implemented as a simple one hidden layer resblock with 64 neurons. It maps one-hot action vector to 4-dimensional vector. The policy and value function share one hidden layer with 1024 neurons. The policy uses another fully connected layer to generate categorical distribution to select the discrete action. The value function uses another fully connected layer to generate the value for an input representation. All activation functions are ReLU activations.
Augmentation of Visual Observations. We select augmentation types based on the best reported augmentation types for each environment. DrAC (Raileanu et al., 2020) reported best augmentation types for RAD and DrAC in Table 4 and 5 of the DrAC paper. We list the augmentation types used in DRIBO in Table and 4. We use the same settings for each augmentation type DrAC. Note that we only performed limited experiments to select the augmentations reported in the tables due to time constraints. So, the tables do not show the best augmentation types in each environment for DRIBO.
Table 3: Augmentation type used for each game.
Fav
BigFish
StarPilot
FruitBot
BossFight
Ninja
Plunder
CaveFlyer
CoinRun
Augmentation
crop
cutout
cutout
cutout
random-conv
crop
random-cony
random-conv
Table 4: Augmentation type used for each game.
Eny
Jumper
Chaser
Climber
DodgeBall
Heist
Leaper
Maze
Miner
Augmentation
random-conv
crop
random-conv
cutout
crop
crop
crop
flip
Hyperparameters. We use the same 3 scheduler the DMC experiments. The starting 3 value is 1e 8 and the final / value is le 3. We show other hyperpurameters for Procgen environments in Table 5.
Here, we extend the discussion on why our method underperforms on some environments, whose screenshots are shown in Figure 18.
Discussion.
Our approach, DRIBO, only consider global MI (Hjelm al., 2019) in the current implementation. As result, local structures can be easily ignored in the representation. More specifically, representations containing the same static local features within a single execution but at different timesteps are treated as negative examples in the mutual information maximization. Then, the information of these local features shared between representations is not maximized. Negative pairs of representations sharing this type of local features are globally negative pairs but locally positive pairs.
For Plunder, the goal is to destroy moving enemy pirate ships by firing cannonballs. The enemy ships can be identified with the color of the target in the bottom left corner. The background of the game maintains the same within single execution. Wooden obstacles capable of blocking the player's s cannonballs. In single execution, critical features like the target label and wooden obstacles remain unchanged. Failing to capture these local features results in poor performance of an agent.
For Chaser, the goal is to collect all green orbs as well as stars. collision with an enemy that not vulnerable (red) results in the death of the player. The background remains the same across different executions. Walls in the environment are dense and remain static during a single execution. Failing to capture walls' positions in representations hinders the agent from navigating to avoid enemies and collect orbs/stars. In Maze and Leaper, walls also remain static, but the backgrounds are different across different executions. This difference reduces the influence introduced by globally negative pairs but locally positive pairs. By contrast, walls in DolgeBall remain static but more critical since hitting at wall ends the game.
29

---

Under review as a conference paper at ICLR 2022
Table 5: Hyperparameters used for Procgen experiments.
Hyperparameters
Value
Observation size
(64 x 64)
Replay buffer size
1000000
Num. of steps per rollout
256
Num. of epochs per rollout
3
Num. minibatches per epoch
Stacked frames
No
Evaluation episodes
10
Optimizer
Adam
Learning rate
encoder learning rate: lc-4;
policy learning rate: 5e-4;
learning rate: lc-4.
Batch size
8 x 256, where T 256
Entropy bonus
0.01
PPO clip range
0.2
Discount
.99
GAE parameter
0.95
Reward normalization
yes
Num. of workers
Num. environments per worker
64
Total timesteps
25M
8 scheduler start episode
10
8 scheduler end episode
110
D.6 COMPUTE RESOURCES AND LICENSE
Compute Resources. We used desktop with 12-core CPU and single GTX 1080 Ti GPU for benchmarking. Each seed for DMC benchmarks takes 3 days to finish. For Procgen suite, takes days finish experiments each seed.
License. DMC benchmarks are simulated in MuloCo 2.0. We perform the experiments in this paper under an Academic Lab License. We perform experiments in Procgen suite with its open source code under MIT License.
Figure 18: Screenshots of the three Proegen games where approach DRIBO does not improve generalization performance compared to the other methods. From left to right, they are Plunder, Chaser and DodgeBall.
30


Reviews:
[{"review_id": "s3QP4Mp83x9", "reviewer": "Reviewer_uNzG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths and Weaknesses:\n\n\\+ The perspective of using mutual information to increase the robustness is interesting. The authors provide clear analyses of how to use mutual information to learn task-irrelevant information based on multi-view settings. \n\n\\+ The proposed method is a plug-in penalty that can be integrated into many modern RL algorithms. \n\n\\- The representation encoder and the estimator of mutual information (using InfoNCE) are borrowed from existing works, which may limit the contribution of this work.\n\n\\- The proposed methods show advantages in DMC and ProcGen environments. However, the settings in both environments are very similar: changing the background image in the same task. I am not sure if there are any real-world tasks that can be better solved by the proposed method. It would be great if the author can conduct experiments on other realistic tasks.\n\n\n2. General questions\n\n(1)\tHow is the data augmentation conducted to create the multi-view setting? For example, in Figure 1, is the entire image rotated or only the background rotated? If the entire image is rotated, will the state change and the action not be consistent with the state?\n\n(2)\tHow to create the multi-view setting for layouts in the ProcGen suite? In my understanding, the layout of environments is corresponding to the difficulty level. How to conduct image augmentation on such difficulty levels? Or, do the author only change the background image in ProcGen? \n\n(3)    Based on (2), I am also wondering if this method can be extended to more general settings to increase the robustness?", "summary_of_the_paper": "Motivated by the fact that RL agents are sensitive to unseen environments, this paper proposes a method to separate task-relevant and task-irrelevant information from observations based on unsupervised multi-view settings. They use data augmentation methods to create multi-view data for calculating contrastive Multi-View Information Bottleneck (MIB) and use it as the objective to increase the generalization of model representation.", "summary_of_the_review": "In general, I think the proposed idea of this paper is interesting, and the theoretical analysis seems correct to me. However, I still have questions about the details of how to implement the multi-view settings (In my general question (1)). Also, the experiment part seems a little weak since the two environment suites are very similar. \n\nI think this paper is marginally above the borderline. I may raise my score if the author provides reasonable answers to my question.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oPCD8tE2-_j", "reviewer": "Reviewer_m1E2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper shows a technically approach to compress task-irrelevant information, making the agent focus on task-relevant information.\n* The paper shows good empirical improvements on DM Control with natural video background and procgen, compared to existing methods in system-wise comparisons.\n* Comprehensive ablation studies on DM Control that identify the importance of history length and information compression.\n\nWeaknesses:\n* It’s not clear to me whether there are as many task-irrelevant observation changes in procgen as in DM Control with video background. The generalization requirement in procgen seems to be different. I would like to see if the authors can clarify how the procgen experiment align with the main story of this paper. \n* Many of DRIBO’s design choices (using RSSM, long history length, etc) are different to existing methods. The system-wise comparison on procgen (table 1) doesn’t provide much insight on why DRIBO improves generalization on procgen. \n* The authors claim that $I(S_t^{(1)};S_t^{(2)}|S_{t-1},A_{t-1})$ can be maximized by maximizing $I(S_t^{(1)};S_t^{(2)})$. I don't think this is mathematically correct without making certain assumptions.\n\nSmaller presentation issues:\n* In Sec.3, the author states that \"task-irrelevant information does not contribute to the choice of actions\". I think it should be \"task-irrelevant information does not affected by choice of actions\".\n* The author parameterizes mutual information as like $I_\\theta$. Mutual information is a true measurement and can't be parameterized. Only MI estimates can be parameterized.\n* CEB is a generic information bottleneck. It's incorrect to say that it cannot be applied to sequential data.\n* The appendix shows in the main paper file.", "summary_of_the_paper": "This paper studies the problem of pixel-based control with reinforcement learning. The authors categorize observation changes into task-relevant and task-irrelevant changes. They define task-irrelevant changes as those do not have casual relations with actions, and introduce a conditional prior to compress task-irrelevant information, such that agent can focus on task-relevant information. The authors evaluated the proposed DRIBO method on pixel-based control tasks (DM Control) with natural video background and the procgen suite.", "summary_of_the_review": "The approach is novel. The paper shows good empirical results on DM Control with natural video background and procgen. However, it's not clear to me why the procgen-type of generalization and the procgen experiment are relevant to the main story and I have some concerns about the mathematical correctness. I'm slightly leaning to acceptance at this point, but I hope the authors would be able to clarify. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bX1kQc3ySwn", "reviewer": "Reviewer_kUtq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated and the main problem is interesting in my opinion. The related work is well covered and most of the writing flows well. Experimenting over two benchmarks (ProcGen + DMC) is quite extensive relative to other similar papers. \n\nI start to have issues in understanding around Definition 1. I get the overall concept, that we can deconstruct the overall mutual information in a task-relevant and a task-irrelevant terms. However, why should we maximize the sum of both (as noted in the 5th line below Eq. 2)? You say maximizing I(O, S) gets you a sufficient representation. That would mean maximizing the first term as well. But you also say that the first term needs to be minimized. This seems contradictory. I understand Theorem 1 and also get the derivation of the components of Eq. 5 and 6. However in Eq. 5, do we want to maximize both terms? One is the task irrelevant term (aka minimality) while the other is the task relevant term (aka sufficiency). They should have opposite signs right (as is in Equation 7)? \n\nFinally, regardless of the derivation of the DRIBO loss, my main concern is the empirical results here. The scores reported for DMC are not accurate in my opinion. I have run RAD on the distractor suite and it gets much better performance than what is reported. Why is RAD performance this low? DRIBO is using augmentations from RAD to generate the multi-view observations, and so as a first baseline, it should be performing better than RAD. Note that RAD achieves the same score (roughly) as reported in the original paper even when run with distractors. Can the authors clarify this?\n\nMoreover, the performance reported for the proposed method is not good enough. In particular, running SAC with a reward prediction head achieves similar or even better performance than of the proposed method. In light of these two observations, I am not convinced that DRIBO really is doing more than a simple baseline, and certainely not doing so much better than RAD.\n\nFor the ProcGen experiments, why don't you compare with DAAC [1] and IDAAC [1] since I believe they are the state of the art methods on ProcGen currently.\n\nWhy compare the t-sne with CURL? We already know that CURL does not do a good job and that RAD is much better and even simpler. Also, in Figure 3, CURL is outperforming RAD in some of the tasks, which I think is again almost certainly not the case (based on the experiments I have run with RAD).\n\nReferences:\n\n[1] Decoupling Value and Policy for Generalization in Reinforcement Learning", "summary_of_the_paper": "This paper tackles the problem of generalization amidst visual distractors for control tasks. In particular, the distractors have no dependence on the optimal policy and thus clearly form a task-irrelevant component. The proposal is to use mutual information between two views as a proxy for how much task-relevant information is present in the constructed representation. This objective is adapted for RL to consider the long term sequential nature. Finally, the authors test this approach on ProcGen and DMC Suite with distractors, while also performing certain ablations. ", "summary_of_the_review": "The paper is motivated by an interesting problem to the community, proposes a fairly novel method for leraning better representations, but there remain certain gaps in my understanding currently. I hope the authors can clarify this during the rebuttal. In any case, the experimental results are not correct in my prior experience of running these exact methods. For this reason, I am advocating for a reject. I'm happy to revise my review if these issues can be addressed. \n\n\n-------------------------- Post Rebuttal ----------------------------------\n\nThe new results provided by the authors convince me fairly that the method is somewhat useful. I am still skeptical about the RAD results and the overall scores reported for DMC tasks, since the absolute numbers are not state-of-the-art. This might be due to a different evaluation setting where the distractor videos are changing across training, and at test time a video is drawn from a different distribution. I am unsure how much this sort of an evaluation scheme affects RAD's results, which should produce much better numbers than reported. I am giving the benefit of doubt here to the authors and hoping that they have done a fair evaluation for the baseline RAD method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "46d63rAXqk", "reviewer": "Reviewer_LDpb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- To me, the final experiment result on ProcGen is impressive.\n- Supplementary material shows that the authors seem to have digged into the experiment data with some depth. \n\nWeakness:\n- The writing is not so clean. Needs polishing or even rewriting.\n  - The paper introduction is quite vague and covers very general ideas, lacking specificity and advantage to the proposed approach; \n  - In Eq (5) and (6), there abruptly comes the Lagrange multiplier. Many other places also show that the writing is very rough.\n- The results on DeepMind Control Suite are somewhat marginal improvement. \n- While the results on ProcGen are impressive, the analysis and interpretation into these results are somewhat thin. Because the method involves many differences in comparison to the baselines, it is unclear whether the proposed main idea, mutual information loss, is the source of the major contributions. To me, the breakdown of contributions of each design choice (esp. including fair comparison with baselines with the same type and level of data augmentation) is important. ", "summary_of_the_paper": "This paper aims to learn robust representation from the replay buffer for reinforcement learning. The key idea is to leverage the concept of mutual information and the InfoNCE tool to compute the mutual information as a regularizer (a.k.a. DRIBO loss). The authors conducted experiments on some standard benchmarks (DeepMind Visual Control Suite and ProcGen). ", "summary_of_the_review": "Although there might be some good core contribution, the paper in its current state is not quite ready for publication. See details in my Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a method that improves the representation learned by RL agents, making them more robust to visual distractions. In particular, their approach proposes to use mutual information between two views as a proxy for that objective. This is clearly a borderline paper that required many discussions among the reviewers and the authors. The reviewers mention that the approach is novel, addresses an important problem of robustness in RL and some of the experiments provided are impressive. On the other hand, the reviewers point out that the baselines seem to achieve lower results than previously reported, writing could be improved and some of the results don't show significant improvement over baselines. 

Given that some of the results cause confusion around the evaluation protocol (it's still not 100% clear why the performance of baselines is lower than expected) and other doubts expressed by the reviewers, I encourage the authors to continue working on the paper and resubmit. I believe that with a little bit of extra work and clarifications this can be a very strong submission.
Decision: Reject