Learning Representations for Pixel-based Control: What Matters and Why?
Authors: Manan Tomar
Utkarsh Aashu Mishra
Amy Zhang
Matthew E. Taylor
Venue: Accepted by TMLR ()

Abstract: Learning representations for pixel-based control has garnered significant attention recently in reinforcement learning. A wide range of methods have been proposed to enable efficient learning, leading to sample complexities similar to those in the full state setting. However, moving beyond carefully curated pixel data sets (centered crop, appropriate lighting, clear background, etc.) remains challenging. In this paper, we adopt a more difficult setting, incorporating background distractors, as a first step towards addressing this challenge. We start by exploring a simple baseline approach that does not use metric-based learning, data augmentations, world-model learning, or contrastive learning. We then analyze when and why previously proposed methods are likely to fail or reduce to the same performance as the baseline in this harder setting and why we should think carefully about extending such methods beyond the well curated environments. Our results show that finer categorization of benchmarks on the basis of characteristics like density of reward, planning horizon of the problem, presence of task-irrelevant components, etc., is crucial in evaluating algorithms. Based on these observations, we propose different metrics to consider when evaluating an algorithm on benchmark tasks. We hope such a data-centric view can motivate researchers to rethink representation learning when investigating how to best apply RL to real-world tasks. Code available: https://github.com/UtkarshMishra04/pixel-representations-RL

Published in Transactions on Machine Learning Research (01/2023)
Learning Representations for Pixel-based Control: What Matters and Why?
Manan Tomar" University of Alberta Amii
Utkarsh A. Mishra University of Alberta Amii
Amy Zhang
FAIR, Menlo Park University of California, Berkeley
Matthew E. Taylor
University of Alberta
Amii
Reviewed on /ttptt/www....r./oo.oo/////////e
Abstract
Learning representations for pixel-based control has garnered significant attention recently in re- inforcement learning. A wide range of methods have been proposed to enable efficient learning, leading to sample complexities similar to those the full state setting. However, moving beyond carefully curated pixel data sets (centered crop, appropriate lighting, clear background, etc.) remains challenging. In this paper, we adopt a more difficult setting, incorporating background distractors, as first step towards addressing this challenge. We start by exploring a simple baseline approach that does not metric-based learning, data augmentations, world-model learning, or contrastive learning. We then analyze when and why previously proposed methods are likely to fail or reduce to the same performance as the baseline in this harder setting and why we should think carefully about extending such methods beyond the well curated environments. Our results show that finer categorization of benchmarks on the basis of characteristics like density of reward, planning horizon of the problem, presence of task-irrelevant components, etc., is crucial in evaluating algorithms. Based on these observations, we propose different metries to consider when evaluating an algorithm on benchmark tasks. We hope such data-centric view motivate researchers to rethink repre- sentation learning when investigating how to best apply RL to real-world tasks. Code available: https: www.tu...oo/ttteeeeeeeee rshtii shra / reppxeellrerreeetttee ons-RL.
1 Introduction
Learning useful representations for downstream tasks is key component for success rich observation environments (Du et al., 2019; Mnih et al., 2015; Silver al., 2017: Wahlström al., 2015; Watter al., 2015). Consequently. significant amount of work proposes various representation learning objectives that can be tied to the original reinforcement learning (RL) problem. Such auxiliary objectives include the likes of contrastive learning losses (Oord et al., 2018; Laskin et al., 2020b; Chen et al., 2020), state similarity metrics like bisimulation policy similarity (Zhang et al., 2020b;a; Agarwal et al., 2021a), and pixel reconstruction losses (Jaderberg et al., 2016; Gelada et al., 2019; Hafner et al., 2020; 2019). On a separate axis, data augmentations have been shown to provide huge performance boosts
Equual Contribution. Corresponding author: manan. tomm r 3gma c.cm

---

Published in Transactions on Machine Learning Research (01/2023)
Methods: Baseline
Baseline-v0
Contrastive
Metric
SSL w/o Contrastive
Reconstruction
Probability of Rank 1 %)
100
80-
60
20
C2
C3
C5 Dataset Categories
C4
C6
C7
CB
Figure 1: Comparing pixel-based RL methods across finer categorizations of cvaluation benchmarks. Each category 'Cx" denotes diffcrent data-centric properties of the evaluation benchmark (e.g.. CI refers to discrete action, dense reward, without distractors, and with data randomly cropped (Kostrikov et al., 2020; Laskin et al,, 2020a)). Exact descriptions of each category and the algorithms are provided in Table and Table S. Baseline-vO refers to applying the standard deep RL agent (e.g., Rainbow DQN (van Hasselt et al., 2019) and SAC (Haarmoja el al,, 2018)); Baseline refers to adding reward and transition prediction to baseline-vO, described in Section 3; Contrastive includes algorithms such as PI-SAC Lec al. (2020b) and CURL Laskin et al. (2020b); Metric denotes the state metric losses such as DBC (Zhang al., 2020b); SSL w/o Contrastive includes algorithms such as SPR (Schwarzer et al., 2020); Reconstruction includes DREAMER (Hafner et al., 2020) and TIA (Fu et al., 2021). For given method, always consider the best performing algorithm. Every method leads varied performance across data categories, making comparison which an average across all categories highly uninformative.
when learning to control from pixels (Laskin et al., 2020a; Kostrikov et al., 2020). Each of these methods has been shown to work well for particular settings and hence displayed promise to be part of a general purpose representation learning toolkit. Unfortunately, these methods were proposed with different motivations and tested on different tasks, making the following question hard to answer:
What really matters when learning representations for downstream control tasks?
Learning directly from pixels offers much richer applicability than when learning from carefully constructed states. Consider the example of a self-driving car, where it is nearly impossible to provide a complete state description of the position and velocity of all objects of interest, such as road edges, highway markers, other vehicles, etc. In such real world applications, learning from pixels offers a much more feasible option. However, this requires algorithms that can discern between task-relevant and task-irrelevant components in the pixel input, i.e., learn good representations. Focusing on task-irrelevant components can lead to brittle or non-robust behavior when put in slightly different environments. For instance, billboard signs over buildings in the background have no dependence on the task in hand while self-driving car tries to change lanes. However, if such task-irrelevant components are not discarded, they can lead to sudden failure when the car drives through different environment, say forest where there are buildings or billboards. Avoiding brittle behavior is therefore key to efficient deployment ment of artificial agents in the real world.
There has been a lot of work recently that tries to learn efficiently from pixels. A dominant idea throughout prior work has been that of attaching an auxiliary loss to the standard RL objective, with the exact mechanics of the loss varying for each method (Jaderberg et al., 2016; Zhang et al., 2020b; Laskin et al., 2020b). A related line of work learns representations by constructing world models directly from pixels (Schmidhuber, 2010; Oh et al., 2015; Ha & Schmidhuber, 2018; Hafner et al., 2020). We show that these work well when the world model is simple. However, as the world model gets even slightly more complicated, which is true of the real world and imitated in simulation with the use of video distractors (Zhang et al., 2018; Kay et al., 2017; Stone et al., 2021), such approaches can fail. For other methods, it is not entirely clear what componen/'s in auxiliary objectives can lead to failure when changing the environment, thus making robust behavior hard to achieve. Another distinct idea is of using data augmentations (Laskin et al., 2020a; Kostrikov al., 2020) over the original observation samples, which seem to be quite robust across different environments. However, as we will show, a lot of the success of data augmentations is an artifact of how the benchmark environments save data, which is not replicable in the real world (Stone et al., 2021), thus resulting in failure?. It is important to note that some of these methods are not designed for robustness but instead for enhanced performance on particular benchmarks. For instance, the ALE (Bellemare et al., 2013) benchmark involves simple, easy to model objects, and it becomes hard to discern if methods that perform well are actually good candidates for answering "what really matters for robust learning in the real world."
Contributions. In this paper, we explore the major components responsible for the successful application of various representation learning algorithms. Based on recent work in RL theory for learning with rich observations (Farahmand
For ALE we the performance of DREAMER after IM steps, whereas for DMC we consider the performance after 500k steps. I1 also hard to pick exactly which data augmentation will work for a particular environment task (Raileanu al., 2020; Grigshy Qi, 2020)
2

---

Published in Transactions on Machine Learning Research (01/2023)
et al., 2017; Ayoub et al., 2020; Castro, 2020), we hypothesize certain key components to be responsible for sample efficient learning. We test the role these play in previously proposed representation learning objectives and then consider an exceedingly simple baseline (see Figure 2) which takes away the extra "knobs" and instead combines two simple but key ideas, that of reward and transition prediction. We conduct experiments across multiple settings, including the MuJoCo domains from DMC Suite (Tassa et al., 2018) with natural distractors (Zhang et al., 2018; Kay et al., 2017; Stone et al., 2021), and Atari100K Kaiser et al. (2019) from ALE (Bellemare et al., 2013). Following this, we identify the failure modes of previously proposed objectives and highlight why they result in comparable or worse performance than the considered baseline. Our observations suggest that relying on a particular method across multiple evaluation settings does not work, as the efficacy varies with the exact details of the task, even within the same benchmark (see Figure 1). We note that finer categorization of available benchmarks based on metrics like density of reward, presence of task-irreeevant components, inherent horizon of tasks, etc., play a crucial role in determining the efficacy of a method. We list such categorizations as suggestions for more informative future evaluations. The findings of this paper advocate for a more data-centric view of evaluating RL algorithms (Co-Reyes al., 2020), largely missing in current practice. We hope the findings and insights presented in this paper can lead to better representation learning objectives for real-world applications.
2 Related Work
Prior work on auxiliary objectives includes the Horde architecture (Sutton et al., 2011), UVFA (Schaul et al., 2015) and the UNREAL agent (Jaderberg et al., 2016). These involve making predictions about features or pseudo-rewards, however only the UNREAL agent used these predictions for learning representations. Even so, the benchmark environments considered there always included only task-relevant pixel information, thus not pertaining to the hard setting we consider in this work. Representations can also be fit so as to obey certain state similarities. If these state metrics preserve the optimal policies and are easy to learn/given priori, such a technique can be very useful. Recent works have shown that we can learn efficient representations either by learning the metrics like that in bisimulation (Ferns et al., 2011; Zhang et al., 2020b;a; Biza et al., 2020), by recursively sampling states (Castro et al., 2021) or by exploiting sparsity in dynamics (Tomar et al., 2021). Data augmentations modify the input image into different distinct views, each corresponding to a certain type of modulation in the pixel data. These include cropping, color jitter, flip, rotate, random convolution, etc. Latest works (Laskin et al., 2020a; Yarats al., 2021b) have shown that augmenting the state samples in the replay buffer with such techniques alone can lead to impressive gains when learning directly from pixels. Recently, Stone et al. (2021) illustrated the advantages and failure cases of augmentations. Contrastive learning involves optimizing for representations such that positive pairs (those coming from the same sample) are pulled closer while negative pairs (those coming from different samples) are pushed away (Oord et al., 2018; Chen et al., 2020). The most widely used method to generate positive/negative pairs is through various data augmentations (Laskin et al., 2020b; Schwarzer et al., 2020; Lee et al., 2020b; Stooke et al., 2021; Yarats et al., 2021a). a)). However, temporal structure can induce positive/negative pairs as well. In such a case, the positive pair comes from the current state and the actual next state while the negative pair comes from the current state and any other next state in the current batch (Oord et al., 2018). Other ways of generating positive/negatiie pairs can be through learnt state metrics (Agarwal et al., 2021a) or encoding instances (Hafner et al., 2020). Another popular idea for learning representations is learning world models (Ha & Schmidhuber, 2018) in the pixel space. This involves learning prediction models of the world in the pixel space using a pixel reconstruction loss (Gelada et al., 2019; Hafner et al., 2019; 2020). Other methods that do not explicitly learn a world model involve learning representations using reconstruction based approaches like autoencoders (Yarats et al., 2019).
Quite a few papers in the past have analysed different sub-topics in RL through large scale studies. Engstrom et al. (2019) and Andrychowicz et al. (2020) have focused on analysing different policy optimization methods with varying hyperparameters. Our focus is specifically pixel-based environments. Henderson et al. (2018) showed how RL methods in general can be susceptible to lucky seedings. Recently, Agarwal et al. (2021b) proposed statistical metrics for reliable evaluation. Despite having similar structure, our work is largely complimentary to these past investigations. Babacizadeh et al. (2020) analysed reward and transition but only focused on the Atari 200M benchmark and pixel reconstruction methods. In comparison, our work is spread across multiple evaluation benchmarks, and our results show that reconstruction can be fine technique only in particular benchmark category.
representation learning methods that improve sample efficiency in
3

---

Published in Transactions on Machine Learning Research (01/2023)
Z 1
Transition Lass
Reward Loss
Encoded
fo
8t++1
LT
L
Stop Gradient
fo
P(-lee, ax)
R((m)))
St
86+1
ft
State Prediction
Reward Prediction
Loss
Loss
Observation
Adder
Contte
at
Action
Figure 2: (Leftt) Baseline for control over pixels. We employ two losses besides the standard actor and critic losses, one being a reward prediction loss and the other a latent transition prediction loss. The encoded state st is the Iearnt representation. Gradicnts from both the transition/reward prediction and the critic are used to Iearn the representation, whereas the actor gradients are stopped. In the ALE setting. the actor and critic losses are replaced by a Rainbow DQN loss (van Husselt et al., 2019). (Right) Natural Distractor in the background for standard DMC setting (left column) and custom off-center setting (right column). More details about the distractors can be found in Appendix 2.
3 Method
We model the RL problem using the framework of contextual decision processes (CDPs), a term introduced in Krishnamurthy et al. (2016) to broadly refer to any sequential decision making task where an agent must act on the basis of rich observations (context) 24 to optimize long-term reward. The true state of the environment is not available and the agent must construct on its own, which required for acting optimally on the downstream task. Furthermore, the emission function which dictates what contexts are observed for given state is assumed only inject noise that is uncorrelated the task in hand, i.e. it only changes parts of the context that are irrelevant the task (Zhang al., 2020b; Stone al., 2021). Consider again the example of people walking the sides of road while self-driving car changes lanes. Invariance parts of the context that have no dependence the task, e.g. people in the background, is an important property for any representation learning algorithm since we cannot expect all situations to remain exactly the same when learning in the real world. A more detailed description of the setup and all the prior methods used is provided in Appendix 1.
We start by exploring the utility of two fundamental components in RL, that of reward and transition prediction, in learning representations. lot of prior work has incorporated these objectives either individually or in the presence of more nuanced architectures. Here, one by one. Particularly, we use simple soft actor-critic setup with an embedding function So X + S (similar to SAC-AE (Yarats et al., 2019)) as the base architecture, and attach the reward and transition prediction modules to it (See Figure 2). We define the transition prediction by (( 0²) and the reward prediction by R((++1) such that, $c+1 = ((s,, ax) and f.- = R()41). Note that the transition network is over the encoded state š, = fo( and not over the observations se (Lee al., 2020a). The overall auxiliary loss function is thus defined as follows:
aim is start with the most basic components and establish their importance
Baseline (S(+1 -s Pasi-P(een (s))
(1)
Transition prodiction loss
Reward prediction loss
Unless noted otherwise, we call this architecture the baseline for all our experiments. Details about the implementa- tion, network sizes and all hyperparameters is provided in Appendix 3 and Appendix 4 (Table 3) respectively.
Almost all methods considered in this paper can be boiled down to using two key ideas: that of reward and transition prediction. These can be instantiated in various ways, hence giving rise to differently performing methods. We therefore start with the simplest possible instantiation these ideas, which corresponds to predicting the reward and the next encoded state.
Throughout the paper, we broadly categorize approaches into the following (based on how they were initially motivated):
baseline (that simply predicts reward and next latent state),
contrastive-based.
metric-based,
4

---

Published in Transactions on Machine Learning Research (01/2023)
non-contrastive, self supervised based
reconstruction-basee
Depending on the environment under consideration, each method category can be instantiated into a particular algorithm. For example, contrastive learning applied to MuloCo environments is represented by DrQ. Note that even though we might have the same method category being applied to all possible environments, it usually not possible to apply the same algorithm across all environments. This distinction most apparent when switching from Atari to MuloCo. We therefore discuss the reward and transition baselines only on the MuJoCo domains for simplicity and then describe the Atari results in separate section to avoid confusion.
4 Empirical Study
In this section, we analyze the baseline architecture across six DMC tasks: Cartpole Swingup, Cheetah Run, Finger Spin, Hopper Hop, Reacher Easy, and Walker Walk. A common observation in our experiments is that the baseline is able to reduce the gap to more sophisticated methods significantly, sometimes even outperforming them in certain cases. This highlights that the baseline might serve as a stepping stone for other methods to build over. We test the importance of having both the reward and transition modules individually, by removing each of them one by one.
4.1 Reward Prediction
Figure 3 (left) shows comparison of 'with without reward prediction'. All other settings are kept unchanged and the only difference is the reward prediction. When the reward model is removed, there remains no grounding objective for the transition model. This results in a representation collapse as the transition model loss is minimized by the trivial representation which maps all observations to the same encoded state leading to degraded performance. This hints at the fact that without a valid grounding objective (in this case from predicting rewards), learning good representations can be very hard. Note that it not the case that there is no reward information available to the agent, since learning the critic does provide enough signal to learn efficiently when there are no distractions present. However, in the presence of distractions the signal from the critic be extremely noisy since it is based on the current value functions, which are not well developed in the initial stages of training. One potential fix for such collapse is to not use the standard maximum likelihood based approaches for the transition model loss and instead use contrastive version of the loss, which has been shown to learn general representations in the self-supervised learning setting. We test this later in the paper and observe that although it does help prevent collapse, the performance is still heavily inferior to when we include the reward model. Complete performances for individual tasks are shown in Appendix 8.
Linear Reward Predictor. We also compare to the case when the reward decoder is a linear network instead of the standard layer MLP. We see that performance decreases significantly in this case as shown in Figure 3 (middle), but still does not collapse like in the absence of reward prediction. We hypothesize that the reward model is potentially removing useful information for predicting the optimal actions. Therefore, when it is attached directly to the encoded state, i.c., in the linear reward predictor case, it might force the representation to only preserve information required to predict the reward well, which might not always be enough to predict the optimal actions well. For instance, consider robot locomotion task. The reward in this case only depends on one variable, the center of mass, and thus the representation module would only need to preserve that in order to predict the reward well. However, to predict optimal actions, information about all the joint angular positions and velocities is required, which might be discarded if the reward model is directly attached to the encoded state. This idea is similar to why contrastive learning objectives in the self-supervised learning setting always enforce consistency between two positive/negative pairs after projecting the representation to another space. It has been shown that enforcing consistency in the representation space can remove excess information, which hampers final performance (Chen et al., 2020). We indeed see a similar trend in the RL case as well.
4.2 Transition Prediction
Similarly, Figure 3 (right) shows comparison of 'with vs without transition prediction'. The transition model loss enforces temporal consistencies among the encoded states. When this module is removed, observe a slight dip in
5

---

Published in Transactions on Machine Learning Research (01/2023)
w/ Reward
w/o Reward
Non-linear Rew Dec
Linear Rew Dec
w/ Transition
w/o Transition
Normalized Score
0.4
Normllzed aiizellzed Score
0.4
Normalized Score
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Environment Steps (x109)
Environment ent Steps (x10*)
Environment Steps (x109)
Figure 3: Baseline Ablations. Average normalized performance across six standard domains from DMC. Mean and std err. for 5 runs. Left plot: Baseline with without reward prediction Middle plot: Baseline with non-linear vs lincar reward predictor/decnder. Right plot: Baseline with without transition prediction.
Baceline
RAD Rotate
Baseline
DBC
Crop
Baseline Crop
Baseline
PI-SAC (Contrastive) Contrastive- Ree
RAD Flip
SPR (SSL w/o Contrastive!
Normalized Score
Normalized Score
Normelized Score
0.4
0.3
0.3
0.4
0.2
0.7
0.2
0.1
0.0
0.0
0.0
0.1
0.3
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x20*)
Environment Steps (x10%)
Environment Steps (x10%)
Figure Baseline Ablations. Average normalized performance across standard domains from DMC. Mean and std err. for 5 runs. Left plot: Baseline va state metric losses (DBC (Zhang al., 2020b)). The performance of baseline compared with bisimulation metrics employed by DBC Middle plot: Data Augmentations. Cropping removes irrelevant segments while flip and rotate not, performing similar to the baseline. Baseline with random crop performs oqually as good as RAD. Right plot: Contrastive and SSL w/o Contrastive. We replace the transition loss bascline with contrastive version (Contrastive- + Rew). Further, we consider simple contrastive (PI-SAC (Lee et al., 2020b)) and SSL w/o contrastive (variant of SPR (Schwarzer et al., 2020) for DMC) losses as well.
performance across most tasks, with the most prominent drop in cartpole as shown in Appendix 8.1 (Figure 16). This suggests that enforcing such temporal consistencies in the representation space indeed an important component for robust learning, but not a sufficient one. To examine if the marginal gain is an artifact of the exact architecture used, we explored other architectures in Appendix 8.2 but did not observe any difference in performance.
4.3 Connections to Value-Aware Learning
The baseline introduced above also resembles a prominent idea in theory, that of learning value aware models (Farahmand et al., 2017; Ayoub et al., 2020). Value-aware learning advocates for learning model by fitting it to the value function of the task in hand, instead of fitting it to the true model of the world. The above baseline can be looked at doing value aware learning in the following sense: the grounding to the representation is provided by the reward function, thus defining the components responsible for the task in hand and then the transition dynamics are learnt only for these components and not for all components in the observation space. There remains one crucial difference though. Value aware methods Iearn the dynamics based on the value function (multi-step) and not the reward function (1-step), since the value function captures the long term nature of the task in hand. To that end, we also test a more exact variant of the value-aware setup where we use the critic function as the target for optimizing the transition prediction, both with and without a reward prediction module (Table 1). Complete performances are provided in Appendix 8.8. We see that the value aware losses perform worse than the baseline. A potential reason for this could be that since the value estimates are noisy when using distractors, directly using these as targets inhibits learning a stable latent state representation. Indeed, more sophisticated value-aware methods such as in Temporal Predictive Coding (Nguyen et al., 2021) lead to similar scores as the baseline.
Table I: Truly value objectives. We report average final score after 500K steps across six standard domains from DMC.
Baseline
Value-aware (w/ reward)
Value-aware (w/o reward)
Average Scores
0.42 + 0.02
0.36 + 0.03
0.23 + 0.03
PBC (Zhang et al., 2020b) performance data taken from their publication.

---

Published in Transactions on Machine Learning Research (01/2023)
5 Comparison
So far, we have discussed why the two modules we identify as being vital for minimal and robust learning are actually necessary. Now we ask what other components could be added to this architecture which might improve performance, as has been done in prior methods. We then ask when do these added components actually improve performance, and when do they fail. More implementation details are provided in Appendix 3.
Metric Losses. Two recent works that are similar to the baseline above are DBC (Zhang al., 2020b) and MiCO (Castro al., 2021), both of which learn representations by obeying distance metric. DBC learns the metric by estimating the reward and transition models while MiCO uses transition samples to directly compute the metric distance. We compare baseline's 's performance with DBC as shown in Figure 4 (left). Note that without the metric loss, DBC is similar to the baseline barring architectural differences such as the use of probabilistic transition models in DBC compared to deterministic models in the baseline. Surprisingly, we observe that the performance of the baseline exceeds that of DBC. To double check, we ran a version of DBC without the metric loss. Again, the "without metric" version lead to superior performance than the "with metric" one (DBC).
Data Augmentations. A separate line of work has shown strong results when using data augmentations over the observation samples. These include the RAD (Laskin et al., 2020a) and DRQ (Kostrikov et al., 2020) algorithms, both of which differ very minimally in their implementations. We run experiments for three different augmentations- 'crop', 'flip', and 'rotate'. The 'crop' augmentation always crops the image by some shifted margin from the center. Interestingly, the image of the agent is also always centered, thus allowing 'crop' to always only remove background or task-irrelevant information and never remove the agent or task-relevant information. This essentially amounts to not having background distractors and thus we see that this technique performs quite well as shown in Figure 4 (middle). However, augmentations that do not explicitly remove the distractors, such rotate and flip, lead to similar performance as the baseline. This suggests that augmentations might not be helpful when distractor information cannot be removed, or when we do not know where the objects of interest lie in the image, something true of the real world. We test this by shifting the agent to the side, thus making the task-relevant components off-center and by zooming out i.e. increasing the amount of irrelevant information even after cropping. We see that performance of 'crop' drops drastically in this case, showcasing that most of the performance gains from augmentations can be attributed to how the data is collected and not the algorithm itself. Additional ablations are provided in Appendix 8.3.
Note that DrQ uses different cropping scheme than RAD, one which preserves all irrelevant information. This highlights that invariance to irrelevant information might not be the only reason for the success of augmentations. Two other potential reasons are: 1) cropping leads to explicitly modeling translational invariance in the network, and 2) how the DrQ augmentation affects Q estimation (both in the current and target networks). These two can act as potential confounders in understanding the benefits of augmentation accurately.
Table 2: RAD additional ablations. We report average final score after 500K steps across Cheetah Run and Walker Walk domains from DMC. This illustrates that the performance of augmentations is susceptible quality of data. Also, for the "Off-center Zoomed Out" setting, worth noting that performance of 'Baseline' without augmentations is more robust changes as compared augmentations.
Bascline
Baseline Crop
Bascline Flip
RAD Crop
RAD Flip
Standard
0.44 + 0.04
0.89 + 0.02
0.75 + 0.06
0.93 + 0.02
0.49 + 0.06
Off-center Zoomed Out 0.56 + 0.01
0.72 = 0.06
0.53 + 0.06
0.70 + 0.05
0.48 0.04
Contrastive and SSL w/o Contrastive Losses. A lot of recent methods also deploy contrastive losses (for example, CPC Oord et al. (2018) to learn representations, which essentially refers to computing positive/negative pairs and pushing together/pulling apart representations respectively. In practice, this can be done for any kind of loss function, such as the encoding function fo (Hafner et al., 2020), or using random augmentations (Laskin et al., 2020b; Lee et al., 2020b), so on and so forth. Therefore, we test a simple modification to the baseline, that of using the contrastive variant of the transition prediction loss than the maximum likelihood version. Specifically, the encoded current state 8; and the encoded next state 8:+1 form a positive pair while any other random state in the batch forms a negative pair with 8t. Such a loss can be written down in terms of the standard InfoNCE Oord al. (2018) objective, as in Eq. 2.
(2) where f(xz) includes a backbone network (typically a ResNet) and a projection network that maps the output of the backbone network to much smaller space. We see, in Figure 4 (right), that the contrastive version leads to
CContrastive Inf)N f(x++1)).
7

---

Published in Transactions on Machine Learning Research (01/2023)
inferior results than the baseline, potentially suggesting that contrastive learning might not add a lot of performance improvement, particularly when there is grounding available from supervised losses. A similar trend has been witnessed in the self-supervised literature with methods like SIMSIAM (Chen & He, 2021), BARLOW TWINS (Zbontar et al., 2021), and B YOL (Grill et al., 2020) getting similar or better performance than contrastive methods like SIMCLR (Chen et al., 2020). Complete performances are provided Appendix 8.5.
SPR (Schwarzer et al., 2020) known to be prominent algorithm in the ALE domain, leading to the best results overall. SPR deploys specific similarity loss for transition prediction motivated by BYOL (Grill et al., 2020). We follow the same setup and test variant of the baseline which uses the cosine similarity loss from SPR and test its performance on DMC based tasks. A general description of such kind of loss functions is provided in Eq. 3 while a more specific version is provided in the Appendix 1. We again show in Figure 4 (right) that there is very little or no improvement in performance as compared to the baseline performance. This again suggests that the same algorithmic idea can have an entirely different performance just by changing the evaluation setting" (ALE to DMC).
Pt
Pt+1
Luw Contrastive
(3)
Ipel
||pe+1||2
P:- - =x((())) is obtained by passing the encoder output through a prediction network, denoted by h.
Learning World Models. We test DREAMER (Hafner et al., 2020), a state of the art model-based method that Iearns world models though pixel reconstruction on two settings, with and without distractors. Although the performance in the "without distractors" case is strong, we see that with distractors, DREAMER fails on some tasks, while performing inferior to the baseline in most tasks (see Figure 5).
This suggests that learning world models through reconstruction might only be good idea when the world models are fairly simple to learn. If world models are hard to learn, as is the case with distrac- tors, reconstruction based learning can lead to severe divergence that results in no learning at all. We also compare against the more re- cently introduced method from Fu et al. (2021). Their method, called TIA(Fu et al., 2021) incorporates several other modules in addition to DREAMER and learns a decoupling between the distractor back- ground and task relevant components. We illustrate the performance of each of the above algorithms in Figure along with version where add full reconstruction loss to the baseline. Interestingly, TIA still fails to be superior to the baseline, particularly for simpler domains like Cartpole. Complete performances are provided in Appendix 8.6
Baseline
TIA dist. DREAMER w/o dist.
Full Recon
DREAMER w/ dist
0.6
Normalized Score
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10*)
Figure 5: Pixel reconstruction. Average normalized per- formance across six DMC domains with distractors. Base- line achieves better performance than SOTA methods like DREAMER and TIA 5.
Relevant Reconstruction and Sparse Rewards. Since thus far we only considered dense reward based tasks, using the reward model for grounding is sufficient learn good representations. More sophis- ticated auxiliary tasks considered in past works include prediction of ensemble of value networks, prediction of past value functions, prediction of value functions of random cumulants, and observation reconstruction. However, in the sparse reward case, grounding on only the reward model or on past value functions can lead to representation collapse if the agent continues to receive zero reward for a long period of time. Therefore, in such cases where good exploration is necessary, tasks such as observation reconstruction can help prevent collapse. Although this has been shown to be an effective technique in the past, we argue that full reconstruction can still harm the representations in the presence of distractors. Instead, we claim that reconstruction of only the task relevant components in the observation space results in learning good representations (Fu et al., 2021), especially when concerned with realistic settings like that of distractors. We conduct simple experiment to show that in the sparse reward case, task-relevant reconstruction is sufficient for robust performance. We show this in Figure 6 along with
"The SPR version without augmentations actually uses two separate ideas for improvement in performance, a cosine similarity transition prediction loss and separate convolution encoder for the transition network, making it hard to attribute gains over the basc DER (yan Hasselt et al., 2019) 9) to just transition loss.
*TIA (Fu et al., 2021) performance data taken from their publication.
6 Part Recons, Figure 6 amounts reconstructing the DMC agent over a solid black background.
8

---

Published in Transactions on Machine Learning Research (01/2023)
performance of baseline and augmentations. Of course, how one should come up with techniques that differentiate
between task-relevant and task-irrelevant components in the observations, remains an open question ?. Additional
ablations are provided in Appendix 8.6.
Atari 100K. We study the effect of techniques discussed thus far for the Atari 100K benchmark, which involves 26 Atari games and compares performance relative steps 400K frames. We consider the categorization proposed by
Baseline
RAD Flip
serman-achieved scores 100K
Part Recons
DREAMER
Crop
Bellemare et al. (2016) based the nature of reward (dense, human optimal, score exploit and sparse) and implement two versions of the baseline algorithm, one with both the transition and reward prediction modules and the other with only reward prediction. Our average results over all games show that the baseline performs comparably to CURL (Laskin et al., 2020b), SimPLe (Kaiser et al., 2019), DER (van Hasselt al., 2019), and OTR (Kielak, 2020) while being quite inferior to DRQ$ (Kostrikov et al., 2020; Agarwal et al., 2021b) and SPR (Schwarzer et al., 2020). Since our implementation of the baseline is over the DER code, similar performance to DER might suggest that the reward and transition prediction do not help much in this benchmark. Note that ALE does not involve the use of distractors and so learning directly from the RL head (DQN in this case) should be enough to encode information about the reward and the transition dynamics in the representation. This comes stark contrast to the without distractors case in DMC Suite, where transition and reward prediction still lead to better performance. Such differences can also be attributed to the continuous discrete nature of DMC and ALE benchmarks. More interestingly, we find that when plotting the average performance for only the dense reward environments, the gap in performance between DER and SPR/DRQ decreases drastically. Note that SPR builds over DER but DRQ builds over OTR.
1.0
Normalized Score
0.6
0.4
0.2
0.0
0.2
0.4
Environment Steps (x106)
Figure Reconstruction and augmentations for sparse settings. Normalized performance for ball-in-cup catch do- main from DMC.
We further delve into understanding the superior performance of SPR and DRQ. In particular, SPR combines a cosine similarity transition prediction loss with data augmentations. To understand the effect of each of these individually, we run SPR without data augmentations, referring this version by SPR**' We see that SPR** leads performance similar to the baseline and the DER agent, suggesting that such a sule-super loss may not lead to gains when run without data augmentations. Finally, we take the DER agent and add data augmentations to it (from DRQ). This is shown as DER+ + AUG in Figure 7. We see that this leads to collapse, with the worst performance across all algorithms. Note that DRQ builds over OTR and performs quite well whereas when the same augmentations are used with DER, which includes a distributional agent in it, we observe a collapse. This again indicates that augmentations can change data in a fragile manner, sometimes leading to enhanced performance with certain algorithms, while failing with other algorithms. Segregating evaluation of algorithms based on these differences is therefore of utmost importance. We show the individual performance all 25 games in Appendix 8.5 (Table 7).
Mean
Median
Mean
Median
Rew+Trans
Rew+Trans Rew Only SPR
Rew Only
SPR- SPR Dro CURL OTR DER+Aug
SPR
DrQ
CURL
OTR
AER+Aug
DER
DER
SimPLe
SimPLe
0.25 0.50 0.75
0.15 0.30
0.45
0.05 0.12 0.18 0.24
0.05 0.10 0.15
Human Normalized Score on all games
Human Normalized Score on dense games
Figure 7: Atari 100K. Human nommalize performance (mean/median) across 25 games from the Atari 100K benchuark. Mean and 95% confidence interval for runs. Left plot: Comparison for all 25 games. Right plot: Comparison for only dense reward games (7 games from Table 7).
As also evident by TIA's (Fu al., 2021) performance for DMC ball-in-cup catch experiments. *We use the DRQ(c) version from Agarwal al. (2021b) for fair evaluation and denote DRQ. "Note that this different from the SPR without augmentations version reported Schwarzer al. (2020) since that version uses dropout as well which not fair comparison.

---

Published in Transactions on Machine Learning Research (01/2023)
6 Discussion
The above description of results on DMC Suite and Atari 100K point to very interesting observation, that evaluation of different algorithms is very much correlated with finer categorization of the evaluation benchmark, and not the whole benchmark itself. Specifically, focusing on finer categorizations such as density of reward, inherent horizon of the problem, presence of irrelevant and relevant task components, discreteness vs continuity of actions etc. is vital in recognizing certain algorithms are indeed better than others. Figure stands as a clear example of such discrepancies. These observations pave the way for better evaluation protocol for algorithms, one where we rank algorithms for different categories, each governed by specific data-centric property of the evaluation benchmark. Instead of saying that algorithm X is better than algorithm Y in benchmark Z, our results advocate for evaluation methodology which claims algorithm X to be better than algorithm Y in dense reward, short horizon problems (considered from benchmark Z), i.c. enforeing less emphasis on the benchmark itself and more on certain properties of subset of the benchmark. Having started with the question of what matters when learning representations over pixels, our experiments and discussion clearly show that largely is the data-centric properties of the evaluation problems that matter the most.
7 Conclusion
In this paper we explore what components in representation learning methods matter the most for robust performance. As a starting point, we focused on the DMC Suite with distractors and the Atari 100k benchmark. Our results show that simple baseline, one involving a reward and transition prediction modules can be attributed to a lot of performance benefits in DMC Suite with distractors. We then analysed why and when existing methods fail to perform as good or better than the baseline, also touching on similar observations the ALE simulator. Some of our most interesting findings are as follows:
Pixel reconstruction is sound technique in the absence of clutter in the pixels, but suffers massively when distractors are added. In particular, DREAMER and adding simple pixel reconstruction loss leads to worse performance than the baseline in DMC Suite (Figure 5).
Contrastive losses in and of itself do not seem provide gains when there a supervised loss available in place of it. We observe that replacing the supervised state prediction loss of the baseline by the InfoNCE contrastive loss does not lead to performance improvements over the baseline in DMC Suite (Figure 4 right plot). On the other hand, using contrastive losses with data augmentations can lead to more robust improvements (Leee et al., 2020b; Fan & Li, 2021).
Certain augmentations ('crop") do well when data is centered while dropping in performance when data is off-center or when cropping fails to remove considerable amounts of task-irrelevant information. Other augmentations ('flip' and 'rotate") show the opposite behavior (RAD ablations on DMC Suite in Table 2).
SSL w/o contrastive losses does not provide much gains when used alone. With data augmentations, they lead to more significant gains. For Atari100k, Figure 7 shows that SPR, state of the art non contrastive method leads to similar performance as the base DER agent when used without data augmentations (denoted by *P Using the SPR inspired loss in DMC Suite also did not lead to gains over the baseline (in Figure 4 right plot).
Augmentations are susceptible to collapse in the presence of distributional Q networks. Figure 7 shows that 'crop' and 'intensity" augmentations added to the DER agent lead to complete failure in performance in Atari100k.
These results elicit the observation that claiming dominance over other methods for an entire benchmark may not be informative evaluation methodology. Instead, focusing the discussion to more data-centric view, one where specific properties of the environment are considered, forms the basis of much more informative evaluation methodology. We argue that as datasets become larger and more diverse, the need for such an evaluation protocol would become more critical. We hope this work can provide valuable insights in developing better representation learning algorithms and spur further discussion in categorizing evaluation domains in more complex scenarios, such with real world datasets and over wider class of algorithmic approaches.
10

---

Published in Transactions on Machine Learning Research (01/2023)
8
Acknowledgement
Part of this work has taken place in the Intelligent Robot Learning (IRL) Lab the University of Alberta, which is supported in part by research grants from the Alberta Machine Intelligence Institute (Amii); Canada CIFAR Al Chair, Amii; Compute Canada; Huawei; Mitacs; and NSERC.
References
Agarwal, R., Machado, M. C., Castro, S., and Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a.
Agarwal, R., Schwarzer, M., Castro, P. S.,, Courville, A., and Bellemare, M. G. Deep reinforcement learning the edge of the statistical precipice. arXiv preprint arXiv:2 13264, 2021b.
Andrychowicz, M., Raichuk, A., Staúczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al. What matters in on-policy reinforcement learning? a large-scale empirical study. ar preprint urXiv:2006.05990 2020.
Ayoub, A., Jia, Z.., Szepesvari, C., Wang, M., and Yang, L. Model-based reinforcement learning with value-targeted regression. In International Conference Machine Learning, pp. 463-4744 PMLR, 2020.
Babaeizadeh, M., Saffar, M. T., Hafner, D., Erhan, D., Kannan, H., Finn, C., and Levine, S. On trade-off's of image prediction in visual model-based reinforcement learning. 2020.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29:1471-1479, 2016.
Bellemare, M. G., Naddaf, Y.. Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 27:233.279,0013.
Biza, O., Platt, R., van de Meent, J.-W., and Wong, L. L.. Learning discrete state abstractions with deep variational inference. arXiv preprint arXiv:2003.00 04300, 2020.
Castro, S. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Castro, P. S., Kastner, T., Panangaden, P., and Rowland, M. Mico: Learning improved representations via sampling- based state similarity for markov decision processes, CoRR, abs/2106.08229, 2021. URL https::///// http::///xxvvvvrie abs/ / /20.6.66..... 08229.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representa- tions. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Chen, X. and He, K. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Co-Reyes, J. D., Sanjeev, S., Berseth, G., Gupta, A., and Levine, Ecological reinforcement learning. arXiv preprint urXiv:2006. 12478, 2020.
Du, S., Krishnamurthy, A., Jiang, N., Agarwal, A., Dudik, M., and Langford, J. Provably efficient rl with rich observations via latent state decoding, In International Conference Machine Learning, pp. 1665-1674, PMLR, 2019.
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. Implementation matters deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Fan, J. and Li, W. Robust deep reinforcement learning via multi-view information bottleneck. arXiv preprint arXiv:2102. 13268, 2021.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Farahmand, A.m..,
Barreto, A., and Nikovski, D. Value-aware loss function for model-basee reinforcement learning, In Artificial Intelligence and Statistics, pp. 1486-1494, PMLR, 2017.
Ferns, N., Panangaden, P., and Precup, D. Bisimulation metrics for continuous markov decision processes. SIAM J. Comput., 40(6):1662-1711 December 2011. ISSN 0097-5397. doi: 10.1137/10080484X, URL https: / otoo..org/...13///77
Fu, X., Yang, G., Agrawal, P., and Jaakkola, T. Learning task informed abstractions. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3480-3491, PMLR, 18-24 Jul 2021. URL http::/ppoceeeeeeeeieeee mamiinggssmm mlr.
press/ vv39//13///////////// 0039/ff2211 ff2lb. html.
Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Bellemare, M. G. DeepMDP: Learning continuous latent space models for representation learning. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2170-2179. PMLR, 09-15 Jun 2019. URL https: httpttt////coceeeaeaee http:///ooceeddedeee ml pre.s.ppeessseee /
/ pttpss/v9/geeaaaaa 9a html.
Grigsby, J. and Qi, Y. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.06740, 2020.
Grill, J.-B., Strub, F, Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Ha, D. and Schmidhuber, J. World models. arXiv preprint 1rXhv:1803.1012,, 2018.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL https: httppppooeddnggeee//eee ngs press/ v8 ////0//aaaaaa haa.aoaaaaoajaaaaaaoaaoo.. 8b. html.
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019.
Hafner, D., Lillicrap. T., Ba, J., and Norouzi, M. Dream control: Learning behaviors by latent imagination. net forum? idd
International Conference Learning Representations, 2020. URL Setttps://operreeee S110TC4tDS.
Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. Mastering atari with discrete world models. In International Confer- ence on Learning Representations, 2021. URL https: htttpa://operreview wttp:/.eeee/oroueeeeeeeeee doaiddd ZbOu.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D.,, and Meger, D. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, Rainbow: Combining improvements deep reinforcement learning, In Thirty-second AAAI conference on artificial intelligence. 2018.
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. 7., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 1rX:v:1611.05397, 2016.
Kaiser, I.,, Babaeizadeh, M., Milos, P., Osinski, B., Camphell, H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, Levine, S., al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374 2019.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F, Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv: 2017.
Kielak, Importance of using appropriate baselines for evaluation of data-efficiency deep reinforcement learning for atari. arXiv preprint arXiv:2003. 10181, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2023)
Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, 1122004113649, 2020.
2004, 13649.
Krishnamurthy, A., Agarwal, A., and Langford, J. Pac reinforcement learning with rich observations. arXiv preprint arXiv:/602.02222, 2016.
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data.
In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19884-19895. Curran Associates, Inc., 2020a. URL htwp:: httpee///ccoccceeeeeeeee
Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020b. arXiv:200..04136.
Lee, A. X., Nagabandi, A., Abbeel, P. and Levine, S. Stochastic latent actor-critic: Deep reinforce- ment learning with latent variable model. In Larochelle, H., Ranzato. M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 741-752. Curran Associates, Inc., 2020a. URL https: http:s/s///pcceeiigg...... neurips...........p............. co wttppppppee//////1eeeee. F20.0000222222200000000000000000
Lee, K.-H., Fischer, I., Liu, A., Guo, Y., Lee, H., Canny, J., and Guadarrama, Predictive information accelerates learning rl. arXiv preprint arXiv:2007. /2401, 2020b.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland,
K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-33,, 2015.
Nguyen, T., Shu, R., Pham, T., Bui, H., and Ermon, S. Temporal predictive coding for model-based planning in latent space, arXiv preprint wrXiv:2106.07156 2021.
Oh, J., Guo, X., Lee, H., Lewis, R., and Singh, S. Action-conditional video prediction using deep networks in atari games. arXiv preprint arXiv: 2015.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint NexXiv/807.0374:: 2018
Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., and Fergus, R. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. 100.2220...22,, 2020.
Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions Autonomous Mental Development, 2(3): 2010.
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint MxXiv:2007.05929 2020.
Silver, D., Schritwieser,, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354 2017.
Stone, A., Ramirez, O., Konolige, K., and Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
Stooke, A., Lee, K., Abbeel, P., and Laskin, M. Decoupling representation learning from reinforcement learning. International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
Sutton, S., Modayil, J., Delp, M., Degris, T., Pilarski, M., White, A., and Precup, D. Horde: A scalable real- time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Systems-Volume 2, pp. 761-768, 2011.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefranog. A., et al. Deepmind control suite. arXiv preprint arXiv: /00v:0800000000, 2018.
Tomar, M., Zhang, A.. Calandra, R., Taylor, M. E., and Pineau, J. Model-invariann state abstractions for model-based reinforcement learning. arXiv preprint arXiv: 2021.
van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F,, Fox, E., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: /ttps:///p/pocceeeeeeee http:////ccceooeoeeeeoooo neurips. paper/2019 9/ e 2 5ad f1
Wahlström, N., Schön, T. B., and Deisenroth, M. P. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint 1XXi::502..0251,, 2015.
Watter, M., Springenberg. J. T., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint 1rXiv:1506.07365 2015.
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint /1100:1110 2019.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Reinforcement learning with prototypical representations. arXiv preprint arXiv:2102. 1271, 2021a.
Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning
from pixels. In International Conference on Learning Representations, 2021b. URL htwp::tt////eee /ottteev//peerrviiee. net/forumPid-GGGe. 6sTaaff...
Zbontar, J., Jing, L., Misra, I, LeCun, Y., and Deny, S. Barlow twins: Sulf-super learning via redundancy reduction. arXiv preprint arXiv:2/03.03230) 2021.
Zhang, A., Wu, Y. and Pineau, J. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv: /81 1.06032, 2018.
Zhang, A., Lyle, C., Sodhani, S., Filos, A., Kwiatkowska, M., Pineau, J., Gal, Y., and Precup, D. Invariant causal prediction for block MDPs. In III, D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11214-121111 14441122444 PMLR. 13-18 Jul 2020a. URL htt.tttt://sec htttt:::s//rooeeeeeeee ings. ml r. press / vt1l/vl11v/////// htt///aannn00mm html.
Zhang, A., McAllister, R., Calandra, R., Gal, Y., and Levine, S. Learning Invariant Representations for Reinforcement Learning without Reconstruction. arXiv, jun 2020b. ISSN 23318422. URL http:///////xixi.o/r////// 10742.
Zhang, A., Sodhani, S., Khetarpal, K., and Pineau, J. Learnnn Robust State Abstractions for Hidden- Parameter Block MDPs. pp. 1-22, 2020c. URL http: / wwwiiv.ivvvv.////vi///.a////iiv. v. wwtp://///bb/....0002222 07206
14

---

Published in Transactions on Machine Learning Research (01/2023)
Appendix
1 Methods
We model the setting in this paper using the framework of contextual decision processes (CDPs), term introduced in Krishnamurthy et al. (2016) to broadly refer to any sequential decision making task where an agent must act on the basis of rich observations (context) to optimize long-term reward. Typically, a CDP is defined as the tuple S, P, 7, R), with observations or contexts 24 E X, states 8, E S, actions a, E A, state transition distribution P($ ae) and reward function = r(st, de; 11+1), ), which defines the nature of the task and dynamics of the system. Here d refers to a dish-simeenional observation while the true state of the environment st not directly available (usually low dimensional) and the agent must construct its own. Without knowing P and R, the RL agent's goal is to use the collected experience to maximize expected sum of rewards, R Et=0 7'r,, consider a discount factor y [0, 1).
1.1 SAC
SAC or Soft Actor-Critic (Haarnoja al., 2018) deploys soft entropy regularized actor critic framework, wherein the policy entropy is added as additional loss to the standard function maximization loss. In particular, SAC uses Qa functions and separate value function V. For transition (Iz, ats 1 from replay buffer D, the exact losses are described as follows:
LQ( E,-D [(o( a.) - (ri + (x
(4)
The value function is then fit w.r.t to the entropy regularized (soft) Q function.
v( Eay's QQ a') a log ny (a' mmtt))
(5)
Finally the policy is fit to maximize both the Q function and the entropy.
L (v) -Baws [Qo(Er: ae) (
(6)
1.2 SAC-AE
SAC-AE (Yarats al., 2019) extends SAC by adding an autoencoder for learning the representation 8; (xr)), which is implicit in the SAC description through the use of deep convolutional networks. A key observation made in this work is that actor gradients should not be used to learn the representation fa. Only the critic gradients train the representation. This idea has been used as standard practice in most papers that follow.
1.3 DREAMER
DREAMER (Hafner et al., 2020; 2021) Iearns latent space dynamics model, both for the transition and reward dynamics, and then uses model based updates with Recurrent State Space Model (RSSM (Hafner et al., 2019)) to optimize the policy. In learning the transition dynamics model, prediction in the latent state as well as in the pixel space is used to construct a representation model, p(st-1/se, a,, zt), and define targets for the dynamics model. The pixel space prediction loss is referred to as the pixel reconstruction loss and is defined with a reconstruction model g(-1s;). The agent minimizes the expectation (CDREAMER) over a finite horizon summation of reconstruction (C)). reward prediction (Lf,) and transition prediction (Lb) losses, such that:
CDREAMER =
Σ (Co
(Co + C22 + cb))
(7)
C. In (x(((())))
C - In R((r((8))
c== 3 (1. 21, ((8441188
15

---

Published in Transactions on Machine Learning Research (01/2023)
1.4 RAD and DrQ
RAD (Laskin et al., 2020a) uses SAC as the main algorithm and adds data augmentations to replay buffer data. These augmentations include: random cropping, translate, grayscale, cutout, rotate, flip, color jitter, and random convolution. For ALE, in general, rangom shifts and intensity are used as augmentations by DrQ (Kostrikov et al., 2020). It is shown that random crop achieves by far the best performance as compared to other augmentations. Therefore, when no mention of the exact augmentations provided, assume that random crop being used.
1.5 CURL
CURL (Laskin et al., 2020b) uses data augmentations (just RAD) over a standard SAC agent and additionally deploys an auxiliary loss that tries to learn representations such that augmented versions of the same observation or context are pulled together in the representation space while pushing apart augmented versions of two different observations. CURL. uses the bi-linear inner-product g" WWk k, where W a learned parameter. Here, and are the latent representations of the raw anchors (query) 2i and targets (keys) Ig, such that q = ff((x)) and ls (g(f)(r))), where sg denotes the stop gradient operation.
1.6 DER and OTR
Both DER (van Hasselt et al., 2019) and OTR (Kielak, 2020) refer to a highly tuned version of the Rainbow (Hessel et al., 2018) agent. The hyperparameters are tuned so as to get superior performance over Rainbow on Atari which uses lot less samples than the standard training of 200M steps. There also exist similar versions for the DQN agent (data efficient DQN, overtrained DQN), which most notably does not use distributional component in
1.7 SPR
SPR (Schwarzer et al., 2020) or self-predictive representations deploys an auxiliary transition prediction loss which is learned a non-contrastive self-supervised manner by using cosine similarity loss. In particular, instead of using the actual next state 8t+k as the target the predicted next state & the authors use projection network to transform both the targets (9m) and the predictions (go). Furthermore, the predictions are then then passed through a predictor network (q), and the output of this network and of the target projector are used define cosine similarity loss Cossine- The use of the projector and predictor networks is similar to that of in self-supervised learning methods in vision, such as B BYOL (Grill et al., 2020).
Coosine Σ
We+h |1
(
T
De-h
(8)
110
h=1
where 8L+* = (st, Gt, at....,, 0t+k-1), 0eth (go (8++t)))s
Utth = Im (8t+h)
In particular, SPR also uses data augmentations and a convolutional transition dynamics predictor. When not using data augmentations, SPR uses dropout in the encoder network fe.
1.8 DBC
DBC (Zhang et al., 2020c) or Deep Bisimulation for Control is an algorithm based on bisimulation metric losses which tries to pull observations having the same long term reward closer in the representation space. The specific implementation uses an encoder, reward prediction model and probabilistic latent transition model. In particular, the encoder is trained by sampling batches experiences (z, a, T, x') and minimizing the LI norm between any non-identical transitions in the sampled batch follows:
LDBC = (1Is;- - -r; - ryl - (P P(),,,,)))
where s; = fo(x,), sg( (f)())))) and W2 is the 2- Wasserstein distance metric between two transition distributions. Please refer to the paper for further details.
16

---

Published in Transactions on Machine Learning Research (01/2023)
1.9 Baseline
We use a simple soft actor-critic setup with embedding function So X' - 8 (similar to SAC-AE (Yarats et al., 2019)) as the base architecture, and attach the reward and transition prediction modules to it (See Figure 2). We define the transition prediction by P(st, at) and the reward prediction by R(s++1) such that, 8 = P(st, Qr) and ft = R($++1). Note that the transition network is over the encoded state & = fx()). and not over the observations
24 (Lee al., 2020a). The overall auxiliary loss function is thus defined as follows:
Lbuseline
(51+1 (.),aa)) (R(s
ai)))
(10)
Transition prodiction loss
Reward prediction loss
1.10 Value-aware Model Learning
Based on the bascline architecture, the value aware experiments are performed with the SAC critic Q, policy ι and the corresponding value aware loss, Av as (sg(V++1) V++1)= where
V4+1 Ee ~et (14111,0441) log ((c+11144))))
1/41 E [Q ($++1, a(+1) log ग (1:41 [141)],
Here, sg denotes the stop gradient operation. We test the approach both with and without the reward prediction module (R in equation equation 1) and report the results in Table Complete performance plots are provided in Appendix 8.8.
17

---

Published in Transactions on Machine Learning Research (01/2023)
2 Background Distractor Details
We use the distracting control suite with natural distractors previously introduced in Kay et al. (2017); Zhang et al. (2018) and used in Zhang et al. (2020c); Stone et al. (2021). The background distractor frames are picked up sequentially from videos randomly sampled from the Kinetic dataset ('driving car' class) and were used during both training and evaluation. The choice of video for any two runs of a single DMC task as well as during training and evaluation can vary randomly. The selected sequence of frames are masked alongside the observations obtained from the DMC suite. The resulting observations have two sequences, one corresponds task-relevant agent movement while the other corresponds to task-irrelevant background video streams. We consider the addition of distractors to two different settings: centered and off-centered, as shown in Figures 8 and 9 respectively.
Figure& 8: Distractor on Agent Centered configuration We show the sequence of observations with distractor backgrounds on Choetah Run, Hopper Hop and Walker Walk tasks. It is to be noted that there can be more than one video streams playing sequentially for environment.
13
Figure 9: Distractor on Agent Off-centered configuration The background distractor configuration is same as the agent-centered setting, the agent's ' size and position are changed. This setting increases difficulty for methods dependent on well positioning of the agents.
18

---

Published in Transactions on Machine Learning Research (01/2023)
3 Implementation Details
We use the SAC algorithm as the RL objective added with reward and deterministic state transition model both of which are MLPs. In this section, we present in detail the architecture, source code links and the computational usage.
Architecture. The encoder architecture taken from the open-source implementation of Yarats et al. (2019) which uses the same encoder for the actor and critic to embed image observations. Only critic gradients are used to train the encoder. Both the reward and transition prediction losses are equally weighed in the gradient update and there is no extra hyperparameter balance between the two.
The ablation on augmentation uses the same configuration of cropping shown in RAD and CURL i.e. random cropping (84 84) from observations of (100 100) dimensions for DMC and random shifts of +44 pixels for ALE. Random flip and rotate are applied in similar way as well. For the reconstruction loss augmentations, deconvolution layer is introduced to reconstruct the original pixel-state from the latent state vectors.
Source Codes. For the methods used for comparison, the corresponding code was directly borrowed from their open-source implementations or their performance was taken from Agarwal et al. (2021b). The natural distractor background added from the open source implementation of Kinetic dataset to the other methods for comparison. The respective URLs given below.
Method
Open-Source URL
Kinetic dataset
attps://github wwt.:///wwwoaaaaeeeeeeeeeeeeeeea http:://gggtabbb.c.c/////aaaaaap
SAC-AE
RAD
httpp:///githhbb.com////f int.
CURL-DMC
DrQ
httpes//ggttaab.ccc////alappppp
DBC
https://github.
PI-SAC
Dreamer
https://ggttub..com//an////
TIA
hstps:
CURL-ALE
SPR
http:///gthhabb.oo///aaaa1.....
Computation. All experiments were conducted on either system configuration of:
1. 6 CPU cores of @Intel Gold 6148 Skylake@2.4 GHz, one @NVidia V100SXM2 (16G memory) GPU and 84
GB RAM.
2. 6 CPU cores of @Intel Xeon Gold 5120 (kylake@2.2GGi one @NVIDIA V100 Volta (16GB HBM2 memory) GPU and 84 GB RAM
The average completion time of the baseline experiments around 20 hours. The time taken less (around 12 hours) for the SAC or baseline-v0 experiments. For the value aware experiments, RAD and CURL average time of completion was around 15 hours. The time required for Dreamer, baseline with reconstruction losses, SPR variant DMC and PI-SAC is approximately 28 hours. On an average, each experiment is expected to be done in 24 hours. A total of around 1500 experiments were performed for the final version of this paper which corresponds to roughly 4 years of GPU training.
19

---

Published in Transactions on Machine Learning Research (01/2023)
4 Hyperparameters
The full set of hyperparameters used for the bascline experiments are provided in Table below.
Table 3: Hyperparameters for Baseline and related ablations.
Hyperparameter
Values
Observation shape
(84, 84, 3)
Latent dimension
50
Replay buffer
100000
Initial steps
Stacked frames
Action repeat
2 finger, spin; walker, walk caripole, swingup
4 otherwise
SAC: Hidden units (MLP)
1024
Transition Network: Hidden units (MIPP
128
Transition Network: Num Layers (MLP)
Reward Network: Hidden units (MLP)
512
Reward Network: Num Layers (MLP)
Evaluation episodes
10
Optimizer
Adam
(81, 92) (fo. Tw. Qo)
(99.999)
(81,82) (a) Learning rale (fo, T Qp)
(5,9999)
2e-4 cheetah, run
le-3 otherwise
Learning rate (a)
lc-4
Batch Size
128
Q function EMA
0.005
Critic target update freq
Convolutional layers
Number of filters
32
Non-linearity
ReLU
Encoder EMA
0.005
Discount
.99
20

---

Published in Transactions on Machine Learning Research (01/2023)
5
Additional Baseline Ablations
Cartpole Swingup
Cheetah Run
finger
1.00
0.75
0.75
0.50
0.2
0.25
0.25
0.00
0.0
0.00
Normalized
0.0
0.1
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Reacher Easy
Walker Walk
Hopper Hop
0.6
0.6
0.15
0.4
0.10
0.2
0.2
0.0
0.00
0.2
0.0
0.2
0.4
Environment Steps (x10f)
Baseline w/ Distractor
Baselinee wo Distractor
SAC Distractor
SAC w/o Distractor
Figure 10: Baseline distractor Baseline w/o distractor ys SAC and distractor Performance of the proposed baseline DMC
tasks the presence absence of distractors along with performnce SAC (Baseline-vO) the same settings. The minor difference performance baseline two conditions clearly illustrates the learning backgronnddiistratoor invariant representations. This contrast with several experimental analysis showing overfitting by other algorithms. Additionally, performance gain as compared to SAC is visible both with and without distractor cases.
Cartpale Swingup
Cheetah Run
Finger
1.00
0.75
0.75
0.50
0.2
Normalized Score
0.25
0.25
0.00
0.0
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Reacher Easy
Walker Walk
Hopper Hop
1.00
0.75
0.75
0.2
0.50
0.50
0.1
0.25
0.00
0.00
0.0
Environment Steps (x10')
Baseline Crop
Baseline Flip
RAD Crop
RAD Flip
Figure 11: Baseline augmentations ys Baseline-v0 augmentations Performance of the haseline with crop fop augmentations the task data. comparison conducted based the results RAD (Laskin et al., 2020a) similar augmentations. Performance curves for each of DMC lasks is shown. With cropped data, baseline performs equally well as RAD due removal irrelevant features by cropping. Whereas for flipped data, where none of the task irrelevant features are excluded, bascline surpasses RAD.
21

---

Published in Transactions on Machine Learning Research (01/2023)
6
Baseline with Multistep Training
Cartpole Swingup
Cheetah Run
Angerr Spin
1.00
0.8
0.4
0.75
0.50
0.2
0.25
Normal Score
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Easy
Hopper Hop
0.6
0.06
0.4
0.04
0.02
0.1
0.0
0.0
0.00
0.0
Environment Steps (x10f)
horizon=1
horizon=2
Baseline horizon=4
horizon=6
Figure 12: Baseline w/ multi-step training Performance of baseline with recurrent latent state space model and reward prediction network (MLP) is shown for fixed horizon for euch of the DMC tasks. Increasing horizon degrades performance all the tasks.
Cartpole Swingup
Cheatah Run
Finger Spin
0.6
0.75
0.2
0.4
0.50
0.1
0.7
0.0
0.0
0.00
Normaized
Reacher Easy
Walker Walk
Hopper Hop
0.04
0.15
0.07
0.2
0.05
0.03
0.00
0.0
0.00
0.5
Reconstruction Environment Steps (x10*)
Reconstruction horizon-1
horizon-4
Reconstruction horizon=6
Figure 13: Baseline w/ multi-step training and reconstruction Performance of the proposed baseline six DMC tasks with multi-step training und added reconstruction loss for each prediction. This resembles the architecture of DREAMER (Hafner al., 2020) corresponding baseline. Full reconstruction tries to capture task- inelevatt information leading decrease in performance. Further, the performance degrades irrespective of the choice of the horizon.
22

---

Published in Transactions on Machine Learning Research (01/2023)
7 Dataset Categorizaions and Specifications
Table 4: Dataset Categorizations. Denset refers games under the 'Human Optimal' 'Score Exploit' category from the cutegorization provided
(Rellemare al., 2016) and Table performance of haseline, metric, contrastive, w/o contrastive reconstruction in each of calegories shown Figure The algurithm corresponding to each method shown Appendix (Table 5), Also, since augmentations only modify data, they should be included evaluation calegorizations.
C2
C4
C5
Environment
ALE
ALE
ALE
DMC
DMC
DMC DMC
DMC
Reward Structure
Dense No
Dense
Denset
Dense
Dense
Dense
Sparse
Dense
Distractor
No
No
Yes
Yes
Yes
Yes
No
Augmentations
None
Shift, Intensity
None
None
Crop
Flip
None
None
Table S: Categorizations and Methods. The algorithm corresponding each of the methods shown. The algorithms considered are DER (van Hasselt al., 2019), DRQ (Kostrikov al., 2020). SAC (Haarnoja 2018). RAD (Laskin 2020a), PI-SAC (Lee al., 2020b). DBC (Zhang 2020b), CURL (Laskin al., 2020b). SPR ((Schwarzer al., 2020) and SPR""), DREAMER (Hafner 2020; 2021) and TIA al., 2021). The performance of cach of the methods each the categories shown Figure For ALE, DREAMER scores after steps were used for comparison.
Methods
C1
C2
C4
C5
C7
Baseline
Baseline-v0
DER
DRQ
DER
SAC
RAD
RAD
SAC
SAC
State Metric
DBC
DBC
Contrastive
CURL
CURL
CURL
PI-SAC PI-SAC
PI-SAC
PI-SAC
SSL w/o Contrastive
SPR*
SPR**
SPR
SPR**
Reconstruction
DREAMER
DREAMER
TIA
TIA DREAMER
Table 6: Categorizations and Environments. The environments included each categorics shown. The environments considered from both DMC and ALE Atari-100k benchmarks. Dense+ ALE refers games under the 'Human Optimal' and 'Score Exploit' category from categorization provided (Bellemare al., 2016) and Table The performance each of the methods each the categories shown Figure
Environments
C2
C6
C8
Caripole-Swingup
Choctah-Run
Hopper-Hop
Walker-Wall Walk
Reacher-Easy
Finger-Spin
Ball-in-Cup-Catch
Dense Atari100k
Dense+ Atari100k
23

---

Published in Transactions on Machine Learning Research (01/2023)
8
DMC Suite: Complete Results
8.1
Role of Each Component in Baseline
Cartpole Swingup
Cheetah Run
Engeer Spin
0.8
0.75
0.6
0.50
0.2
0.2
0.00
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
Reacher Easy
Wallker Walk
Hopper Hop
0.4
0.04
0.2
0.1
0.0
0.0
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10f)
w/ Reward
Reward Transition
Reward
wlo Reward
Transition
Transition
w/o Transition
Figure 14: Baseline component ablations. The importance of the two components of the propused baseline: Transition and Reward, were analyzed
based on the effect of their absence the performance achieved for cach of the six DMC tasks. The comparison shows that both are necessary to achieve the best performance overall. While absence of both results in some learning, addition of transition collapses the representations. On the other hand, reward proves to he non-negligible component.
Cartpale Swingup
Choetah Run
Finger Spin
0.8
0.6
0.4
0.2
0.2
0.1
0.2
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallker Walk
Hopper Hop
0.3
0.4
0.2
0.04
0.2
0.1
0.02
0.0
0.0
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0,4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (K10*)
Baseline w/ Non-Uneer Reward Decoder
Baseline Linear Reward Decoder
Figure 15: Baseline w/ non-linear ES w/ linear reward decoder. Performance across six DMC tasks with linear and non-linear reward decnders with the baseline architecture. Reward decoder formulation plays significant role in the representation learning objective. As a linear reward decoder discards mask-ircleevnt features too quickly, the left features are not sufficient act optimally. On the other hand, non linelincar reward decoder is comparatively sophisticated in recognizing task-inrelevant features and features required to optimally.
24

---

Published in Transactions on Machine Learning Research (01/2023)
8.2
Arrangement of Reward and Transition Losses
Cartpole Swingup
Cheetah Run
Finger Spin
0.8
0.6
0.3
0.4
0.2
0.4
0.2
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallker Walk
Hopper Hop
0.4
0.06
0.3
0.04
0.0
0.00
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
Enviranment Stops (x10')
Reward through Transittion
Independent Reward Transittion
Figure 16: Arrangement of Reward and Transition in baseline architecture. Performance across six DMC tasks with reward prediction through transition and independent reward + transition prediction modules. The average performance is similar for both the cases. Having identified the importance of the reward and transition models, the remaining question is how to actually assemble these in the overall architecture. There are workuble possibilities for this, being attaching both independently the encoded state, R(s, a, s'). and the other being attaching the transition model the encoded and then attaching the reward model the output of the transition model, i.e. learning reward through transition marginally better.
R(s, 5'). We both of these and conclude that both perform equally well. We choose reward through transition
25

---

Published in Transactions on Machine Learning Research (01/2023)
8.3 Role of Augmentations
Cartpole Swingup
Cheetah Run
Finger Spin
1.00
0.75
0.4
0.50
0.2
0.25
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wa lell Walk
Hopper Hop
1.00
1.00
0.75
0.75
0.2
0.50
0.25
0.25
0.00
0.00
0.0
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.5
0.0
0.2
Enviranment Stops (x10')
Baseline
RAD
RAD
RAD Rotate
Figure 17: Baseline vs Augmentations. Performance of proposed baseline and RAD (Laskin al., 2020a) with crop/tliprroaae augmentations the data from the task. Cropping removes the distractor data and excels in getting the best performance in all the six DMC tasks, Baseline performs close to RAD flip and rotate which consider the complete pixel data. This indicates that RAD crop takes advantage of the centered position of the task-relevant objects and well curated structure of the data. This eventually introduces considerable inductive bias.
Cartpole Swingup
Cheetah Run
finger Spin
1.00
1.00
0.75
0.4
0.50
0.50
0.2
Normalized Score
0.25
0.25
0.00
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallker Walk
Hopper Hop
1.00
1.00
0.75
0.2
0.50
0.1
0.25
0.25
0.00
0.00
0.0
0.0
0.1
0.2
0.3
0,4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Enviranment Stops (x10')
RAD Crop
Flip
Small Robots RAD
Small Robots RAD Flip
Figure 18: RAD w/ standard DMC setting vs w/ zoomed out DMC setting. We conducted an ablation on performance of RAD with cmop and flip data augmentations for two diverse camera positions for six DMC tasks. The modified camera settings zooms the agent frame. The drop in the performance of R.D (Laskin et al., 2020a) in the modified setting validates that the success of cropping is due to the well-centered and appropriate settings of DMC.
26

---

Published in Transactions on Machine Learning Research (01/2023)
8.4
Role of Contrastive Losses with Augmentations
Cartpole Swingup
Cheetah Run
Finger Spin
1.00
0.4
0.50
0.50
0.2
0.25
0.25
0.00
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallerr Walk
Hopper Hop
1.00
1.00
0.75
0.75
0.2
0.50
0.25
0.25
0.00
0.00
0.0
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.0
0.2
Enviranment Stops (x10*)
Baseline
RAD
PISSAC
CURL (Contrastive)
Crap)
Figure 19: Baseline RAD PI-SAC vs CURL. We conducted ablation performance of SAC with contrastive losses on DMC task with data augmentations. Baseline and RAD (Laskin al., 2020a) with random erop compared with PI-SAC Lee et al. (2020h) and CURL. (Laskin al., 20206). While CURL uses the contrastive loss of CPC Ourd al. (2018), PI-SAC uses Conditional Entropy Bottleneck (CEB) objective. PI-SAC performance is significanlly betler compared to CURL.
27

---

Published in Transactions on Machine Learning Research (01/2023)
8.5
Role of Contrastive and SSL w/o Contrastive losses
Cartpole Swingup
Cheetah
Finger
1.00
0.4
0.75
0.75
0.3
0.50
Normalized Score
0.25
0.25
0.00
0.0
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
0.5
Reacher Easy
Walleer Walk
Hopper Hop
0.6
0.15
0.4
0.10
0.05
0.0
0.0
0.00
Environment Steps (x10t)
w/ Reward
w/ w/ Contrastive
PI-SAC (Contrastive)
CURL
SPR (Cosine Similarit)
w/ Transition
(Contrasti astive)
Figure 20: Baseline contrastive vy SSL contrastive. Performance of baseline compared of transition the baseline for each of six DMC tasks. Further, we also compare simple contrastive losses like PI-SAC et al. (2020b) (without augmentations), CURL (Laskin et al., 2020b) SSL contrastive losses an extension of SPR (Schwarzer al., 2020) (which cosine similarity loss) for DMC, Contrastive and SSL Contrastive approaches not seem to be very effective in absence of data augmentations.
addition contrastive loss instead
28

---

Published in Transactions on Machine Learning Research (01/2023)
Role of Reconstruction losses 8.6
Cartpole Swingup
Cheetah Run
Finger
0.8
0.6
0.75
0.4
0.50
0.2
0.25
0.0
0.0
0.00
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
0.5
Reacher Easy
Walker Walk
Hopper Hop
1.00
1.00
0.20
0.75
0.75
0.15
0.50
0.50
0.10
0.25
0.25
0.05
0.00
0.00
0.00
Environment Steps (x10*)
Baseline
Dreamer wlo Distractor
Dreamer w/ Distractor
Di Distractor
Figure 21: Baseline Methods with Reconstruction Losses. Performance of proposed baseline with the of the algorithms employing reconstruction losses, DREAMER (Hafner al,, 2020) and TIA (Fu al., 2021). DREAMER without distractors marks the upper limit for the maximum achievable performance. The performance of baseline, having relatively simple architecture, considerably better DREAMER and TIA.
Cartpole Swingup
Cheetah Run
Finger Spin
1.00
0.6
0.4
0.50
0.2
0.25
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Reacher Easy
Wallker Walk
Hopper Hop
1.00
0.6
0.15
0.75
0.10
0.50
0.25
0.05
0.0
0.00
0.00
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.0
0.2
Enviranment Stops (x10')
Baseline
Full Reconstruction
Partial Reconstruction
Figure 22: Baseline Baseline with Reconstruction losses. The performance of the variants of bascline with additional reconstruction losses in analyzed for each of the six DMC tasks. The variants include full reconstruction and partial/relevant reconstruction losses. While the former is same the approach employed by DREAMER (Halner al., 2020), the later only tries reconstruct the tusk-relevant features, which provided explicitly. We observe that the performance achieved including the partial reconstruction is the ceiling for the performance which be achieved by employing reconstruction losses. However, still observe failure the Reacher Easy task.
29

---

Published in Transactions on Machine Learning Research (01/2023)
Cartpole Swingup
Cheetah Run
Finger Spin
1.00
0.75
0.6
0.75
0.50
0.4
0.50
0.25
0.2
0.00
0.0
0.00
Normalized
Reacher Easy
Wallker Walk
Hopper Hop
1.00
1.00
0.75
0.75
0.50
0.50
0.1
0.25
0.25
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Baseline + Partial Reconstruction
RAD Crop
Dreamer w/ Distractor
TIA w/ Distractor
Figure 23: Baseline vs Augmentations vs Reconstruction. Based on the ablation studies with full and relevant reconstruction losses added to the baseline, the relevant reconstruction variant is contrasted with RAD (Laskin et al., 2020a). DREAMER (Hafner et al., 2020) and TIA (Fu et al., 2021). RAD with cropping removes most of the distractor data and the performance illustrates that the representations learned by RAD with cropping is equivalent to reconstructing only the relevant pixels. Full reconstruction by DREAMER constructs the lower margin of the plots and TIA improves moderately on DREAMER by adding decoupling between task relevant and irrelevant features. However, the efficacy of these methods are significantly compromised as compared performance achievable actually learning task-relevant features.
30

---

Published in Transactions on Machine Learning Research (01/2023)
8.7
Role of Augmentations with Reconstruction Losses
Cartpole Swingup
Cheetah Run
Finger Spin
1.00
0.3
0.2
0.50
0.25
Normalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallerr Walk
Hopper Hop
1.00
1.00
0.06
0.75
0.75
0.04
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.0
0.2
Enviranment Stops (x10')
Baseline
Baseline
Baseline
Random Crop)
Reconstruction)
Reconstruction Random Crop)
Figure 24: Augmentations with Baseline vs Reconstruction losses. The role of augmentations in the tinal performance of reconstruction loss the distractor observations is analyzed. While augmentations tend to boost the performance for all algorithms including Baseline, RAD (Baseline-vI) and PI-SAC (Contrastive), they do not show much improvement with recunstruction losses.
31

---

Published in Transactions on Machine Learning Research (01/2023)
8.8 Value Aware Learning
Cartpole Swingup
Cheetah Run
Finger Spin
0.75
0.6
0.4
0.2
0.50
0.25
Narmalized
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Reacher Easy
Wallker Walk
Hopper Hop
0.6
0.06
0.4
0.04
0.2
0.0
0.0
0.00
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.0
Enviranment Stops (x10')
Baseline
Value through Transition Reward
Value through Transition Reward
Figure 25: Baseline vs Value Aware Learning. Performance of the proposed bascline and truly value aware learning on each of the six DMC tasks. We correlate the relation between the baseline and value aware learning. We, then, conduct ablations on the transition loss where we replace 'reward through transition' with "valuc through transition'. The above plots show the performance of "value through transition' with/without added reward loss.
32

---

Published in Transactions on Machine Learning Research (01/2023)
9 ALE Benchmark: Atari 100K
Individual performance scores 25 garms from that augmentations fail to work with DER (van Hasselt et al., 2019) but perform reasonably well with OTR (Kielak, 2020) as evident from performance of DRQ (Kostrikov et al., 2020). Further, to analyse the reason behind the performance boost of SPR (Schwarzer et al., 2020). we perform an experiment by remaving dropout and augmentations and denote as SPR**. The performance drop for SPR** as compared with SPR is significant.
benchmark categorized
Bellemare (2016), observe
Game
DER DER+Aug
CURL
DrQ
SPR""
SPR
Baseline
Human Optimal
Assault
431.2
155.5
351.9
600.6
452.4
647.9
571.0
474.8
Asterix
470.8
254.4
628.5
734.5
603.5
435.5
977.8
638.4
BattleZone
10124.6
6083.9
4060.6
14870.0
12954.0
12208.0
16651.0
9124.0
Boxing
0.2
9.3
1.2
6.0
-0.7
35.8
1.2
Breakout
1.9
4.48
9.8
4,9
16.1
5.612
17.1
2,9
ChopperCommand
861.8
270.7
1033.3
1058.5
780.3
695.2
974.8
675.2
Crazy Climber
16185.3
5052.6
21327.8
12146.5
20516.5
16441.4
42923.6
23912.6
Domon Attack
508.0
628.8
7118
817.6
1113.4
397.6
545.2
635.8
Jamesbond
301.6
145.6
112.3
471.0
236.0
404.5
365.4
228.1
Pong
-19.3
6.4
1.3
-16.5
-8.5
-0.5
-5.9
-18.8
Score Exploit
Kangaroo
779.3
1391.3
605.4
872.5
940.6
3749.4
3276.4
653.4
Krull
2851.5
838.4
3277.9
4229.6
4018.1
2266.6
3688.9
2775.3
Kung Fu Master
14346.1
4004.9
5722.2
14307.8
8939.0
15469.7
13192.7
10987.6
Road Runner
9600.0
2274.9
2696.7
5661.0
8895.1
4503.6
14220.5
6408.4
Seaquest
354.1
114,3
286.9
384.5
301.2
448.4
583.1
518.9
Up N Down
2877.4
2240.1
2847.6
2955.2
3180.8
2413.3
28138.5
2543.2
Dense Reward
Alien
739.9
223.2
824.7
558.2
771.2
835.1
801.5
699.7
Amidar
188.6
42.8
82.8
142.1
102.8
138.5
176.3
115.654
Bank Hcist
51.0
53.99
182.1
131.6
168.9
205.3
380,9
239.2
Frostbite
866.8
325.4
231.6
1181.3
331.1
1248.7
1821.5
1740.9
Hero
68.57.0
339.4
6458.8
6279.3
3736.3
6076.2
7019.2
4464.4
Ms Pacman
1204.
393.6
941.9
1465.5
960.5
1346.9
1313.2
999.2
Qbert
1152.9
1530.3
509.3
1042.4
854.4
1841.75
669.1
1007.9
Sparse Reward
Freeway
27.9
0.6
25.0
26.7
9.8
29.5
24.4
27.54
Private Eye
97.8
54.8
100.0
218.4
-13.6
-49.5
124.0
100.0
33


Reviews:
[{"review_id": "5ySFsKLlYE", "reviewer": "Reviewer_U1dd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There is a very large number of experiments done for several methods in the paper. The descriptions of the results plots are backed up by the data and each run is run for at least 5 seeds. I would also generally agree that short-horizon dense RL may have very different learning challenges compared to long-horizon sparse RL, for example.\n\nI feel some of the *specific* claims made by the paper are overreaching. Some specifics:\n\n\"For other methods, it is not entirely clear what components in auxiliary objectives can lead to failure\". I feel this is actually addressed in said papers quite often? Ablations for different components are pretty common and it's often acknowledged that trying to reconstruct all pixels has its flaws. Perhaps the more specific claim is that it is hard to reproduce these results when changing environment or when reproducing the paper's algorithm - but this is less a problem of the method and more a problem of the field, if that makes sense.\n\n\"It is also hard to pick which data augmentations will work for a particular environment\" - I don't disagree, but it is somewhat strange to cite a paper that claims to automatically determine the right data augmentations per environment.\n\nFor the categorization of environments in general, the authors say that every method leads to varied performance across different categorizations, which makes a comparison across the average highly uninformative. This doesn't feel like a very strong point. Let me draw an analogy to ImageNet - performance varies wildly across image classes (dogs are overrepresented, dumbbells are often confused with muscles because they often appear together). Yet we still report an overall average and assume that the different sources of noise are not correlated and sufficiently cancel out. How would the authors distinguish ImageNet from different ALE games?\n\nIt also seems to me that just giving different performance profiles across different subsets of games does not say anything interesting about the underlying methods. There are many ways to partition a set of environments. Although partitioning by augmentation and reward structure is a reasonable one based on our priors, I could also partition them randomly and I suspect I would get plots similar to Figure 1 in the paper. Is there a section where the authors discuss patterns across different classes of methods - and then verify their analysis would not equally hold if they split the benchmarks up randomly?\n\nWhen analyzing what the random crop of RAD is doing, the authors propose that it helps primarily because the edges of the camera image contain task irrelevant information and the crop just removes it. Although this is how RAD describes random cropping, in DrQ random cropping is implemented by first padding the original 100x100 image to a larger one (say 108x108), then taking a random 100x100 crop within it. The image is padded with either 0s or repeating the border pixels. In either case, this should strictly be adding irrelevant information to the image - we add a border of 8x8 irrelevant pixels then crop another 8x8 border of pixels back out. Yet DrQ also reports seeing improvements from random cropping. So I don't think the author's explanation of this is the full story. Off-centering or zooming out the image should cause RAD performance to go down, but what matters is whether it causes a larger relative drop compared to the baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper does a comparison of many different algorithms proposing learning representations for pixel-based reinforcement learning. The methods are run on a mix of ALE and MuJoCo tasks, using different levels of data augmentation or background distractions, in both sparse reward and dense reward environments. The methods themselves range between methods that just do data augmentation, to auxiliary task prediction, and methods that try to directly learn a representation via contrastive or self-supervised learning.\n\nBased on the results, the authors make a number of claims. I would characterize many of these claims as obvious, but not all of them. For example, the authors find that data augmentation is generally helpful for all methods, model-based methods based on pixel-reconstruction do worse with more elaborate backgrounds, and representations based on task-relevant features do better than ones based on full pixel learning. However, there are more specific claims that are interesting, such as a claim that random cropping's effectiveness is tied to particulars of the MuJoCo benchmark, or that using a linear layer as the reward decoder causes methods to fail compared to a 1 hidden layer decoder. The authors generally argue that comparisons of RL algorithms should be more multi-faceted, and instead of comparing over an entire benchmark like ALE, we should categorize environments based on their learning properties (short vs long horizon, dense vs sparse reward), and compare methods along this access.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate if the authors could try doing the control group analysis of seeing what differences in performance they see if they partition tasks randomly, rather than along dense vs sparse lines. I would also appreciate if Figure 7 was split across all vs dense vs sparse rather than just all vs dense. Finally, I would appreciate more investigation of whether random cropping is as tied to MuJoCo as the authors suggest it is - perhaps experiments of random crop on ALE would help because the camera view is not as centered on the relevant agent in such games. (In my experience random cropping has worked quite well on other environments, including ones not based on MuJoCo. MuJoCo may be a best case for this but it doesn't seem likely it is the only reason the method works.)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "quHqp6qebI", "reviewer": "Reviewer_TtNM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- many different representation learning methods have been proposed in the past — analyzing their tradeoffs with a unified implementation across multiple environments is valuable\n\n- the experimentation in the paper seems thorough: a representative set of representation learning methods is compared across standard environments with diverse characteristics (discrete action games, continuous control, w/ vs w/o distractors) — each experiment is run with multiple seeds —> this makes the results of the experiments trustworthy\n\n- the conclusions drawn in Section 7 are interesting, especially showing that: (1) contrastive losses in the presence of rewards do not bring much benefit, (2) the success of certain augmentations, like random crop, on DM Control might be quite brittle\n\n- the writing is easy to follow, with intuitive examples; the overview of existing representation learning approaches seems thorough\n\n\n# Weaknesses\n\nThe paper has two main weaknesses in my opinion: (1) hard-to-follow organization of experimental results, (2) stops short of providing a better evaluation methodology for representation learning approaches.\n\n(A) **Hard-to-follow experimental results**: \nThe experimental section of the paper presents a sequence of interesting results, but it is hard to follow the story in sections 4-6. The methods and environments are not introduced at once and then compared to each other, but instead methods and environments are introduced one-by-one and tested only on specific versions of the environments without a clear, overarching thread that ties all sections together. This makes it hard to see the bigger picture and leaves it to the reader until the very end (section 7) to distill more general conclusions from the results. I think the paper could be much easier to digest if all methods would be compared on all environments and then a sequence of key takeaways would be mentioned and subsequently backed up by picking out results from the comparisons. This structure of first presenting the full experimental setup and then analyzing the results with respect to a few key takeaways could make it easier to understand. See for example Mandlekar et al. 2021 (https://arxiv.org/abs/2108.03298) for an example of clear experiment description and then analysis with clear takeaways.\n\n(B) **No better evaluation proposed**: \nThe key message of the paper is that evaluating representations by providing an average performance number on a benchmark is insufficient, since existing representation learning approaches can vary widely in performance even across the different task within a single benchmark. Instead, the paper argues to focus on different properties of evaluation tasks (eg reward density, presence of visual distractors, …) when comparing representation learning approaches. In the light of the presented results I agree with this conclusion, *however* I wonder why the paper does not continue to propose such an improved evaluation scheme. It already performs all of the necessary experiments and to me it seems it is just a question of sorting the evaluated tasks by more meaningful characteristics and figuring out a way to make results across benchmarks comparable. The insight provided by the analysis is valuable by itself, but I wonder why the paper does not act on its recommendation and provide a better evaluation. As such, I see this as a weakness of the current submission.\n\n\n### Minor Weaknesses\n(i) the text is sometimes ambiguous about which exact environment is used for an evaluation — e.g. Section 4.3 talks about distractors, but it seems like the experiments were performed on DMC w/o distractors\n\n(ii) the “natural distractors” used for DMC are not actually very natural —> the agent is moving over a completely disconnected video in the background — this is in line with prior work on “distracted DMC” (Stone et al., Zhang et al.), but since this paper puts a lot of emphasis on analyzing representation under natural distractors, it would be better to have an actual environment with a lot of “more natural” task-irrelevant distractors (eg a driving simulator like GTA or CARLA in [2])\n\n(ii) for the experiments with off-center and zoomed-out versions of DMC (Table 2) it would be good to also provide the performance of the non-augmented baseline to see whether the task itself gets harder, e.g. because the effective resolution is reduced in the zoomed-out case\n\n(iii) some implementation details are lacking, e.g. how the negatives for the contrastive objective are sampled\n\n(iv) for the models with reconstruction it would be good to show some example reconstructions to show that the models are powerful enough to reconstruct the scene meaningfully —> otherwise their bad performance could stem from being too under-parametrized\n\n(v) the paper refers to the DMC agent as “robot” —> I would rather refer to it as agent since the term “robot” is usually associated with an actual robotic agent\n\n(vi) the paper is lacking references to some relevant related work: [1] compares different representation learning approaches in the context of offline RL, [2] investigates the robustness of different representation learning approaches under distractors\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper compares a range of different representation learning approaches for RL across multiple environments (Atari, DM Control w/ & w/o distractors). The paper shows that all approaches have substantially varying performance across the different environments and analyzes how the characteristics of the environment influence the performance of the methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I am okay to accept the paper — it fulfills the TMLR requirements of being technically sound and providing a new insight to the community. I think that adding the baseline numbers in (ii) could be helpful.\n\nMore generally however, I think the paper could be substantially improved by addressing the main weaknesses I listed above: structuring the presentation of experimental results better and providing a more meaningful evaluation scheme for future representation learning approaches. It is upon the authors to decide whether they agree and want to make those changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZmEjHNkXbs", "reviewer": "Reviewer_JcWC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The paper's reasonings on why certain methods fail, generally make sense (although please see below about improving clarity). For example, it is definitely agreeable that world models may fail when reconstruction is very difficult to perform.\n* The paper provides extensive experimental evidence to support its claims, by running numerous algorithmic baselines across a variety of different literature.\n\n\n# Weaknesses\n* The paper is written in such a way that requires a large amount of background knowledge on many of the RL techniques presented. There are a lack of equations and concise mathematical notation to explain the techniques used, and thus the paper can be excessively wordy and hard to follow. \n* Following up on the above wordiness, it was difficult to understand (at a conceptual level) why e.g. the baseline in Figure 2 was used to prove that the previous RL methods would fail. \n* The use of the distracting background control suite is a double-edged sword here, because one could argue that such environments are actually too out-of-the-ordinary and are on the extreme case of RL environments. Perhaps only the closest graphics that look like the distracting control benchmark would be self-driving (and even then, the background is dependent on the agent's movement and actions, rather than an uncontrollable video), whereas e.g. real life robotics generally do not have this type of issue. \n* (Minor) The paper seems to not be following the TMLR 1-column style format and looks like an ICML 2-column format. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of this paper to is to raise awareness that many of the recent improvements to representation learning in RL, are not universal improvements. The paper uses a \"distracting background\" variant of DM-Control to present this issue, as it contains much more detailed background distractions than previous benchmarks such as ALE or ProcGen. \n\nThe paper runs an extensive collection of methods from the literature, such as metric loss, data augmentation, world models, and reconstruction loss methods. The results all suggest that there is no one single method which universally achieves strong performance over the distracting-background task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Proposed changes:\n* (Moderately Critical): Can concise mathematical notation (detailing some of the methods such as contrastive losses) be added to the paper? This would help a long way for the average reader to understand the context for the work. Currently it is difficult to judge the paper's worth because of the wordiness. Adding bullet points to summarize / organize the paper would also be very helpful.\n    * Currently I would vote borderline reject/accept purely for this reason. I believe that the paper is technically sound, but the presentation and wordiness can make it quite difficult to read.\n", "broader_impact_concerns": "No concerns found.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-Review of *Learning Representations for Pixel-based Control: What Matters and Why?*

As summarized by Reviewer U1dd: The work compares many different algorithms proposing learning representations for pixel-based reinforcement learning. The comparison of methods are run on a mix of ALE and MuJoCo tasks, using different levels of data augmentation or background distractions, in both sparse reward and dense reward environments. The methods themselves range between methods that just do data augmentation, to auxiliary task prediction, and methods that try to directly learn a representation via contrastive or self-supervised learning.

This paper will be of value to the RL community, and the growing community of folks working on self-supervised learning approaches for RL. As mentioned by Reviewer TtNM and JcWC, the conclusions drawn in Section 7 are interesting (esp to RL folks), especially showing that: (1) contrastive losses in the presence of rewards do not bring much benefit, (2) the success of certain augmentations, like random crop, on DM Control might be quite brittle. The work provides extensive experimental evidence to support its claims, by running numerous algorithmic baselines across a variety of different literature.

All 3 reviewers suggested fixes and improvements for the paper to improve readability and experiments, and the authors have addressed or responded to the suggestions, with the final paper in a much stronger shape compared to the initial manuscript. Therefore, I confidently recommend acceptance of the work.

Decision: Accept (poster)