Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning
Authors: Trevor McInroe
Lukas Schäfer
Stefano V Albrecht
Venue: Accepted by TMLR ()

Abstract: Learning control from pixels is difficult for reinforcement learning (RL) agents because representation learning and policy learning are intertwined. Previous approaches remedy this issue with auxiliary representation learning tasks, but they either do not consider the temporal aspect of the problem or only consider single-step transitions, which may cause learning inefficiencies if important environmental changes take many steps to manifest. We propose Hierarchical $k$-Step Latent (HKSL), an auxiliary task that learns multiple representations via a hierarchy of forward models that learn to communicate and an ensemble of $n$-step critics that all operate at varying magnitudes of step skipping. We evaluate HKSL in a suite of 30 robotic control tasks with and without distractors and a task of our creation. We find that HKSL either converges to higher or optimal episodic returns more quickly than several alternative representation learning approaches. Furthermore, we find that HKSL's representations capture task-relevant details accurately across timescales (even in the presence of distractors) and that communication channels between hierarchy levels organize information based on both sides of the communication process, both of which improve sample efficiency.

Published in Transactions on Machine Learning Research (01/2024)
Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning
Trevor Mclnroe
t. mmcinroe@ed.ccm
School of Informatics
The University of Edinburgh
Lukas Schäfer
Lschafer@ed..co ac.uk
School Informatics
The University of Edinburgh.
Stefano V. Albrecht
s.albrecht@ed.cco uk
School of Informatics
The University of Edinburgh
Reviewed on OpenReview: https: /ttps://openne wpw.r/vee...e.eee.eoeee net/ forum? id- fovUNTs
Abstract
Learning control from pixels is difficult for reinforcement learning (RL) agents because rep- resentation learning and policy learning are intertwined. Previous approaches remedy this issue with auxiliary representation learning tasks, but they either do not consider the tempo- ral aspect of the problem or only consider single-step transitions, which may cause learning inefficiencies if important environmental changes lake many steps to manifest. We propose Hierarchical k-Step Latent (HKSL), an auxiliary task that learns multiple representations via hierarchy of forward models that learn to communicate and ensemble of n-step crit- ics that all operate at varying magnitudes of step skipping. We evaluate HKSL in suite of 30 robotic control tasks with and without distractors and task of our creation. We find that HKSL either converges to higher or optimal episodic returns more quickly than several alternative representation learning approaches. Furthermore, we find that HKSL's 's repre- sentations capture lask-relevant details accurately across timescales (even the presence of distractors) and that communication channels between hierarchy levels organize information based on both sides of the communication process, both of which improve sample efficiency.
1 Introduction
Recently, reinforcement learning (RL) has had significant empirical success in the robotics domain (Kalash- nikov et al. 2018 2021 Lu et al.) 2021 Chebotar et al. 2021). However, previous methods often require dataset of hundreds of thousands or millions of agent-environment interactions to achieve their perfor- mance. This level of data collection may not be feasible for the average industry group. Therefore, RL's widespread real-world adoption requires agents learn satisfactory control policy in the smallest. number of agent-cnnvirnmenn interactions possible.
Pixel-based state spaces increase the sample efficiency challenge because the RL algorithm is required to learn useful representation and control policy simultaneously. A recent thread of research has focused developing auxiliary learning tasks to address this dual-objective learning problem. These approaches aim
to learn a compressed representation of the high-dinennsioall state space upon which agents learn control.
Several auxiliary task types have been proposed such as image reconstruction (Jaderberg et al. 2017
Yarats
let al.
2020),
contrastive objectives (Laskin et al. 2020al Stooke et al. 2021
image augmentation Laskin
et al.
2020b
Yarats al. 2021), and forward models GGelada al.
2019 Hafner et al. 2019 2020
Lee
et al.
2020a Zhang et al. 2021

---

Published in Transactions on Machine Learning Research (01/2024)
Forward models are a natural fit for RL because they target information across time by generating repre- sentations of the state space that capture information relevant to the environment's transition dynamics. However, previous approaches learn representations by predicting single-step transitions, which may not capture relovant information efficiently if various important environmental changes manifest on different timescales. For example, suppose an agent is learning to catch objects that fall from the sky at varying speeds. Here, the agent must make informed decisions that consider its own movement speed relative to the order in which the objects reach the agent's catching range. We demonstrate empirically in & 5.2 that various auxiliary tasks from the literature perform poorly in an environment that reflects this exact scenario. We also perform a linear probing exercise in & 5.4 that shows that representations learned by these current auxiliary tasks tend to be poor predictors of task-relevant information over various timescales.
In this paper, we introduce Hierarchical k-Step Latent (HKSL an auxiliary task for RL agents that explicitly captures information in the environment at varying magnitudes of temporal coarseness. HKST. accomplishes this by leveraging a hicrarchical latent forward model where each level in the hicrarchy predicts transitions with varying number of steps skipped. Levels that skip more steps should caplure a coarser understanding of the environment by focusing on changes that take more steps to manifest, and vice versa for levels that skip fewer steps. For cach level, HKSL trains an encoder paired with n-step critic function so that targets of the same temporal coarseness produce gradients for the learned representations. Also, HKSL learns to share information between levels via a communication module that extracts representations from coarser trajectories to help inform forward models that produce finer trajectories. As a result, HKSL learns a set of representations that give the downstream RL algorithm information on objects that move at different speeds. These representations are leveraged individually for value learning across various temporal coarseness levels by an ensemble of critics and jointly for action selection.
We evaluate HKSL and various baselines in a suite of 30 DMControl tasks (Tasa et al. 2018 Stone et al. 2021) that contains enviromments without and with distractors of varying types and intensities. Also, we evaluate our algorithms in "Falling Pixels", an environment of our crcation that requires agents to track objects that move at varying speeds, a lask that exactly reflects the scenario we described previously. We test our algorithms with and without distractors because real-world RL policies need to work well in controlled scttings (c.g., a laboratory) and uncontrolled settings (c.g., public strect). Also, distractors may change at speeds independently from task-relevant information, thereby increasing the challenge of relating agent actions to changes in pixels. The goal in our study is to learn a well-performing control policy in the smallest number of agent-environment interactions as possible.
In our DMControl experiments, HKSL reaches an interquartile mean of evaluation returns that is 29% higher than DrQ (Varats et al. 2021), 74% higher than CURL. (Laskin et al., 2020a), 24% higher than PI-SAC (Lee) et al. 2020b), 359% higher than DBC (Zhang et al. 2021), and 56% higher than Dreamer (Hafner et al. 2021 Also, our experiments in Falling Pixels show that HKSL converges to an interquartile mean of eval- uation returns that is 24% higher than DrQ, 35% higher than CURL, 31% higher than PI-SAC, 44% higher than DBC, and V2 fails to learn. We analyze HKSL's hicrarchical model and find that its represen- tations more accurately capture task-relevant details earlier on in training than the baselines. Additionally, we find that HKSL's communication manager considers both sides of the communication process, thereby giving forward models information that better contextualizes their learning process. Finally, we provide data from all training runs for all benchmarked methods.
2 Background
We study an RL formulation wherein an agent learns a control policy within a partially observable Markov de- cision process (POMDP) (Bellman 1957 Kaelbling et al. 1998), defined by the tuple (S, O, ps ps, R, 7). S is the ground-truth state space, O is a bixel-based obscrvation space, A is the action space, p Axx [0, 1] is the state transition probability function, po S x A O + [0, 1] is the observation probability fune- tion, R: : S x A + R is the reward function that maps states and actions to a scalar signal, and 7 E [0, 1) is a discount factor. The agent does not directly observe the state &t E S at step f, but instead receives an observation o E O which we specify as a stack of the last three images. At each step t, the agent
https://githuub..
2

---

Published in Transactions on Machine Learning Research (01/2024)
samples an action de E A with probability given by its control policy which is conditioned on the ob- servation at time 4, T(aeloz). Given the action, the agent receives reward transitions into a next state 5t+1 E S with probability ("(st, Ct St+1), and the next observation (stack of pixels) O++1 E O is sampled with probability (3(1+11, at 0t+1). Within this POMDP, the agent must learn a control policy that maximizes the sum of discounted returns over the time horizon T of the POMDP's episode: arg max= / EEi=1/ree ?/re]-
(s,,,4)) the POMDP
3 Related Work
Representation learning in RL. Some research has pinpointed the development of representation learning
methods that can aid policy learning for RL agents. In model-free RL, using representation learning objec-
tives as auxiliary tasks has been explored in ways such as contrastive objectives (Laskin et al. 2020a Stooke
ct al.
2021), image augmentation (Laskin et al. 2020b Yarats et al. 2021). image reconstruction Yarats
et al.
2020), information theoretic objectives (Lee el al., 2020b). causal disenlanglerment (Dunion et. al.
2023a b), and inverse models (Pathak et al.) 2017a Burda et al. 2019p. Other works build on Generalized
Value Functions (Schlegel et al.) 2021) to learn reprosentations with meta-gradient methods with predic- tions tasks that are bidirectional in time (Veeriah et al., 2019), or with random graphs (Zheng et al. 2021 HKST. fits within the auxiliary task literature but does not use contrastive objectives, image reconstruction, information-theoredd objectives, getta-gaadeenss, bi-directional prediction objectives, causal disentanglement, nor inverse models.
predeiciioo
Forward models and hierarchical models. Forward models for model-free RT. approaches learn repre- sentations that capture the environment's transition dynamics via a next-step prediction objective. Some methods learn stochastic models that are aided with image reconstruction (Lee et al. 2020a) or reward- prediction objectives (Gelada et al., 2019). Other methods combine forward models with reward prediction and bisimulation metrics (Zhang et al. 2021), momentum regression targets (Schwarzer ct al. 2021), : build on successor representations (Dayan 1993) to learn representations useful for tasks like transfer between MDPs (Barreto et al. 2017). shape the intermediate representations of an RNN with multi-step predic- tions
/Venkatraman et al.]
2017). Outside of the purpose of representation learning, forward models are
used extensively in model- mosed-based RL approaches to learn control policies via planning procedures (Ha &
Schmidhuber 2018 Zhang et al. 2019 Hafner el al., 2019: 2020). and to guide exploration lowards novel
states (Schmidhuber 1991a Pathak et al. 2017b Raileanu & Rocktäschel 2020 Schäfer et al, 2022 Mclnroe
ct al. 2023).
Stacking several forward models on top of one another forms the levels of hierarchical model. This type of model has been studied in the context of multiscale temporal inference (Schmidhuber 1991b), varia- tional inference CChung et al. (2017), and pixel-prediction objectives (Kim et 2019 Saxena et al. 2021 Additionally, hierarchical models have been used for speech synthesis (Kenter et al. 2019), learning graph embeddings (Chen et al. 2018), decomposing MDPs (Steccanella et al.) 2002]] modeling human cognition at various visual resolutions (Rao & Ballard 1999). and population-based multi-agent RL (Jaderberg et al., 2019). Sequence prediction literature has explored the use of hierarchical models via manually-defin con- nections between levels (Koutnik et al. 2014: Saxena et al. 2021) and using levels with uniform time-step
skipping (Castrejon et al. 2019 Kumar et al. 2020
Unlike the aforementioned forward model approaches, HKSL combines a set of forward models that step in the latent space with independent step sizes without additional prediction objectives. Also, HKSL uses a differentiable connection between forward models that Iearns what to share when by using the context from the entire rollout from higher levels and the current timestep of lower levels, which leads to faster learning.
4 Hierarchical k-Step Latent
HKSL's hierarchical model composed of forward models that take steps in the latent space at varying magnitudes of temporal coorseness. We define temporal coarseness as the degree to which a level's forward model skips environment steps. For example, if a forward model predicts the latent representation of state five steps into the future, it is considered more coarse than a forward model that predicts only one step
3

---

Published in Transactions on Machine Learning Research (01/2024)
forward. Coarser levels should learn to attend to information in the environment that takes many steps to manifest in response to agent's 's action. In contrast, finer levels should learn to attend to environmental properties that immediately respond to agent actions. This is because coarser levels need to make fewer prodictions to reach steps further into the future than finer levels.
At each learning step, batch of B trajectories of length k are sampled from the replay memory T = o++ The initial observation of each trajectory or is uniformly randomly sampled a per-episode basis t ~ - k In the following, we will denote the first and last timestep of cach trajectory with t == and t k respectively.
{(o,, dt...., 01+k-1, 0.....,(++k-1,,
HKSL's components. See Figure ! for a visual depiction of the HKSL architecture. HKSL's hierarchical model is composed of h levels. Each level I has a forward model I', a nonlinear projection module w' (e.g.. an MLP), an online image encoder ey, and a momentum image encoder e'n that is updated as an exponential moving average of the online encoder (c.g., (He ct al.] 2020)). Between consocutive levels a communication manager c/-11 passes information from one level 1 to the level below it 1-1. The number of steps skipped by a given level n' is independent of the coarseness of other levels in the hierarchy.
Forward models. HKSL's forward models are modified version of the common GRU recurrent cell (Cho et al. 2014) that allows for multiple data inputs at each step. See Appendix D.3 for a detailed mathematical description. At stept f = in a training trajectory, the forward models take the representation produced by the level's online encoder ア = (o1) along with a concatenation of ré' consecutive action vectors ã = [ail... (ull...(a)) to predict the latent representation of a future state " fl (21. ā1). For any following timestep t > 1, the forward models take the predicted latent. representation from the previous timestep as input. instead of the encoder reprosentation.
Communication managers. Communication managers dit-1 pass information from coarser to finer levels in the hierarchy (1 - 1 - 1) while also allowing gradients to flow from finer to coarser levels (1 - 1 + l). At each rollout step for level 1 - 1, the communication manager c/.-1 reccives two inputs. First, it receives all latent representations in a rollout over t produced by level I's forward model l Second, it receives a one- hot-encoded timestep that corresponds to the current rollout's timestep in level / The communication manager's job is to extract information from the above level's rollout that is relevant for the below level's predictions at each step. The context vectors produced by the communication manager are used an additional input into f'-1 for all levels but the uppermost level in the hierarchy. Additionally, gradients from losses computed in level | 1-1 flow upwards into level / through clul11 and are used to update parameters in above levels.
Loss function. HKSL computes loss value at each timestep within each level the hierarchy as the normalized l2 distance between nonlinear projection of the forward model's prediction and the "true" latent representation produced by the level's momentum encoder e. Using this "noisy" approximation of the target ensures smooth changes in the target between learning steps and is hypothesized to reduce the possibility of collapsed representations (Tarvainen & Valpola 2017) in a very similar manner to the Bootstrap Your Own Latent (BYOL) loss (Grill et al. 2020). The nonlinear projection is produced by feeding a forward model's output through the nonlinear module w' assigned to its layer in the hierarchy. Chen et al. (2020) show that a nonlinear projection before a loss computation can improve the representations produced by a previous learned layer. We verify this choice empirically 5.3) and show that it does improve policy learning in our used suit. of learning lasks. Altogether, the HKSL loss of level across the minibatch of trajectories can be written as:
N CHKSL Σ En En Iwo (f' ar (+)) c/n (o
(1)
where N is the number of steps that given level can take in r. Intuitively, Equation 1 encourages the online encoder e, to produce representations from which multiple forward-step predictions of temporal coarseness n' can be made accurately. For this to be possible, the ouline encoder must learn to extract
Fanding the range of numbers on T guarantees that trajectories do not overlap episodes.
4

---

Published in Transactions on Machine Learning Research (01/2024)
mm
8g
CuxsL
C SU
[a] [g [g)
aas [ss
2
01
e
2,11
2
time
1
f1
f1
Husz
C
HEsL
aust
Hus
C
HKSL
ar
Figure Depiction of HKSL architecture with an "unrolled" two-level hicrarchical model where the first level moves at one step n² = 1 and the second level moves at three steps n² =3 3. First, the online encoders e, (blue) encode the initial observation 01 of the sampled trajectory. Next, the forward models f (red) predict the latent representations of the following observations, with level predicting single steps ahead conditioned the level's previous representation and applied action. The forward model of the second level predicts three steps ahead and receives the previous representation and concatenation of the three applied actions. The communication manager (green) forwards information from the representations of the coarser sccond level to each forward model step of the first level as additional inputs. All models are trained end-to-end with a normalized fz loss of the difference between the projected representations of each level and timestep and the target representations of observations using momentum encoders em (purple), and projections done by the projection model (yellow) of the given level.
the predicted timesteps. Target reprosentations are obtained
information from its input pixels that relate directly to contents in the environment that move at speeds related to the given level's temporal coarseness.
HKSL and SAC. We make few adjustments to the base SAC algorithm to help HKSL fit naturally. For one, replace the usual critic with an ensemble of h critics. Each critic and target critic in the ensemble receive the latent representations produced by given level's encoder and momentum encoder, respectively. We allow crities' gradients to update their encoders' weights but we do not allow gradients from actor updates to update encoder weights Each critic is updated using n-stop returns whore n corresponds to the temporal coarseness ni of the level within which the critic's given encoder resides. By matching encoders and critics
Varats al. 2020) show that gradients can harm representation policy learning while critic gradients can help.
5

---

Published in Transactions on Machine Learning Research (01/2024)
in this way, we ensure encoder weights updated by gradients produced by targets of the same temporal coarseness.
Second, the actor receives a concatenation of the representations produced by all online encoders. HKSL's actors will make better-informed action selections because they can consider information in the environment that moves at varying magnitudes of temporal coarseness. Finally, we modify the actor's loss function use sum of Q-values from all crities:
Lattor
E Baws, onuT
Σ (o. -a -olggtallll0)))11e0000
(2)
5 Experiments
We evaluate HKST. with a series of questions and compare it against several relevant baselines. First, is HKSL more sample efficient in terms of agent-environment interactions than other representation learning methods ($ 5.2 Second, what is the efficacy of each of HKSL's components ($ 5.3)? Third, how well do
HKSL's encoders capture task-relevant information relative to the baselines' encoders? ($ 5.4 Finally,
what does cl consider when providing information to l - 1 from I ($ 5.4
5.1 Experimental Setup
Baselines. We use DrQ (Yarats et al. 2021), CURL
(Laskin al. 2020ah, PI-SAC (Lee al 2020b);
DBC (Zhang et al. 2021 and Dreamer V2 Hafner al. by averaging lemporal difference targels across several augmentations of the same images. CURL uses a contrastive loss similar to CPC (van den Oord et al.
2021) basclines. DrQ regularizes Q-value learning
2018) to learn image embeddings. PI-SAC uses a Conditional Entropy Bottleneck (Fischer) (2020) auxiliary loss with both a forward and backward model to learn a representation of observations that capture the environment's transition dynamics. DBC uses a bisimulation metric and a probabilisite forward model to learn representations invariant to task-irrelevant features. DreamerV2 masel-based method that performs planning a discrete latent space All model- free methods use SAC (Haarnoja al., 2018a b) as the base RL algorithm, while DreamerV2 leverages on-policy actor-critic method with X-target critic (Schulman al. 2016). All methods use the same encoder, critic, and actor architectures to ensure a fair comparison. Additionally, each method uses the same image augmentation. See Appendix D for hyperparameter settings.
Environments. We use six cantinuous-contrra environments provided by MuJoCo [Todorov et al. 2012 via the DMControl suite (Tassa al. 2018 2020), a popular set of environments for testing robotic control algorithms. Each of the six environments uses episodes of length 1k environment steps and set number of action repeats that controls the number of times the environment is stepped forward with a given action. We use five variations of each DMControl environment for total of 30 tasks. Four of the variations use distractors provided by the Distracting Control Suite API (Stone et al.] 2021), and the fifth variation uses no distractors. We use the "color" and "camera." distractors on both the "easy" and "medium" difficulty settings. The color distractor changes the color of the agent's pixels on cach environment step, and the camera distractor moves the camera in 3D space each environment step. The difficulty setting controls the range of color values and the magnitude of camera movement in each task[]
environment of our design, which we call "Falling Pixels". In Falling Pixels, the agent controls platform at the bottom of the screen and is rewarded +1 for each pixel it catches. Pixels fall from the top of the screen and are randomly assigned a speed when spawned, which controls how far they travel downwards with each environment step. Soe Appendix C for further information on the environments.
Additionally,
5.2 Sample Efficiency
Training and evaluation procedure. In our training scheme, agents perform an RL and representation learning gradient update once per action selection. Every 10k environment steps in DMControl and 2.5k
Refer Stone al. 2021 for details.
6

---

Published in Transactions on Machine Learning Research (01/2024)
Sample Efficiency in DMControl Suite
DreamerV2
0.5
DBC
IQM
Optimality Gap
Evaluation Retums
0.4
PI-SAC
CURL
HKSL
DrQ
DrQ
0.3
HKSL
CURL
0.2
PI-SAC
DBC
0.1
DreamerV2
0.0
0.15
0.30
0.45
0.60
0.75
2
6
10
Normalized Score
Environment Steps (x109)
Figure 2: 1QM (Ieft) and optimality gap (middle) of evaluation returns at 100k environment steps, and 1QM throughout training (right) across all 30 DMControl tasks. Shaded arcas are 95% confidence intervals.
environment steps Falling Pixels, we perform an evaluation checkpoint, wherein the agent's policy is sampled delerministically as the mean of the produced action distribution, and we compute the average performance across 10 episodes. All methods are trained with batch size of 128. We train agents for 100k and 200k environment steps for five seeds in DMControl and Falling Pixels, respectively.
Results. We use the "rliable" package (Agarwal et. al. 2021 to plot statistically robust summary metrics in our evaluation suite. To produce aggregate metrics, we normalize all DMControl returns to the maximum per-episode returns, which is 1k for all tasks. Specifically, Figure shows the interquartile mean (1QM) (left) and the optimality gap (middle) along with their 95% confidence intervals (CIs) that are generated via stratified bootstrap sampling the 100k steps mark in DMControl. Optimality gap measures the amount by which given algorithm fails to achieve perfect score Additionally, Figure shows IQM and 95% Cls as function of environment steps (right) in DMControl. Both of these results show that HKSL significantly outperforms the baselines across our 30 environment DMControl testing suite. See Appendix environment results. We note that simply using a forward model does not guarantee improved performance, as suggested by the comparison between HKSL, PI-SAC, and DBC.
F for individual
Due to the randomness in Falling Pixels, the maximum per-epi return is difficult to calculate. Therefore, do not aggregate Falling Pixels with DMControl returns, but instead show the IQM and 95% CIs for Falling Pixels a function of environment steps in Figure (left). We highlight that HKSL significantly outperforms all of the baselines, converging to performance of collecting over 20% more pixels per episode than the next-best- -ereee-terfommigggg algorithm. Collecting a large number of pixels in Falling Pixels requires agents to keep track of environment objects that move at varying speeds. HKSL explicitly achieves this with its hierarchy of forward models. Also, we note that. DreamerV2 struggles relative to the other agents in Falling Pixels. We hypothesize that this is due to Falling Pixels' observation space characteristics: the important information is single-pixel-sized. [Hffnr et al. (2021) show that reoostriconnttttiionn gradients are important to Dreamer V2's success (Figure 5 in Hafner et al. (021 and the small details in Falling Pixels cause uninformative reconstruction gradien
5.3 Component Ablations
We probe each component of HKSL to determine its contribution to the overall RL policy learning process.
Specifically, we test SAC without the hierarchical model but with HKSL's ensemble of critics (No Repr), HKST. where each level in the hierarchy moves with a single step (All n = 1), HKSL without e (No c), HKST. where each level in the hicrarchy shares encoders (Shared Encoder), single-level HKSL (h = 1), and HKSL.
For all plots, performed at least 5,000 samples. We note that perfoct score (optimality gap 0) technically impossible the DMControl suite. As such, only the relative positioning of CIs should be considered. Hafner al. 2021 also give this reason for why DreamerV2 does poorly the Video Pinball" environment.

---

Published in Transactions on Machine Learning Research (01/2024)
Evaluation Returns in Falling Pixels
Evaluation Returns in Falling Pixels
14
HKSL
DrQ
Evaluation Returns
12
CURL
Evaluation Returns
DBC
12
10
PI-SAC
Dreamerv2
8
h
h=1
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Environment Steps (x10%)
Environment Steps (x10²)
Figure 3: IQM and 95% CIs of evaluation returns for all algorithms in Falling Pixels (left) and ablations over HKSL's h (right).
Ablations in Cartpole, Swingup
Abl Ablltions in Ball in Cup, Catch
Abl atlinss in Wol ker, Walk
0.8
1GM Noraaizon Score
IOM Normalized Score
0.6
Normalizan
0.6
0.4
0.0
0.0
0.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.6
0.5
0.9
0.4
0.5
Environment Stops Dx
Enviranment Steps (x10')
Environment Staps (x10')
Figure 4: IQM 95% Cls of evaluation returns for HKSL ablations in Cartpole, Swingup (left), Ball in Cup, Catch (middle), and Walker, Walk (right).
with no nonlinear projection (No w). The No Repr ablation tests whether HKSL's performance boost is due to the ensemble of critics or the hierarchical model itself. The All n ablation tests our hypothesis that only learning representations at the environment's presented temporal coarseness can miss out on important information. The No e ablation tests the value sharing information between levels. The Shared Encoder ablation tests if one encoder can learn information at varying temporal coarseness. The h ablation tests the value of the hierarchy itself by using single forward model (e-g., (Schwarzer al. 2021 McInroe al., 2)21)). Finally, the No ablation tests the value in the nonlinear projection between the forward models and the loss compulation.
See Figure for the performance comparison between these ablations and full HKSL in the no distractors setting of Cartpole, Swingup, Ball in Cup, Catch, and Walker, Walk. All results are reported as IQMs and 95% CIs over five seeds. We highlight that variations without all components perform worse than full HKSL. This suggests that HKSL requires each of the individual components achieve its full potential.
Also, ablate across the number of levels h in HKSL's hicrarchy in Falling Pixcls. Figure 2 (right) depicts IQMs and 95% CIs over five seeds for values of h in the set 21,2,3, 4] with temporal coarseness of levels set to [1,3,5, 5, 7] for levels one through four, in order. We highlight that increasing h achieves monotonic improvement in evaluation returns up to when h = 4. We hypothesize that setting h = 3 captures all relevant information in Falling Pixels, and increasing to = 4 leads to similar returns as when and does not destabilize learning.
5.4 Representation Analysis
How well do representations align with task-relevant information? To testt the ability of encoders to retrieve task-relevant information from pixel input, we the weights of the encoders for each method
8

---

Published in Transactions on Machine Learning Research (01/2024)
Cartpole, Swingup
No Distractors
Color
Camera
1.0
1.0
DVQ
0.8
CURL
6.-
1.0
DBC
MSE
Droamer/2
6.
62
0.0
x+
[+3 5
++
t+
(+ 10
Ball in Cup, Catch
Distractors
Color
Camera
11/22)
(1100
0.000
0.000
0.003
0.03
2+2
1+3
2+1
L+2
(+3
t+2
++
Figure MSE task-relevant information in unseen episodes for Cartpole, Swingup (top) and Ball in Cup, Catch (bottom) at the 100k environment steps mark. Non-distraction, color distractor, and camera distractor settings shown from left-to-right. Lower is better.
throughout training in our evaluation suite. We then use the representations produced by these encoders to train linear projection (LP) to predict reskvaelevant information over varying timescales. This process is akin to linear probing (Alain & Bengio 2017) used to analyze representations (e.g., Anand et al. 2019). We note that the encoders' weights are frozen, and the gradient from the prediction task only updates the LP's weights.
In the Cartpole, Swingup task, the objective is to predict the cart's and pole's coordinates. In the Ball in Cup, Catch task, the objective is to predict the ball's coordinates. We collect 10 and five episodes of image- coordinate pairs in each environment for LP training and lesting, respectively. We repeat this data-collection exercise for both environments' non-distraction, easy color distractors, and easy camera distractors versions. After fitting the LP on the training sets, we mcasure the mean squared error (MSE) on the unseen testing set. Figure 5 shows the average MSE and + one standard deviation over the testing episodes using encoders trained for 100k environment steps in our benchmark suite. In Cartpole, Swingup (top row), we use the LPP to predict coordinates from one (t 1), five +t+ 5) and 10 + 10) steps into the future. In Ball in Cup, Catch (bottom row), into the future. We highlight that HKSL's encoders produce representations that more accurately capture task-relevant information with the lowest variance in nearly every case. Also, this accuracy is relatively unaffected by distraction settings, giving a reason for HKSL's relatively strong performance in the presence of distractors, despite not addressing distractors explicitly.
the LP to predict coordinates from one (t-1), two (t+2) and three (t+ steps
We repeat this process using encoders from carlier in the agent-training process. Figure F shows the MSE and + one standard deviation over the testing episodes using encoders trained for 50k environment steps in our benchmark suite. We note that the same pattern from the 100k environment steps encoders persists, These results suggest that HKSL agents benefit from more informative representations in carlier stages of training than the baselines, which leads to better sample efficiency.
What does e consider? We hypothesize that the communication manager d'1-1 provides a wide diversity of information for fl-1 by taking into account the current transition of the below level 1-1 as well as the representations from the above level L. To check this hypothesis, we perform two tests. First, we measure the & distance between the vectors produced by e when the step & is changed and other inputs are held
9

---

Published in Transactions on Machine Learning Research (01/2024)
Cartpole, Swingup
No Distractors
Color
Camera
DIO
1.2
CURL
1.0
1.0
R-SAC
0.8
0.6
0.6
0.4
0.
0.2
0.0
0.0
0.0
1+55
[+
1+5 5
1+
1+
1+5
++
Ball in Cup, Catch
No Distractors
Color
Camera
0.030
0.035
0.025
0.030
MSE
0.020
0.010
0.010
0.005
0.000
0.000
0.000
1+1
4+2
t+3
f+1
f+
++3
Figure 6: MSE on task-relevant information in unseen episodes for Cartpole, Swingup (top) and Ball in Cup, Catch (bottom) at the 50k enviromment steps mark. Non-distraction, color distractor, and camera distractor settings shown from left-to-right. Lower is better.
& Distance Between Vectors from c
PCA Projections of c's Output
0.0
1.3
2.1
3.2
4.2
5.1
10
0.0
1.3
3.7
0.0
1.4
2.7
3.8
0.0
1.5
2.8
0.0
-5
0.0
2
5
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
Figure 7: Average distance between vectors produced by (top). The numbers along the side and bottom correspond to the value of t. PCA projections of representations produced by for multiple timesteps across trajectories (bottom) with colors corresponding to trajectories.
fixed. If e completely ignores I, the distance between e(, 1) and e(.,4) 4), for example, would zero. Second, we examine the scparability of c's outputs on trajectory-wise basis. If two sampled trajectorics are very different, then the representations produced by the above level should change c's output such that either trajectory should be clearly separable.
We first train an HKSL agent where h = 2, n¹ 1, and n² = 3 in Cartpole, Swingup for 100k environment sleps and collect 50 episodes of experiences with a random policy. Then, we randomly sample a trajectory from this collection and step through the latent space with both forward models. We repeat this 100 times and measure the pairwise &a distance between e's outputs for every value of within sampled trajectorios. Figure (top) reports the average distance between each pair. We note that the distance between c's output grows the steps between the pairs grows. This suggests that considers the transition of the level below it
10

---

Published in Transactions on Machine Learning Research (01/2024)
when deciding what information to share. Additionally, we highlight that the distance increases consistently where pairs that are the same number of steps apart are about the same distance apart. For example, pairs (2,5) and (3,6) are both three steps apart and share roughly the same average l2 distance. This suggests that e produces representations that are grouped smoothly in the latent space. Figure [ (bottom) visualizes the PCA projections of c's outputs from 18 randomly sampled trajeclories, where each trajectory is a different color. This figure confirms our second intuition, as the representations are clearly separable on trajectory-wise basis with representations smoothly varying across steps within the same trajectory.
6 Conclusion, Limitations, and Future Work
This paper presented Hicrarchical k-Step Latent (HKSL), an auxiliary task for accelerating control learning from pixels via hierarchical latent forward model. Our experiments showed that HKST's representations substantially improve the performance of downstream RL agents in pixel-based control tasks, both in terms of converged returns and sample elficiency. We also showed that HKSL's representations more accurately capture task-relevant information than the baselines and do so early in training. Finally, we showed that the communication manager organizes information in response to the above and below levels.
Despite its relatively good performance, the nested loop in HKSL's learning step (see Algorithm I can incur significant compute cost if the number of levels in the hierarchy grows large. In addition, the optimal number of levels and temporal coarseness of HKSL model may not be as clear it is in the Falling Pixels environment. Future work could develop method that can learn to adjust the hierarchy dynamically, thereby avoiding the need to tune the number of levels and their temporal coarseness. Also, future work could consider using HKSL-style hierarchical models for purposes other than representation learning, such as general model-based RL algorithms, exploration or planning procedures.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Leurning Representations (Workshop Track), 2017.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXio preprint arXiv:1607.064500 2016.
Andre Barreto, Will Dabney, Remi Munos, Jonathan J. Hunt, Tom Schaul, Hado P. van Hasselt, and David Silver. Successor features for transfer in reinforcement learning, In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Richard Bellman. A markovian decision process. Indiana University Mathematics Journal, 6:679-688 -64, 1957.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexci A. Efros. Large-scale study of curiosity-driven learning. Tn International Conference on Learning Representations (ICLR), 2019.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrons for video prediction. In International Conference on Computer Vision (ICCV), 2019.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, and Chelsea Finn andyou Sergey Levine. Actionable models: Unsu- pervised offline reinforcement learning of robotic skills. In Proceedings of the 38th International Conference on Machine Learning (ICML.), 2021.
11

---

Published in Transactions on Machine Learning Research (01/2024)
Haochen Chen, Bryan Perozzi, Yifan Hu, and Steven Skiena. Harp: Hicrarchical representation learning for networks. In The Shirly-Second AAAT Conference on Arlificial Intelligence (AAAT-18), 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In Internation Conference on Machine (ICML), 2020.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntaz, Semantics and Structure in Stabistionl Translation, pp. 103-111, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations (ICLR), 2017.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Newral Compatation 1993.
Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah P. Hanna, and Stefano V. Albrecht. Con- ditional mutual information for disentangled representations in reinforcement learning. In Conference on Newral Information Processing Systems (NeurIPS), 2023a.
Mhairi Dunion, Trevor Mclnroe, Kevin Sebastian Luck, Josiah P. Hanna, and Stefano V Albrecht. Temporal disentanglement of representations for improved genera lisation in reinforcement learning. In International Conference on Learning Representations (ICLR), 2023b.
Ian Fischer. The conditional entropy bottleneck. Entropy, 2020.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. DeepMDP: Learn- ing contimuous latent space models for representation learning. In Proceedings of the 36th International Conference Machine Learning (TCML), 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning. In 34th Conference on Neural Information Processing Systems (NewrIPS), 2020.
David Ha and Jürgen Schmidhuber. World models. arXio preprint arXiv:1803. 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning (ICML), volume 80, pp. 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900 059005 2018b.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), pp. 2555-2566 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Represenlations (ICLR), 2020.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In Internation Conference on Learning Representations (ICLR), 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 9726 9735, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of The Thérly-Second AAAT Conference on Artificial Intelligence (AAAI-18), 2018.
Riashat. Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. In Proceedings of the ICML 2017 workshop Reproducibilily in Machine Learning (RML), 2017.
Max Jaderberg. Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Jocl Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattic, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukeuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. Science, 364:859 865, 2019.
Leslie Pack Kaelbling, Michael L.. Littman, and Anthony R. Cassandra. Planning and acting partially observable stochastic domains. Artificial Intelligence, 101(1):99-134, 1998.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable deep rein- forcement learning for vision-based robotic manipulation. In 2nd Conference on Robot Learning (CoRL), 2018.
Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Scaling up multi-task robotic reinforcement learning. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Tom Kenter, Vincent Wan, Chun-An Chan, Rob Clark, and Jakub Vit. CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. In 33rd Conference on Newral Information Processing Systems (NewrIPS), 2019.
Jan Koutnik, Klans Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork rmn In Proceedings of the 3ist International Conference on Machine Learning (ICML), 2014.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video generation. In International Conference on Learning Representations (ICLR), 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 5639-5650, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Picter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In 34th Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 11884-19895, 2020b.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 741 752, 2020a.
Kuang-Huc. Loe, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In Advances in Neurol Information Processing Systems (NeurIPS), volume 33, pp. 11890-11901, 2020b.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Trpan, Mohi Khansari, Dmitry Kalashnikov, and Sergey Levine. Aw-opt: Learning robotic skills with imitation and reinforcement at scale. In roceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Trevor McInroe, Lukas Schäfer, and Stefano V. Albrocht. Learning temporally-consistent representations for data-efficient reinforcement learning, arXiv preprinl: arXic:21 10. 04935, 2021.
Trevor Mclnroe, Stefano V. Albrecht, and Amos Storkey. Planning to go out-of-distribution in offline-to- online reinforcement learning. arXiv preprint arXin:2310. 0.70030 2023.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosily-driven exploration by self- supervised prediction. In International Conference on Machine Learning (ICML), 2017a.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference machine learning, pp. 2778-2787 PMLR, 2017b.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference Learning Representations, 2020.
Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation some extra-classical receptive-field effects. Nature Neuroscience, 1999.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autocncoders. In 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
Lukas Schäfer, Filippos Christianos, Josiah P Hanna, and Stefano V Albrecht. Decoupled reinforcement learning to stabilise mntrinsically-motivatee exploration. In International Conference Autonomous Agents and Multiagent Systems, 2022.
Matthew Schlegel, Andrew Jacobsen, Zaheer Abbas, Andrew Patterson, Adam White, and Martha White. General value function networks. Journal of Artificial Intelligence Research, 70, 2021.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural con- trollers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animals, pp. 222 227, 1991a.
Jürgen Schmidhuber. Neural sequence chunkers. Technical report, 1991b.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbocl. High-dimensional con- tinuous roltrol using generalized advantage estimation. In International Conference Learning Repre- sentations (ICLR), 2016.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data- efficient reinforcement lcarning with self-prodictive representations. In International Conference on Learn- ing Represenlations (ICLR), 2021.
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Lorenzo Steccanella, Simone Totaro, and Anders Jonsson. Hierarchical representation learning for markov docision processes. arXiv preprint: arXiv:2106.01655, 2021.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722. 2021.
Adam Stooke, Kimin Loe, Picter Abbocl, and Michael Laskin. Decoupling representation learning from reinforeement learning. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pp. 9870 08709997, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency tar- gets improve semi-supervised deep learning results. In 31st Conference Neural Information Processing Systems (NeurIPS), 2017.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint (1fi::8800000900 2018.
Yuval Tassa, Saran Tunyasuvunkkool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm control: Software and tasks for continuous control. arXiv preprint XrXie:2006. 12983, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 2012.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive
coding. arXiv preprint ar arX:::1877.03 03748, 2018.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard Lewis, Junhyuk Oh, Hado P. van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Arun Venkatraman, Nicholas Rhinchart, Wen Sun, Lerrel Pinto, Martial Hebert, Byron Boots, Kris M. Kitani, and J. Andrew Bagnell. Predictive-state decoders: Encoding the future into recurrent networks. In Advances in Newral Information Processing Systems (NeurIPS), 2017.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. ar preprint arXiv:1910.017/4 2020.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without roconstruction. In International Conference on Learning Representations (ICLR), 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement. learning- In International Conference on Machine Learning (ICML), 2019.
Zeyu Zheng, Vivek Veeriah, Risto Vuorio, Richard L Lewis, and Satinder Singh. Learning state represen- tations from random deep action-conditionall predictions. In Advances Neural Information Processing Systems (NeurIPS), 2021.
A Appendix
B Extended Background
Soft Actor-Critic. Soft Actor-Critic (SAC) (Haarnoja et al. 2018a| b) is popular off-policy, model- free RL algorithm for continuous control. SAC uses stato-action value-functon critic Q and target critic Q, stochastic actor π, and learnable temperature that weighs between reward and entropy: Eor E,, R(or, az) + (H(T--)))))
SAC's critic is updated with the squared Bellman error over historical trajectories T = -or, ae, Te. 01+1) sam- pled from a replay memory D:
Ceritic E..DD((((Q))) ar) (ra + ?)))
(3)
15

---

Published in Transactions on Machine Learning Research (01/2024)
where y is computed by sampling the current policy:
y = Be' (Q((+1, - log
(4)
The target critic Q does not receive gradients, but is updated as exponential moving average (EMA) of Q (e.g.. He et al. (2020)). SAC's actor parameterizes multivariate Gaussian ((, σ)) where μ is vector of means and a is the diagonal of the covariance matrix. The actor is updated via minimizing:
Lactor -R log log=(a)):))s
and or is learned against a static value.
C Environments
Table | outlines the action space, the action repeat hyperparameter, and the reward function type each environment used in this study. The action repeat hyperparameters that are displayed in the table are the standards as defined by (Hafner al.) (219)) and are the same used in most studies in DMControl. The versions of each environment with distractors follow the presented information as well.
Table 1: Dimensions of action spaces, action repeat values, and reward function type for all six environments in the DMControl benchmark suite and Falling Pixels.
Environment, Task
dim((A)
Action Repeat
Reward Type
Finger, spin Cartpole, swingup Reacher, easy Cheetah, run Walker, walk
2
Dense
8
Dense
4
Sparse
4
Dense
6
2
Dense
Ball in Cup, catch 2
4
Sparse
Falling Pixels
1
Dense
The Falling Pixels enviromment is rendered as a 35 x 15 grayscale image. The agent is confined to the bottom row and pixels are spawned at the top row. The agent is placed randomly along the bottom row and the top row filled with pixels the beginning of each episode. With each environment step, the pixels travel downwards until they reach the bottom row. If the agent occupying pixel's column when it reaches the bottom row, that pixel is collected and the agent is rewarded +1. Regardless of whether pixel is collected, it disappears from the board once it reaches the bottom row. When a column does not have pixel within it, there is a 2.5% chance for new pixel to be spawned in that row each environment step. When spawned, the pixel assigned speed from the set (1 uniformly at random. Each episode is 250 environment steps.
Architecture and Hyperparameters
D.1 SAC Settings
All encoders follow the same architecture as defined by @Yarats et al. 2020). These encoders are made of four convolutional layers separated by ReLU nonlinearities, linear layer with 50 hidden units, and final layer norm opertion (Ba al. 2016). Each convolutional layer has 32 3x3 kernels and the layers have stride of 2, 1, 1, and 1, respectively. This in contrast to the encoder used in the PI-SAC study (Lee et al. 2020b). which uses Filter Response Normalization (Singh & Krishnan, 2020 layers between each convolution.
The architectures used by the SAC networks follow the same architecture as deinfed by Yarats et al. 2020 Both the actor and critic networks have two layers with 1024 hidden units, soparated by ReLU nonlincaritics. This is in contrast to the networks used in the PI-SAC study, which uses different number of hidden units in the actor and critic networks.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Several studies have shown that even small differences in neural network architecture cause statistically signficant differences in performance (Islam et al. 2017 Henderson et al. 2018 As such, we avoid using the original PI-SAC encoder and SAC architectures to ensure fair study between all methods.
Table shows the SAC hyperparameters used by all methods in this study. For method-specific hyperpa- rameters (e.g., auxiliary learning rate, architexture of auxiliary networks, etc.), we defaulted the settings provided by the original authors.
Table SAC Hyperparameters used to produce paper's main results.
Hyperparameter
Value pixels
Image padding
Initial steps
1000
Stacked frames
3
Evaluation episodes Optimizer
10
Adam
(31, 92) Optimizer Learning rate
(0.9, 0.999)
3
Batch size
128
Q function EMA Encoder EMA
0.01
0.05
Target critic update freq
2
dim(z)
50
? Initial
0.99
0.1
Target a
|A4
Replay memory capacity Actor log stddev bounds
100,000
[-10,2]
D.2 HKSL Hyperparameters
Table shows the hyperparameterss that control HKSL. represents the number of levels, contains list of the skips of each level from lowest to highest level, & shows the length of the trajectory sampled at each training step, learning rate corresponds to the learning rate of all HKSL's components, and actor update freq corresponds to the number of steps between each actor update. These hyperparameters were found with a brief search over the non-distractor setting of each environment.
HKSL's communication manager e is a simple two-layer nonlinear model. The first layer has 128 hidden units and the second has 50. The two layers are separated by ReLU nonlinearity.
Table 3: Hyperparameters used for HKSL for each environment.
Environment, Task
h
Learning rate Actor Update Freq
Finger, spin
2
[,3]
1e-4
2
Cartpole, swingup
2
[13
1c-3
1
Reacher, easy Cheetah, run
2
[1,3]
le-4
2
2
[4.5]
10
le-4
2
Walker, walk
2
[133]
le-3
I
Ball in Cup, catch
2
[1,3]
1e-3
1
Falling Pixel
[1,,5]
le-3
1
17

---

Published in Transactions on Machine Learning Research (01/2024)
D.3 HKSL's Forward Models
The usual GRU formulation at step t:
"yu = 5 (fgrn (2)-
(6)
1rn = ((rr(((a((zz))))
(7)
hgruu tanh(fgru(( gra @ 2- (1-1(a=))
Sgru -1- (gru) 3 20-1 + ugru gru
where each each distinct is affine transform, is the sigmoid nonlincarity, and O is the Hadamard product. In order to allow the forward models to take the optional input from we add identical set of additional affine transforms:
- = =fo
(10)
r ([Co(((((()))))))))
(11)
h = tanh(fe (1r: & C((=-1))) g (1 u 2-1 uh h
(12)
(13)
where C2 denotes the output from at step t. Finally, the output of the forward model is the average of the two pathways:
ge grru
(14)
2
E
Attention Maps
We examine the encoders within HKSL's hicrarchy ascertain their objects of focus. Each encoder receives gradients relating to different magnitude of temporal coarseness. Therefore, each encoder should learn "focus" on different aspects of input images. The top row in cach plot shows the unstacked frames that go into the past from right to left (e.g,, the framestack depicted with images as [o--2, 0r- o.].) The bottom row of each plot shows the attention maps from each encoder. The attention maps are generated by taking the output of the final convolutional layer, post-activation, and averaging across the feature map dimension. Doing so collapses the output feature maps into a singlo-channel image with the most "active" portions of the image highlighted. All encoders are from HSKL agents after 100k environment steps of training.
Figure g depicts a scenario from Cartpole, Swingup. We note that the encoder from the first level (left) attends to the pole, an object that is not controlled by the agent. In contrast, the encoder from the second level (right) attends to the cart, which is directly controlled by the agent. Figure also depicts scenario from the Cartpole, Swingup environment. Here, the cart is offscreen for one frame in the stack. Here, we The encoder from the first and second level pay more attention to the
see the same pattern as in Figure g
pole and the cart, respectively.
Figure 10 depicts a scenario from the Ball Cup, Catch environment. We highlight that the encoder from the first. level (left) appears atlend entirely the information from the most recent frame in the input stack. In contrast, the encoder from the second level (right) gathers the full trajectory of information from each frame in the stack. This phenomenon is especially apparent in Figure I1 where the encoder from the second level (right) captures the trajectory of the ball as it falls into the cup.
F Individual Environment Results
This section shows the mean (bold lines) + one standard deviation (shaded area) for every individual environment and distractor combination. Figure [2] displays the non-distractor environments, Figure [3] shows the color distractors on the easy setting, Figure 14 shows the color distractors on the modium sotting, Figure 15 shows the camera distractors the easy settings, and Figure 16 shows the camera distractors on the medium setting.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Figure 8: Input frame stack (top row) and corresponding attention maps (bottom row) for a scenario from Cartpole, Swingup. Encoder from first and second level shown the left and right, respectively.
Figure 9: Input frame stack (top row) and corresponding attention maps (bottom row) for a scenario from Cartpole, Swingup. Encoder from first and second level shown on the left and right, respectively.
G Pseudocode
19

---

Published in Transactions on Machine Learning Research (01/2024)
U
Figure 10: Input frame stack (top row) and corresponding attention maps (bottom row) for a scenario from Ball in Cup, Catch. Encoder from first and second level shown on the left and right, respectively.
J
Figure 11: Input frame stack (top row) and corresponding attention maps (bottom row) for a scenario from Ball in Cup, Catch. Encoder from first and second level shown on the left and right, respectively.
20

---

Published in Transactions on Machine Learning Research (01/2024)
No Distractiors
Cartpole, Swingup
Cup. Catchh
Walker, Walk
HKSL DrQ
Eysluation Reuums
CURL
500
600
DresmorV2
400
400
200
0-
Ringer,
Reacher,
700
800
Eyalvation Roums
800
200
100
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10))
Envtroement Stepe (x16f)
Environment Steps (x10")
Figure 12:
Evaluation returns for agents trained DMControl without distractors. Bold line depicts the mean and shaded area represents +- one standard deviation across five seeds.
Color Distractiors (Easy) Cup,
Cartpole, Swingup
HKSL
700
DKQ
800
800
Explustoon Rotims
CURL
500
SAC
400
800
DreamerV22
400
300
200
200
Cheetah, Run
finger. Spin
Reacher, Easy
800
500
Coliaton aetums
400
400
300
200
200
0.2
0.4
0.6
0.8
1.0
6.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Environmen Staps (x107)
Enstronment Steps (x107)
Environment Steps 1x
Figure 13: Evaluation returns for agents trained DMControl with color distractors on the easy setting,
Bold line depicts the mean and shaded area represents +- one standard deviation across five scods.
21

---

Published in Transactions on Machine Learning Research (01/2024)
Color Distractiors (Medium)
Cartpole, Swingup
Cup. Catch
Walker, Walk
1000
800
HKSL
DrQ
Explustion Rdb mo
CURL
500
100
DresmorV2
200
200
Cheetah,
Hinger, Spin
Reacher,
800
Eoliatter Retums
500
300
100
400
260
100
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x10")
Envtromment Stepe (x16f)
Envirinment Steps (x10")
Figure 14: Evaluation returns for agents trained in DMControl with color distractors on the medium setting. Bold line depicts the mean and shaded area represents + one standard deviation across five seeds.
Camera Distractiors (Easy) Cup,
Cartpole, Swingup
600
HKSL
DKQ
400
Eysipatonn Rotums
CURL
SAC
300
DreamerV22
400
200
100
Cheetah, Run
finger. Spin
Reacher, Easy
800
800
900
Volatton Retums
800
250
200
200
200
0.2
0.4
0.6
0.8
1.0
6.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Enwironment Stags (4107)
Enstronment Steps 6:109)
Environment Steps 1x
Figure 15: Evaluation returns for agents trained in DMControl with camera distractors on the easy setting, Bold line depicts the mean and shaded area represents +- one standard deviation across five scods.
22

---

Published in Transactions on Machine Learning Research (01/2024)
Camera Distractiors (Medium)
Cartpole, Swingup
Ball Cup, Catch
Walker, Walk
300
HKSL
Exliuston SalistonReeerre Returio
DVQ
600
250
400
200
DreamerV2
150
200
100
2000
50
0-
finger, Spin
Easy
700
600
600
Eveluation Retums
200
500
150
400
300
200
50
100
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.5
1.0
0.2
0.4
0.6
0.8
1.0
Environment Stoes (x12")
Environment Stope (x10f)
Environment (teps
Figure 16: Evaluation returns for agents trained in DMControl with camera distractors on the medium
setting. Bold line depicts the mean and shaded area represents +- one standard deviation across five seeds.
Algorithm HKSL Learning Loop
Input: trajectory
1: for layer in HKSL (begin with the lowest layer) do
2: for layer j in HKSL (begin with the highest layer) do if i i===j then
break loop end if
Embed first observation in using layer j's encoder if layer j is the top layer then for step layer can take in - do
7:
8:
9:
Compute forward-step using layer j's forward model and actions from 7 Store the prediction end for
10:
11:
12:
else layer not the top layer for step layer can take in do
13:
14:
Compute forward-step using layer j's forward model, actions from t, and output from communication manager using the stored rollout from above level
15:
Store the prediction end for end if end for
16:
17:
18:
19:
Embed first observation in 7 using layer i's encoder
20:
for step layer i can take in r do
21:
Compute forward-step using layer i's forward model, actions from t, and output from communi- cation manager using the stored rollout from above level
22:
Project the forward model's output with layer i's nonlinear projection
23:
Compute loss per Equation end for
24:
25:
Update layer i's weights
26:
end for
23


Reviews:
[{"review_id": "sA07eJTmJK", "reviewer": "Reviewer_XeZD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe intuition driving this research is good, and the forward model approach is laid out well in a (mostly) understandable way (see questions for some clarifications). I think the method is something well worth looking into and could be of interest to the RL community at large.\n\n# Weaknesses\n\nWhile I think the paper is relatively strong, I do have some concerns which need to be addressed by the authors.\n\n1. The paper claims there is a lack of approaches in reinforcement learning using temporally extended objectives for auxiliary tasks to learn representations. This is not true. Representation learning in reinforcement learning has been obsessed with temporally extended representations and objectives for a long time. The background section related to this excludes a significant amount of work. I provide some examples below, but this list is not meant to be exhaustive:\n     - Successor features/representations: [Successor Representations](https://proceedings.neurips.cc/paper/2017/hash/350db081a661525235354dd3e19b8c05-Abstract.html)\n     - The use of general value functions for auxiliary tasks [Discovery of useful questions as auxiliary tasks\n](https://proceedings.neurips.cc/paper_files/paper/2019/hash/10ff0b5e85e5b85cc3095d431d8c08b4-Abstract.html), [Learning State Representations from Random Deep Action-conditional Predictions](https://proceedings.neurips.cc/paper/2021/hash/c71df24045cfddab4a963d3ac9bdc9a3-Abstract.html)\n     - Predictive state representations, TDNs, and GVFNs which purposefully augment the agent state to be predictions of the future: [General Value Function Networks](https://dl.acm.org/doi/abs/10.1613/jair.1.12105) \n     - Predictive state rnns, predictive state decoders, etc...: [Predictive state decoders](https://arxiv.org/abs/1709.08520)\n\nThere are many avenues representation learning has been thought of from a temporally extended perspective beyond the list above. The authors need a much better background section in this regard.\n\n2. The hyperparameter selection procedure is not sufficient in my opinion. \n     - It isn't clear where the hyperparameters in table 2 come from, other than these are the ones used. Are these searched for, did they come from another paper?\n     - \"For method-specific hyperparameters ... we defaulted to the settings provided by the original authors.\" While I understand the need to minimize the amount of compute used, if this procedure was followed then you MUST use the same hyperparameters for the other parts of those systems. I know this is not done in one instance, DrQ uses a mini-batch size of 512 vs your 128. This makes me hesitant to trust the results presented in the rest of the paper, as your method was given the ability to search for its specific hyperparameters using the set of hypers used for all the methods. This is the case with changing the architectures from other approaches as well (mentioned in Appendix D)\n\n3. It seems to me the HKSL method has quite an advantage in terms of model size due to the mixture of critics and the large hierarchical model employed. The paper doesn't seem to enumerate if the model is indeed bigger or of different size to the other architectures. It is mentioned the encoder, critic, and actor are all the same, but I'm not sure how that can be true given the form of HKSL (This is even seen in the ablation studies in figure 4 where a lot of performance comes from the ensemble of critics).\n\n# Questions\n\n1. I don't understand how the communication managers are working. I think the text needs to be clarified. For instance, how is it using the one-hot encoded timestep? I thought each timestep had a separate cm. Or how is the CM pushed to attend to the correct information? Is it just through the gradient process?\n2. How is your pixel environment different from the catcher environment? Did you just simplify it to a smaller image so you could do a better set of ablations?\n3. How is this method related to other hierarcical predictive representation schemes like Hierarchical Predictive Coding, etc...?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new set of auxiliary tasks and a new architecture for learning representations in reinforcement learning. The main intuition in this paper is that reinforcement learning objectives are temporally extended and thus representation learning schemes (i.e. auxiliary tasks in RL parlance) should be temporally extended. The hierarchical forward model is an architecture which balances accuracy with the coarseness of information the agent is expected to be able to model. They proved empirical results in a custom domain and a set of 30 robotics tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors need to address the two main weaknesses above in my opinion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1BvNaZ7gWK", "reviewer": "Reviewer_17be", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ This paper proposes a novel strategy for learning good representations using multiple forward models trained at different time scales.\n+ Experimental results are solid and appear statistically significant.\n+ Explicit tests on the representation show that it can be used to train linear projections with lower error than alternatives.\n+ Ablations show that the different pieces of the algorithm are important for its success\n\n- Some of the motivation behind the details of HKSL is lacking.\n-  I did not see mention of how the number of hierarchies and prediction frequencies were chosen other than the ablation on the falling pixels. But I may have missed this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper seeks to make pixel-level RL more sample efficient by training representations by training hierarchical forward models that predict different horizons. The proposed approach is shown to perform better than prior work and to learn better representations. Detailed experimental results over 30 tasks showcase the performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'm curious whether the learned forward models could be used for model-based RL?\n\nAs noted in the paper, the hierarchy adds significant computational and memory costs. The Falling pixels domain has a nice ablation on the depth of the hierarchy. Was a similar process performed for each other domain? Is training a different model for each hierarchy depth the only way to know what depth to use or are there potentially other ways of determining how many forward models to train and at what time scales? What considerations were made when running the results in the current draft?\n\nIt is common to train forward models to not just predict the next state but also n-states out. Would this kind of training provide any of the same benefits of HKSL?\n\nWhy compute the loss over projected representations rather than the actual representations? It seems that if the if the momentum image encoder is the exponential moving average of the online encoder that computing a loss on the difference between the representation and exponential moving average would be better than computing the difference between the moving average and a projected version of the representation? I'm assuming that the RL policy receives the representation and not the projected version.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NkyACTBRAg", "reviewer": "Reviewer_u44v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* compelling empirical results on environments considered\n* ablation of key design choices\n* thorough evaluation and analysis\n\nWeaknesses:\n* The chosen environments appear not terribly well suited to show the benefits of hierarchy and different timescales. It is promising that the method improves on baselines in this setting but other environments would lead to a more compelling paper.\n* complexity: \n    * potentially difficult tuning of hierarchy details (number of levels, temporal coarseness)\n    * computational overhead", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of reinforcement learning from pixels. It proposes a new auxiliary loss (hierarchical k-step latent) to learn representations via a hierarchy of forward models. The new loss is evaluated on a number of dm control tasks with and without distractors as well as a new task called falling pixels and performs well relative to adequate baselines. Furthermore the paper conducts some representational analysis to provide insights into the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "minor comments:\n* page 1 \"exploit the temporal axis\" I'm not sure what this means. Consider rephrasing. \n* page 2 \"altogether\" -> \"jointly\"?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7viE9reWFP", "reviewer": "Reviewer_esry", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method is sound and interesting\n- The method empirically improves data efficiency\n- The experiments are comprehensive \n\nWeaknesses\n- The prior work overview obscures close connections to prior work like BYOL and SPR, which are very similar to the proposed method. \n- The exposition could be quite a bit clearer. Figure 1 clarity is especially important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a hierarchical representation learning method based on self-predictive representations (Schwarzer'21). Specifically, several forward models in the representation space are learned that predict at different time scales. The method improves data efficiency on atari, dm control, and the \"falling pixels\" environment which evaluates the perception of small changes in the observation. There are extensive ablations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\\-", "broader_impact_concerns": "\\-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: See my comments above.
Decision: Accept (poster)