Dynamics-Aware Embeddings
Authors: William Whitney
Rajat Agarwal
Kyunghyun Cho
Abhinav Gupta
Venue:  (2020)
Keywords: representation learning
reinforcement learning
rl

Abstract: In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and actions. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.

Published as a conference paper at ICLR 2020
DYNAMICS-AWANE EMBEDDINGS
William E. Whitney', Rajat Agarwal', Kyunghyun Cho 13 and Abhinav Gupta 1 'Department of Computer Science, New York University *oobotics Institute, Carnegie Mellon University 3 Facebook Al Research
wwhi Achitneyccs nyu. edu
ABSTRACT
In this paper we consider sulf-super representation learning to improve sam- ple efficiency in reinforcement learning (RL). We propose forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure the environment's 's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations only 1-2 million environment steps.
INTRODUCTION
In recent years, there has been lot of excitement around end-to-end model-free reinforcement learning for control, both in simulation (Lillicrap et al. 2015 Andrychowicz et al. 2018 Haarnoja al. 20188; Fujimoto et al., (2018) and on real hardware (Kalashnikov et al.) [2018: Haarnoja et al. 2018c). In this paradigm, we simultaneously learn intermediate representations and policies by maximizing rewards provided by environment. End-to-end learning has one indisputable advantage: since every component of the system is optimized for the end objective, there are no sub-optimal modules that limit best-case performance by losing task-relevant information.
Learning only from the target task is however double-edged sword. When the end objective provides only weak signal for learning, a policy with a poor representation may require many samples to learn better one. By contrast, policy with good representation may be able to rapidly fit simple function of that representation even with weak signal.
Consider the environment shown in Figure [ and two represen- tations of its state: coordinates and pixels. As a function of the agent's 's coordinate, the value function is simple and smooth. The coordinate representation has structure which is useful for learning about the task; namely, points which are close in T,2 distance have similar values. By contrast, pixel representation of the agent's 's state (below, blue) is practically one-hot vector. Two states whose x coordinates differ by one unit have pixels exactly as different as states which differ by 100 units. This illustrates the importance of good representations and the potential of representation learning to aid RL.
V(x)
ID environment. The agent (blue dot) can move continuously left and right to reach the goal (gold star).
Figure 1:
We propose a self-supervised objective for learning embeddings of states and action sequences such that a pair states or action sequences will be close together if they have similar outcomes. This objective simultaneously trains a smooth embedding space for states and temporally abstract action space for control which is task- independent and generalizes across goals and objects.
We demonstrate the effectiveness of our representation learning objective by training the twin delayed deep deterministic policy gradient algorithm (TD3) (Fujimoto et al. (018)) with learned action and state spaces. With learned representation of temporally abstract actions, method

---

Published as a conference paper at ICLR 2020
Z
$
Nin
2
8k-
Figure Computational architecture for training the DynE encoders 6a and cs. The encoders are trained to minimize the information content of the learned embeddings while still allowing the predictor f to make accurate predictions.
exhibits improved sample efficiency compared to state-of-the-ar RL methods on control tasks, with larger gains on more complex environments. When additionally combined with our learned state representation, our method allows TD3 to scale to pixel observations. We demonstrate good performance on simple family of goal-conditioned 2D control tasks within few million environment steps without adjusting any TD3 hyperparameters. This stands in contrast to end-to-end model-free RL from pixels, which requires extensive tuning (Lillicrap et al.) 2015) and on the order of 100 million environment steps" (Barth-Maron et al. 2018).
DYNAMICS-WWARE EMBEDDINGS
2.1 NOTATION
We consider the framework reinforcement learning in Markov decision processes (MDPs) We denote the state of an environment (e.g- joint angles of a robot pixels) by E S, and we assume that the states given by the environment satisfy the Markov property. We refer to a sequence of actions {a1,... ..., [2} E A using the shorthand a². We use s' ~ T(s, a) to refer to the environment's (stochastic) transition function, and overload to accept sequences of actions: 5148 ~ T(s,, at).
2.2 MODEL AND LEARNING OBJECTIVE
We propose that a good representation for reinforcement learning should represent states or actions close together if they have similar outcomes (resulting trajectories). This allows the agent generalize from small number of samples since each sample accurately reflects the value of all the states or actions in its neighborhood. In Markov decision process the outcome of taking an action in state is summarized by the distribution of resulting states p(s'| |s, a) T(s, a). Therefore we construct method which embeds states and actions such that nearby embeddings have similar distributions of next states.
Our method, which we call Dynamics-aware Embedding (DynE), learns encoders e, and 6, which embed a state and action sequence into latent spaces 2, E Z, and 2a E Z, respectively. These encodings are optimized to form a maximally compressed representation of the sufficient statisties of p(s')s fslls, a²) such that p(s"|s, a so p(s'lzz, 2a). We approximate this by maximizing the following objective:
L(d., 0)
E a mpp
p(s 1/,,,, 0)
predict s'
(1)
+ 3DK.. (es Φx) (0)
compress
(2)
+ DDLL 0.) 1| (O.
compress a
Number of steps required train D4PG taken from Hafner not include this information.
2018
Barth-Maron
2018 does
In the interest of space omit the usual recap of Markov decision processes and reinforcement learning. We refer the reader Section Silver (2014) for notation and background MDPs.
2

---

Published as a conference paper at ICLR 2020
where 2, re(ss), 2a ~ ea(a"), and p" is the distribution of transitions under behavior policy и.
The DynE objective is similar to 3- V-V (Higgins et al. 2017a) for s' but with a different variational family; like a 3-VAE, it forms variational lower bound on p(s') when 3
= 1. Where variational autoencoder (Kingma & Welling 2013 Rezende et al. 2014) or 3- E-VAE chooses the variational family to be Q {=(=)')),
a factored latent space { Za and independent
posterior approximations given the previous state and the action: Q = ( (8), ((2 ((,,)("))) ()))). This
factorization yields separate encoders for states and actions where the state encoder's output is valid
for any action and vice versa.
The DynE objective
also be interpreted in the information bottleneck (IB) framework (Tishyy et al., 2000). In the IB framework term ( is the prediction objective and terms and (3 regularize the latent representation to remove all extraneous information. Our construction nearly identical to the approximate information bottleneck proposed by Alemi et al. (2016), with the main difference being the factorization of the representation into separate state and action components.
In our experiments we use an isotropic Normal distribution for p(s'|zs, Za; 8) such that term reduces to [2f Fai 0) s' s111 where computes the mean. We use -cogoral-covvriance Normal distributions for e, and ea such that {Ms, o?} (s). {Na: ca (a*), ), 2, ~ NNs,, and Za ~ N [Naa o.). The behavior policy we use for data collection is π = Uni.
3
USING LEARNED EMBEDDINGS FOR REINFORCEMENT LEARNING
3.1 DECODING TO RAW ACTIONS
In order to be useful for RL, the abstract action space produced by the encoder must be decodcable to raw actions in the environment. Since the mapping from action sequences to high-level actions is many-to-one, inverting it is nontrivial. We simplify this ill-posed problem by defining an objective with single optimum.
Once the action encoder ea is fully trained, we hold it fixed and train an action decoder d, to minimize
((d.) = (0.1)
[lea(da(za)) - zal12 + AlIda (z,)
(4)
The first term of this objective ensures that the action decoder d is a one-sided inverse of ea; that is, ea (za)) 2a but (ca ax)) # ai,..., dg. The second term of the loss ensures that d, is in particular the minimum-norm one-sided inverse of 6a and gives the objective for the output of d, a single minimum. Out of all the action sequences which have the same outcome, the minimum-norm sequence is desireable as it leads to trajectories which are smooth and consume less energy. We choose be small (e.g. 10 10-2) ensure that the reconstruction criterion dominates the optimization.
3.2 EFFICIENT RL WITH TEMPORAL ABSTRACTION
Once equipped with a decoder which maps from high-level actions to sequences of actions, we train a high-level policy that solves task by selecting high-level actions. In this section we extend the deterministic policy gradient (Silver et al., 2014) family of algorithms work with temporally- extended actions while maintaining off off-policy updates and learning from every environment step. This allows our method achieve superior sample efficiency when working with high-level actions. In particular, we extend the twin delayed deep deterministic policy gradient (TD3) algorithm (Fujimoto al. 2018) to work with the DynE representation of actions to form an algorithm we call DynH-TDD3.
We first describe why DPG requires modifications to accommodate remporally-abstractee One simple approach to combining DynE with DPG would be to incorporate the k-step DynE action space into the environment to form a new MDP. This MDP allows the use of DPG without modification; however, it only emits observations once every timesteps. As a result, after N steps in the original environment, the deterministic policy μ and critic function Q can only be trained on N/k observations. This has a substantial impact on sample efficiency when measured in the original environment.
actions.
3

---

Published as a conference paper at ICLR 2020
Instead require algorithm which perform updates to the policy μ and critic for every environment step. To do this, we train both μز and Q in the abstract action space with minor changes to their updates. We distinguish these functions which use DynE actions from their raw equivalents by adding a superscript DynE, i.e. y yy and QDynE, We augment the critic function with an additional input, which represents the number of steps i < le of the current embedded action that have already been executed. This forms the DynE- BDB critic:
Amiml1 1
Quynt (estt)
Σ
(Pre4s) + +
Pynt (esteeee (co(BetR-s), (estee (es(+)4x-)))) (5)
In plain language, the value of being on step of abstract action e is the value of finishing the remaining (A - steps of 2 and then continuing on following the policy, This is similar to the idea of k-step returns (Sutton & Barto 2018), but with a variable which depends on the step within the current plan. Whereas k-step returns would typically require off off-policy correction such as Retrace (Munos et al., 2016), conditioning on 2, and i determines all k actions in the return. In effect, they remain a single action, making the update valid off policy. The DynE critic is trained by minimizing the Bellman error implied by eq. (5).
To update the policy we follow the standard DPG technique of using the gradient of the critic. We modify the algorithm to take into account that i = O at the time of issuing new high-level action. The gradient of the return with respect to the policy parameters then
V (Pg
DynE
E Snup
Vepe
DynE, (e)(s)
V V. DynE (e, 0
(6)
given that data was collected according to a behavior policy
4 RELATED WORK
Successor representations, an inspiration for this work, represent a state by the expected rate of future visits to other states (Dayan [993]] Kulkarni et al. [016b] Barreto et al., (2017). Successor representations have been demonstrated to be an effective model of animal and human learning (Momennejad et al., 2017 Stachenfeld al. 2017). They are also one of the earliest realizations of the idea of representing each state by its future. Whereas successor representations learn future occupancy maps for a particular policy, we learn an embedding space where states are close together if they have similar outcomes for any policy.
Several papers have proposed using (variational) auto-encoders to learn embeddings for observations (Lange & Riedmiller, 2010; Van Hoof et al. 2016 Higgins al.) 2017b; Caselles- Dupré al.) 2018); unlike our work, these models operate on a single observation at a time and do not depend on the environment dynamics. Forward prediction has also been used as auxiliary task to speed RL training (Jaderberg et al.) (2066), and Jonschkowski et al.) (2017) learn representations which adhere physical constraints. Ghosh al. (2018) propose learn state embeddings using the action distribution of a goal-conditioned policy; however, their technique depends on already having successful policy. Other work has proposed use mutual information maximization learn embeddings which facilitate exploration via intrinsic motivation (Kim et al. 2018).
Similarly to this work, hicrarchical reinforcement learning seeks learn temporal abstractions. These abstractions are variously defined as skills (Florensa et al.) 2017: Hausman et al. 2018). options (Sutton et al.. 1999. Bacon al., 2017), or goal-directed sub-policies (Kulkarni et al. 2016
Vezhnevets et al. (017). Most closely related are SeCTAR (Co-Reyes et al., (018) and HIRO (Nachum et al. 2018). SeCTAR simultaneously learns a generative model of future states and low-level policy which can reach those states. HIRO learns representation of goals such that high-level policy can induce any action in a low-level policy. Unlike this work, both SeCTAR and HIRO learn state-dependent low-level policies, not action representations. Furthermore SeCTAR assumes the reward function is given ahead of time, and HIRO's s off off- ooficooliy performance depends on an approximate re-labeling of action sequences to train the high-level policy.
4

---

Published as a conference paper at ICLR 2020
Also related methods which attempt learn embeddings of single actions to enable efficient learning in very large action spaces (Dulac-Amold et al. Chandak al. (2019) learns a latent space of actions based the effects of an action the environment. However, their latent spaces are for a single action and they do not consider learned state representations. Another related direction is learning embeddings of one more actions from demonstrations (Tennenholtz & Mannor] 2019);; this embedded action space builds in prior knowledge from the demonstrator and can allow faster learning.
et al. 2019). In particular,
REPRESENTATION EXPERIMENTS
In this section we empirically investigate how the learned DynE representations reshape the problem of reinforcement learning. First we make a connection between temporal abstraction and exploration, revealing that DynE actions result in better state coverage. Then we probe the relationship between DynE state embeddings and the task value function.
5.1 TEMPORAL ABSTRACTION AND EXPLORATION
When embedding an action sequence, the DynE objective seeks to preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence. As shown in Appendix [ this leads representation where all action sequences which have similar outcomes embed close together. We propose that this temporally abstract action space, where actions correspond to multi-step outcomes, allows random actions to explore the environment more efficiently.
We empirically validate the exploration benefits of the temporally abstract DynE actions. Figure shows that uniformly sampling DynE action results in nearly uniform distribution over the states reachable within steps. Over the course of an entire episode, selecting DynE actions uniformly at random reaches faraway states more often than random exploration with raw actions. Appendix shows the qualitative difference between random trajectories the raw and DynE action spaces, and Appendix C studies the impact of varying & on the performance of learned policy.
0.022
DynE
0.070
Raw
Fraction of trajetories
0.078
0.055
11
Ratio
0.016
0 0.03
0.014
0 035
0,012
0.090
0.018
0.000
Distance
Figure 3: The distribution of state distances reached by uniform random exploration using DynE actions (k 4) raw actions in Reacher Vertical. Left: Randomly selecting a 4-step DynE action reaches a state uniformly sampled from those reachable in environment timesteps. Right: Over the length of an episode (100 steps), random exploration with DynE actions reaches faraway states very much more often than exploration with raw actions. The visit ratio shows how frequently DynE exploration reaches certain distance compared to raw exploration.
5.2 STATE REPRESENTATIONS
The DynE objective compresses states while preserving information about the outcome of taking any action in that state. If this compression is successful, states which have similar outcomes will be close together in embedding space. In an MDP, two states which have identical successor states have values which differ by most the range of the reward function Fmaz Tmán- While in general states which lead to merely similar successors may have arbitrarily different value, we suggest that in many tasks of interest, similar successors may entail similar value.
5

---

Published as a conference paper at ICLR 2020
We investigate whether the DynE state embedding leads to neighborhoods with similar value in the Reacher Vertical environment. We collect 10K states from random policy in the environment and perform dimensionality reduction three representations of those states: the DynE embedding of state images, low-dimensional joint states, and pixels. Figure the results of this dimensionality reduction, in which every point is colored by its value under fully-trained TD3 policy the low-d states. DynE embeddings have neighborhoods with more similar values than states or pixels.
Value
60
10
40
15
20
-20
20
-20
-40
60
-50
60
-100
60 100 -120
-100
(a) DynE (pixels)
(b) States
(c) Pixels
Figure 4: The relationship between state representations and task value. Each plot shows the t-SNE dimensionality reduction of a state representation, where each point colored by value under a near-optimal policy. (a) The DynE embedding from pixels places states with similar values close together. (b) The low-dimensional states, which consist of joint angles, relative positions, and velocities, have some neighborhoods of similar value, but also many regions of mixed value. (c) The relationship between the pixel representation and the task value is very complex.
6 REINFORCEMENT LEARNING EXPERIMENTS
In this section we assess the effectiveness of DynE representations for deep RL, individually analyzing the contributions of the action and state representations before combining them. First we evaluate the DynE action space on a set of six tasks with low-dimensional state observations, testing its usefulness across set of tasks and object interactions. Then, we test the DynE state space set of three tasks with pixel observations. Finally, we combine DynE actions with DynE observations, verifying that the two learned representations
complementary.
Appendix B provides a full description of hyperparameters and model architectures, and all of the code for DynE available on GitHub at ltttp::///ithtm com/ dyne submi snion/ dynamics is aware-wmbedee eew.rddggsddddng
Environments We use six continuous control tasks from two families implemented in the MuloCo simulator (Todorov et al. 2012) to evaluate our method. Within each family, the task and observation space change but the robot being controlled stays roughly the same, allowing to test the transferra- bility of the DynE action space between tasks. The Reacher family consists of three of tasks which involve controlling 2D, 2DoF arm interact with various objects. The 7DoF family of tasks from OpenAl Gym (Brockman et al. 2016) is quite difficult, featuring three tasks in which a 3D, 7DoF arm must use different end effectors to push or throw various objects to randomly-generann positions. Images and detailed descriptions of both families of tasks are available in Appendix A
goal
6.1 NOW-DIMENSIONA STATES
For training the DynE action representation we use 100K steps with a uniformly random behavior policy in the simplest environment in each family with no reward other supervisory signal. As this DynE pretraining is unsupervised and only occurs once for each family of environments, the x axis on these training curves refers only to the samples used to train the policy action representation to all three environments the family. When training DynE-TD3 we use all of the default hyperparameters from the TD3 implementation across all environments.
F We then transfer this
We directly test the impact switching from raw to DynE actions by comparing TD3 to DynE-TD3. For completeness we compare with two additional state-of-the-art model-free methods: soft actor-
*On all environments except the simplest (Reacher Vertical) shifting the DynF-TD3 plot by 100K steps does affect the ordering of the results.

---

Published as a conference paper at ICLR 2020
Source tasks
Target tasks
Reacher Vertical
Reacher Push
49
19
199
113
180
133
116
-20
130
-40
-10
Thms stops
Thmee strput
Tire seess
Pusher
Striker
DoF Thrower
-90
300
409
100
499
-00
490
140
191
000000
900539
1390000
2000000
Time steps
Time
DynE-TD3
SAC
SAC-LSP
Figure 5: Performance of DynE-TDS and baselines two families of environments with low- dimensional observations. Dark lines are mean reward over seeds and shaded areas are bootstrapped 95% confidence intervals. Across all the environments, TD3 learns faster with the DynE action space than with the raw actions. Within each family of environments, the DynE action space was trained only the simplest task (left).
critic (SAC) (Haarnjja al. 2018b C) and proximal policy optimization (PPO) (Schulman al. 2017). We also compare with solt actor-critic with latent space policies (SAC-LSP) (Haarnoja et al. 2018a). an innovative hierarchical method which transforms low-level action space into an abstract one by training an invertible low-level policy. In all cases we use the official implementations 1ll and the MuJoCo hyperparameters used by authors. We also attempted to compare with the hierarchical method by Nachum et al. (2018), but after several emails with the authors and dozens of experiments we were unable to get it to converge on tasks other than those their paper.
Results Figure 5 shows the results of these experiments. Most significantly, they show that switching from the raw action space (TD3 curve) to the DynE action space results in faster training and allows TD3 solve the difficult 7DoF suite of tasks. We see that the DynE action space generalizes across several tasks with the same robot, even when interacting with objects unseen during training. is especially worth noting that the gains from DynE increase as the tasks become harder, maintaining convergence, stability, and low variance in the face of high-dimensional control with difficult exploration. Since SAC-LSP (Haarnoja al.) 2a) performs similarly but worse than SAC we test it only the simpler Reacher family of tasks; meanwhile, the PPO curves do not enter the frame on the Reacher family of tasks due to its poor sample efficiency.
6.2 PIXELS
Using the Reacher family of environments we evaluate several state representations by their effective- ness for policy learning with TD3.
We evaluate two established methods for learning representations from single images. "DARLA" is the Disentangled Representation Learning Agent proposed by Higgins et al. (2017b) with the denoising autoencoder loss, which referred to that work as 3-VAEDAE "VAF" is standard variational autoencoder (Kingma & Welling 2013 Rezende et al. 2014), which has previously been
SSAC and SAC-LSP: /ACCIS//itcee///tttaboooooooooba

---

Published as a conference paper at ICLR 2020
Reacher Vertical
Reacher Turn
Reacher Push
Mae TD3
Mec 103
14X T09
Neword
420
244
Time shipo
Tima staps
Time stops
SA DyrE
DyynE
Oaterministic
Datermiiatic
DARLA
VAE
Pixel
Figure 6: Performance of TD3 trained with various representations. Learned representations for state which incorporate the dynamics make dramatic difference. SA-DynE converges stably and rapidly and achieves performance from pixels that nearly equals TD3's performance from states. Dark lines are mean reward over 8 seeds and shaded areas are bootstrapped 95% confidence intervals.
found to learn effective representations for control Van Hoof et al. 2016); it is equivalent to DARLA with the pixel-space loss and 1. Since these representations operate single frame at time, we apply them to the most recent four frames independently and then concatenate the embeddings before feeding them the policy. These representations have compressed latent spaces, but they encode no knowledge of the environment's dynamics, allowing us to evaluate the importance of incorporating the dynamics into our embeddings.
Next we evaluate representation learning methods whose objectives incorporate the dynamics. "S- DynE," for State DynE, is the DynE state embedding cs. and "SA-DynE" combines the DynE state and action representations. "S-Deterministic" and "SA "SA-Dettrministic" are ablations of the corresponding DynE methods which have the same forward-prediction objective but no KI. or noise on the latent representations. Comparing the DynE methods to their respective ablations reveals the contribution of explicitly introducing a compression objective to the latent space.
For training all of the learned representations use a dataset of 100K steps in each environment from a uniformly random policy. In every case we train TD3 with the learned representations using all of the default hyperparameters from the official TD3 implementation.
We compare these representation lcarning methods with TD3 trained from pixels. As there are no experiments pixels in the TD3 paper, we performed extensive search over network architectures and byperparameters. We included our search the configurations used in the pixel experiments of DDPG (Lillicrap et al. 2015) well those used in successful discrete-action RL works from pixels (Schulman et al. 2017 Kostrikov] 2018: Espeholt et al. 2018
Results Figure shows the results of these experiments. We find that the single-image methods are unable to solve any of the three tasks from pixels; TD3 from pixels diverges in all cases, while VAE and DARLA learn gradually best. simply reducing the dimension of the states were sufficient to enable effective policy training, would expect good performance from these methods. S- S-DynE and S- S-Deterministic, which incorporate the dynamics into their representation learning objectives, perform far better. The minimality imposed by the DynE objective allows S-DynE and SA-Dyne to outperform their deterministic ablations. SA-DynE learns rapidly and reliably, finding behaviors which qualitatively solve all three tasks. The improvement of SA-DynE over S-DynE shows that the state and action representations are complementary.
7 DISCUSSION
In this work we proposed a method, Dynamics-aware Embedding (DynE), that jointly learns em- bedded representations of states and actions for reinforcement learning. Our experiments reveal that DynE action embeddings lead to more efficient exploration, resulting in more sample efficient learning on complex tasks, while DynE state embeddings allow unmodified model-free RL algorithms to scale to pixel observations. When combined, the DynE state and action embeddings result in stable, sample-efficiee learning of high-quality policies from pixels.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 1612.0041,0 2016.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. arXiv preprint arXiv: 1008.000871777 2018.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
André Barreto, Will Dabney, Rémi Munos. Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer reinforcement learning. In Advances in neural information processing systems, pp. 4055 4065-4065, 2017.
Gabriel Barth-Maron, Matthew Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint Xrv: /04.0600..0 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01. 2016.
Hugo Caselles-Dupré, Michael Garcia-Ortiz, and David Filliat. Continual state representation learning for reinforcement learning using generative replay. arXiv preprint arXiv: /8 03,880 2018.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S Thomas. Learning action representations for reinforcement learning. arXiv preprint /902:00990.0000 2019.
John D. Co-Reyes, Yuxuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hicrarchical reinforcement learning with trajectory embeddings. In ICML, 2018.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-624, 1993.
Gabriel Dulac-Amold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theoophae Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07669 2015.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint /802.0 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. arXiv preprint arXiv: NXX::1704.03012 2017.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation enror in actor-critic methods. arXiv preprint arXiv: 1802.0947, 2018.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint arXiv:1811.07817 2018
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint arXiv:1804.00 02808. 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor--ritic. Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv: 1801v 011290 2018b.
Tuomas Haarojj,, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.059890 2018c.

---

Published as a conference paper at ICLR 2020
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:181 1181..11511111111111111111111111 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference Learning Reprexentations, 2018
Irina Higgins. Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2b.7b.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 2016.
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin A. Riedmiller. Pves: Position- velocity encoders for unsupervised learning of structured state representations. ArXiv, abs/1705.09805, 2017.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information maximizing state and action embeddings. arXiv preprint arXiv:1810.01176, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://githu: 2018.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016a.
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.00 02396, 2016b.
Sascha Lange and Martin A Riedmiller. Deep learning of visual control policies. In ESANN, 2010.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022
2015.
Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin, al. Cyclical annealing schedule: A simple approach to mitigating kl vanishing, arXiv preprint 1eXXiv:903.10111. 2019.
Ida Momennejad, Evan M Russek, Jin H Cheong, Matthew M Botvinick, Nathaniel Douglass Daw, and Samuel J Gershman. The successor representation in human reinforcement learning. Nature Human Behaviour, 199):680, 2017.
10

---

Published as a conference paper at ICLR 2020
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054-1062, 2016.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. CoRR, abs/1810.01257, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint urXiv:/51.0044: 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1401::/401.4882 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel Gershman. The hippocampus as predictive map. Nature neuroscience, 20(11):1643, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup. and Satinder Singh. Between mdps and mdmi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, (12(12))181-2)) 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Guy Tennenholtz and Shie Mannor. The natural language of actions. arXiv preprint 19xvv1190..01119. 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004005,.00 2000.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, and Jan Peters. Stable rein- forcement learning with autoencoders for tactile and visual data. In 2016 IEEE/SS International Conference on Intelligent Robots and Systems (IROS), pp. 3928-3934. IEEE, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3540-3549. JMLR. org, 2017

---

Published as a conference paper at ICLR 2020
APPENDIX A ENVIRONMENT DESCRIPTION
(a) Reachervertical
ReacherTurn
(c) Reache ush
Figure 7: The Reacher family of environments. ReacherVert cal requires the agent to move the tip of the to the red dot. ReacherTure requires the agent to turn rotating spinner (dark red) so that the tip of the spinner (gray) is close to the target point (red). ReacherPush requires the agent push the brown box onto the red target point. The initial state of the simulator and the target point are randomized for each episode. In each environment the rewards are dense and there is a penalty on the norm of the actions. The robot's 's kinematics are the same in each environment but the state spaces are different.
The first task family, pictured in Figure 7 is the "Reacher family", based on the Reacher-v2 MuJoCo (Toooovv al. 2012) task from OpenAl Gym (Brockman et al. 2016). These tasks form simple new benchmark for multitask robot learning. The first task, which we use as the "source" task for training the DynE space, is ReacherVertical. a standard reach to a location task. The other two tasks are inspired by the DeepMind Control Suite's Finger Turn and stacker environments, respectively (Tassa al., (2018). In Rea cherrrrr,,, the same 2-link Reacher robot must turn spinner to the specified random location. In ReacherPush, the Reacher must push block to the correct random location.
(a) Pusher-v2
(b) Striker-v2
(c) Thrower-v2
C-shaped end effector to push puck across the table onto a red circle. Striker-v2 requires the agent to use flat end effector hit ball that it rolls across the table and reaches the goal. Thrower-v2 requires the agent to throw a ball to a target using small scoop. As with the Reacher family, the dynamics of the robot are the same within the 7DoF family of tasks. However, the morphology of the robot, as well as the object interacts with, is different.
Figure 8: The 7DoF family of environments. Pusher-v2 requires the agent
The second task family is the "7DoF family", which comprises Pusher-v2, Striker-v2, and Thrower-v2 from OpenAl Gym (Brockman et al.) (016)). We use Pusher-v2 as the source task. These tasks use similar (though not identical) robot models, making them a feasible family of tasks for transfer. They are shown in Figure
12

---

Published as a conference paper at ICLR 2020
A.1 PIXELS
We use full-color images rendered at 256x256 and resized to 64x64 pixels. In order to allow the agents perceive motion, we stack the current frame with the three most recent frames, resulting in an observation of dimension 12x64x64.
APPENDIX B HYPERPARAMETERS AND DYNE TRAINING
For DynE-TD3 we use of the default hyperparameters from the TD3 code across all tasks. For all experiments we choose the dimension of the DynE action space be equal to the dimension of single action in the environment. We set the number of actions in the DynE space to be k = 4 for all experiments except Thrower-v2, for which we use k 8. We use the Adam optimizer (Kingma Ba 2014) with learning rate 10-4. All our experiments used recent-model NVidia GPUs.
Training on states When computing licelioos we divide by the number of dimensions in the state in attempt to make the correct settings of y invariant to the observation dimension; the same result could be achieved by multiplying the values that we report by the state dimension and changing the learning rate. With that scaling across all environments. We concatenate all the joint angles and velocities to use as the states during representation learning. We preprocess the s, s' pairs by first taking the difference As s' and then whitening so that As has zero mean and unit variance in each dimension. This preprocessing encourages the encoder represent both position and velocity in the latent space; the scales of these two components are quite different.
hyperparameters y À 10-2
We use fully-connected networks for the action encoder e, and the conditional state predictor f. Each function has two hidden layers of 400 units. Training this model should take 5-10 minutes on GPU.
Training on pixels We train a DynE model for each environment, taking in a stack of frames and sequence of k = 4 actions and predicting future states. To speed training latest frames of the future state (i.e. the picture of the world at time t + k and t + - 1) instead of all four. When doing RL take the state encoder e, from this model and use it to preprocess all states from the environment.
predict only the two
We set the dimension of the state embedding 2, to 100. We did not try other options, and given the sensitivity of RL to state dimension a smaller setting would very likely yield faster learning. We set
7 = which setting DynE is optimizing variational lower bound on p(5t+k 8t, a²). We recommend ensuring that the predictions (not generations) from the model are correctly rendering all the task-relevant objects; if 3 and y are too high, the model may incur lower loss by ignoring details in the image. We use cyclic KL annealing (Liu et al. 2019) to improve convergence over wide range of settings.
We use the DCGAN architecture (Radford et al. 2015 for the image encoder and the predictor f. The action encoder e,, fully connected with two hidden layers 400 units. Training this model takes 1-2 hours GPU.
13

---

Published as a conference paper at ICLR 2020
APPENDIX C VARYING LEVELS OF TEMPORAL ABSTRACTION
We study the impact varying k; the level of temporal abstraction the DynE action space. We find that increasing & improves performance and learning speed up point; beyond this point, performance degrades. The optimal setting of will depend on the environment dynamics. We expect that environments with very slow dynamics will benefit from greater degree of temporal abstraction.
-80
-100
Reward
-110
-120
-130
-140
-150
Figure 9: DynE-TD3 results on Reacher Push with varying k. We find that increased temporal abstraction improves performance up point, beyond which the action space no longer able to represent the optimal policy and performace degrades. Solid points are the mean reward obtained after training for 1M environment steps. Shaded bars represent the min and max performance over seeds.
APPENDIX VISUALIZING THE DYNE ACTION SPACE
To better understand the structure in the DynE action embedding space, we visualize the relationship between the outcome of a sequence of actions and the DynE embedding of those actions. When embedding an action sequence, the DynE objective seeks preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence. Therefore we should see that all action sequences which have similar outcomes embed close together, regardless of the actions along the way. Figure 10 investigates this in simple Point environment with an casy-to-visune visualize 2D (z, y) state. For this simple problem, we see that all pairs of action sequences a and a with similar outcomes correspondence between the two spaces appears to remain strong for high fing--iimnsiooal and nonlinear environments, but is much harder render in two dimensions.
close together in the embedding space. The
14

---

Published as a conference paper at ICLR 2020
(a) Outcome space
(b) DynE action space
The mapping between the outcomes and embeddings of action sequences. We sample 10K random sequences of four actions and evaluate their outcomes in the environment dynamics, measured by (Ar, Ay) 8t-4 St- (a) We plot the outcome (Ax, Ay) of each action sequence and color each point according to its location in the plot. (b) We use DynE to embed each action sequence into two dimensions; each point this plot corresponds to point in (a) and takes its color from that corresponding point. The similarity of the two plots and the smooth color gradient in (b)
Figure 10:
indicate that DynE embedding action sequences according their outcomes.
APPENDIX E EXTENDED RESULTS
ReacherVertical
ReacherTurn
ReacherPush
Method
5 DynE
-323
5A.DmFF-T03
VAE TD3
SAC
330
232
192
-30
335
430
452
1,00.000 2.400.000
1.000000
1000,000
Figure 11: These plots allow for direct comparison between the methods from pixels (Pixel-TD3, VAE-TD3. S-DynE-TDD -D33, and SA-DynE-TD3) and our baselines from cow-dimensional states (PPO and SAC). The DynE methods from pixels perform competitively with some baselines from states.
15

---

Published as a conference paper at ICLR 2020
APPENDIX F EXPLORATION WITH RAW AND DYNE ACTION SPACES
Random exploration with actions
(b) Random exploration with DynE
Figure 12: These figures illustrate the way the DynE action space enables more efficient exploration. Each figure generated by running uniform random policy for ten episodes on PointMase environment. Since the environment has only position dimensions, position of the mass over the course of each episode. Left: A policy which selects actions at each environment timestep uniformly random explores very small region of the state space. Right: A policy which randomly selects DynE actions once every k timesteps explores much more widely.
can plot the actual 2D
16


Reviews:
[{"review_id": "r1lVq9f-5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to learning state and action representations through self-supervision, such that these representation can be used for downstream reinforcement learning. In particular, the proposed approach learns a  time-dilated dynamics model on data collected via self-supervision, where given s_t, and actions (a_t, ..., a_{t+K}) predicts s_{t+K}. The input state and action trajectory and each encoded into latent distributions, which are then used to reconstruct the future state. Then, they demonstrate that using TD3 with the latent action space outperforms existing model-free methods and existing state representation techniques.\n\nOverall the paper is well motivated and clearly written. The key contribution seems to be in learning the latent distribution over multi-step action trajectories, which seems to be important for performance. Lastly the experiments and ablations are thorough and well explained.\n\nMy main comments have to do with (1) the fairness of the comparison to existing model-free RL, (2) an analysis of the temporal abstraction for learning the action distribution.\n\n(1): The proposed method first pre trains the latent dynamics model on 100K steps of random data, then trains the proposed TD3 using this action distribution (and the modified critic to support 1-step Q values). While this does outperform the model-free RL methods trained from scratch, it is also using 100K steps worth of experience that the others don't have access to, which makes it not quite a fair comparison. If you pretrain the critic of TD3 or SAC with the 100K samples, do you still observe the same performance gains?\n\n(2): From the ablation study and comparison to other state representation learning techniques in Figure 6, it seems like the most important aspect of the proposed method is using the latent action distribution. This makes sense as it captures longer action sequences, and thus likely is the reason for better exploration and performance. As a result the exact choice of K seems very important. In the Appendix it states that for the Thrower task K=8, and elsewhere K=4. Do the authors have a sense for how performance changes with choice of K? I think a plot which compares performance over different choices of K would be very valuable.\n\nSome smaller comments:\n- The comparison to other model-free RL methods is done only on low dimensional states, while the ablations are done on pixels. Is this because the model-free comparisons did not work at all on pixels?\n- Is it possible to perform model predictive control with the learned model, and how does it compare to existing latent model based RL methods (Hafner et al.)\n- One more recent work that may be worth comparing to is SLAC (Lee et al.) which also learns a stochastic latent dynamics model, and learns a policy in the latent space of the model. The latent space is of states however, and not actions. \n\n______________________\n\nAlex X. Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine. Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model\n\nDanijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson. Learning Latent Dynamics for Planning from Pixels\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkeQW6MAKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents DynE, a self-supervised approach for learning dynamics-aware state and action representations. DynE learns an encoding of individual states and action sequences, assigning nearby embeddings to those that have similar outcomes. This is achieved via a reconstruction objective that predicts the outcome of a sequence of \"k\" actions from a given state, along with losses that encourage compression in the learned latent representations. Additionally, a learned decoder allows for reconstruction of minimum-norm action sequences from the high-level latent action embedding. Combining DynE state and action embeddings with an actor-critic agent operating directly in the learned high-level action space leads to significant speedups in both from-scratch and transfer learning (on 2D and 3D OpenAI Gym tasks), leading to better data efficiency compared to model-free baselines. Additionally, the learned action and state embeddings lend themselves to better exploration and consistent value prediction, respectively.\n\nThe paper is very well written and the approach looks quite promising. A few comments:\n1. The approach is well validated but additional ablation results can help quantify the effect of different components. For example, it would be useful to see the effect of varying \"k\", the number of actions to be encoded for generating the action embedding.  \n2. A related paper that learns state representations that are physically consistent and dynamics-aware is this work:\nJonschkowski, Rico, et al. \"Pves: Position-velocity encoders for unsupervised learning of structured state representations.\" arXiv preprint arXiv:1705.09805 (2017).\nHere the state representation is learned to implicitly encode physical consistency via self-supervised losses that mimic constraints such as controlability, inertia, conservation of mass etc. Combining such additional self-supervised losses can help structure the state embedding learning further, albeit at the cost of introducing additional hyperparameters during optimization.\n3. It would be useful to know what the actions are (and their dimensions) for the tasks considered in the paper.\n4. The paper would benefit from a short discussion on the limitations of the proposed approach and potential to scale to more complicated tasks.\n5. Fig. 5, bottom right: It is not clear why PPO (blue) performs significantly better on this task compared to the other 7DoF tasks considering that the thrower should be more complex than the pusher and striker. PPO also seems to match the data efficiency of DynE-TD3. Is this correct? \n\nOverall, I find the approach quite interesting and promising. I would suggest an accept.\n\nTypos:\n1. Intro, 2nd para, 2nd line, many samples to learn than a better one\n2. Fig. 1, the pixel representation is very unintuitive", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxtfU92qB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes training dynamics-aware embeddings of the state and k-action sequences to aid the sample efficiency of reinforcement learning algorithms. The authors propose learning a low-dimensional representation of the state space, $z_s$ as a well as a temporally extended action embedding, $z_a$. The latter will be used in conjunction with a higher level policy that plans in this abstract action-space, $z_a$. By using these two embeddings, the authors test the proposed system on a set of Mujoco tasks and show improved results.\n\nPositives:\n1) Fairly simple objective, in line with previous work on unsupervised learning methods to representation learning in RL (like DARLA, variational intrinsic control, etc).\n2) The temporally extended nature of the action embedding makes is particularly attractive for HRL systems as a continuous space of options (via $z_a$).\n\n\n\nQuestions and Points of improvements:\n1) Main concern: The need to pre-train the embedding before the RL task, I strongly believe limits the applicability of the proposed algorithm. The embeddings are trained under a uniformly random policy, which in many cases in RL is not informative enough to reach, with decent probability, many of the states of interest. Thus the embedding will reflect only a small subset of the state/action-space. Thus it will be highly depend on the tasks under consideration if this is enough variety for generalisation across the stationary distribution of more informed RL policies. Implicitly, the authors are making a continuity assumptions over the state and action space.\n(To be more precise: A particular failure case of the action embedding would be if say one of the action (down) has no effect in the part of the space where the uniform policy has explored. Now this becomes an important action in a level down the line where the agents needs to go down a tunnel -- example from Atari's Pitfall. In this case, under the embedding training, since the down has had no effect in training, this action will not be represented at all. This would means the RL algorithm could not ever learn to use it). \nThe co-evolution of the representation and the RL policy, I think it's paramount especially when dealing with exploration.\n\n2) Q: Section 3.2: \"we extend... to work with temporally extended actions while maintaining off-policy updates ..\". Can the authors expand on how this is done? Both updates in this section seem to be on policy ($\\mu$).\n\n3) Q: Section 3.2: \"Q can only be trained on $N/k$ observations. This has a substantial impact on sample efficiency\". Note that this is actually an explicit trade-off between reduced number of samples we see ($N/k$) and the increased horizon in propagating information, due to the effective k step update. This trade-off need not be optimal for $k=1$.\n\n4) Notes of experiments:\na) It is hard to assess the difficulty of the exploration problems investigated. This relates to point 1) and the implicit assumptions highlighted there. \nb) It would have been nice to have a study on $k$ and it's impact on the sample complexity. The larger the $k$ the harder the representation learning problem becomes; and possibly the larger the number of samples needed to learn in this combinatoric space. How does this trade-off with the benefits one could potentially get in the RL phase?\nc) For the comparison algorithms: where any of these using a temporal extended update rule?  Or are all of them 1-step TD like algorithms? It would good to separate the effect of the multiple-step update in Sec. 3.3 and the exploration in this abstract action space.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByxvmHURtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author propose a representation learning method based on predictive information. They compress a start state and an action sequence to predict the following state. Since the latent space is factorized between state and action sequence, it can be used as an abstract action space to accelerate model-free algorithms.\n\nStrengths:\n- While the state representation is a simple successor representation, the action abstraction is a simple method that seems novel.\n- The multi-step return is a nice way of handling variable horizons in the context of temporally abstract actions.\n- It is nice to see that the representation learning method can accelerate learning not just from pixels but also when learning from low-dimensional inputs.\n- The method description and overall writing is very clear.\n\nWeaknesses:\n- Doesn't the multi-step return render the update on-policy, since the reward sequence is tied to the data collecting policy? If so, it might be worth to apply off-policy corrections from the literature. If not, this should be explained in Section 3.2.\n- A comparison across more domains would be desirable. While there are 6 visual tasks, they share only two environments. The paper could be strengthened by comparison on standard benchmarks such as Gym or DMControl. I'm willing to raise my score when these or comparable results are added.\n- I could not find a clear description of how the hyper parameters of baseline methods were selected, so it is unclear how much of the benefit comes from tuning.\n\nComments:\n- Equation numbers are missing on page 4.\n- An assumption of the work is that the pixel observations are Markovian. Maybe I missed this in the paper, but was there any frame stacking that would make this hold at least approximately?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how self-supervised objectives can improve representations for efficient RL. The reviewers are generally in agreement that the method is interesting, the paper is well-written, and the results are convincing. The paper should be accepted.
Decision: Accept (Poster)