Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: The paper proposes training dynamics-aware embeddings of the state and k-action sequences to aid the sample efficiency of reinforcement learning algorithms. The authors propose learning a low-dimensional representation of the state space, $z_s$ as a well as a temporally extended action embedding, $z_a$. The latter will be used in conjunction with a higher level policy that plans in this abstract action-space, $z_a$. By using these two embeddings, the authors test the proposed system on a set of Mujoco tasks and show improved results.

Positives:
1) Fairly simple objective, in line with previous work on unsupervised learning methods to representation learning in RL (like DARLA, variational intrinsic control, etc).
2) The temporally extended nature of the action embedding makes is particularly attractive for HRL systems as a continuous space of options (via $z_a$).



Questions and Points of improvements:
1) Main concern: The need to pre-train the embedding before the RL task, I strongly believe limits the applicability of the proposed algorithm. The embeddings are trained under a uniformly random policy, which in many cases in RL is not informative enough to reach, with decent probability, many of the states of interest. Thus the embedding will reflect only a small subset of the state/action-space. Thus it will be highly depend on the tasks under consideration if this is enough variety for generalisation across the stationary distribution of more informed RL policies. Implicitly, the authors are making a continuity assumptions over the state and action space.
(To be more precise: A particular failure case of the action embedding would be if say one of the action (down) has no effect in the part of the space where the uniform policy has explored. Now this becomes an important action in a level down the line where the agents needs to go down a tunnel -- example from Atari's Pitfall. In this case, under the embedding training, since the down has had no effect in training, this action will not be represented at all. This would means the RL algorithm could not ever learn to use it). 
The co-evolution of the representation and the RL policy, I think it's paramount especially when dealing with exploration.

2) Q: Section 3.2: "we extend... to work with temporally extended actions while maintaining off-policy updates ..". Can the authors expand on how this is done? Both updates in this section seem to be on policy ($\mu$).

3) Q: Section 3.2: "Q can only be trained on $N/k$ observations. This has a substantial impact on sample efficiency". Note that this is actually an explicit trade-off between reduced number of samples we see ($N/k$) and the increased horizon in propagating information, due to the effective k step update. This trade-off need not be optimal for $k=1$.

4) Notes of experiments:
a) It is hard to assess the difficulty of the exploration problems investigated. This relates to point 1) and the implicit assumptions highlighted there. 
b) It would have been nice to have a study on $k$ and it's impact on the sample complexity. The larger the $k$ the harder the representation learning problem becomes; and possibly the larger the number of samples needed to learn in this combinatoric space. How does this trade-off with the benefits one could potentially get in the RL phase?
c) For the comparison algorithms: where any of these using a temporal extended update rule?  Or are all of them 1-step TD like algorithms? It would good to separate the effect of the multiple-step update in Sec. 3.3 and the exploration in this abstract action space.





review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper presents an approach to learning state and action representations through self-supervision, such that these representation can be used for downstream reinforcement learning. In particular, the proposed approach learns a  time-dilated dynamics model on data collected via self-supervision, where given s_t, and actions (a_t, ..., a_{t+K}) predicts s_{t+K}. The input state and action trajectory and each encoded into latent distributions, which are then used to reconstruct the future state. Then, they demonstrate that using TD3 with the latent action space outperforms existing model-free methods and existing state representation techniques.

Overall the paper is well motivated and clearly written. The key contribution seems to be in learning the latent distribution over multi-step action trajectories, which seems to be important for performance. Lastly the experiments and ablations are thorough and well explained.

My main comments have to do with (1) the fairness of the comparison to existing model-free RL, (2) an analysis of the temporal abstraction for learning the action distribution.

(1): The proposed method first pre trains the latent dynamics model on 100K steps of random data, then trains the proposed TD3 using this action distribution (and the modified critic to support 1-step Q values). While this does outperform the model-free RL methods trained from scratch, it is also using 100K steps worth of experience that the others don't have access to, which makes it not quite a fair comparison. If you pretrain the critic of TD3 or SAC with the 100K samples, do you still observe the same performance gains?

(2): From the ablation study and comparison to other state representation learning techniques in Figure 6, it seems like the most important aspect of the proposed method is using the latent action distribution. This makes sense as it captures longer action sequences, and thus likely is the reason for better exploration and performance. As a result the exact choice of K seems very important. In the Appendix it states that for the Thrower task K=8, and elsewhere K=4. Do the authors have a sense for how performance changes with choice of K? I think a plot which compares performance over different choices of K would be very valuable.

Some smaller comments:
- The comparison to other model-free RL methods is done only on low dimensional states, while the ablations are done on pixels. Is this because the model-free comparisons did not work at all on pixels?
- Is it possible to perform model predictive control with the learned model, and how does it compare to existing latent model based RL methods (Hafner et al.)
- One more recent work that may be worth comparing to is SLAC (Lee et al.) which also learns a stochastic latent dynamics model, and learns a policy in the latent space of the model. The latent space is of states however, and not actions. 

______________________

Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine. Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson. Learning Latent Dynamics for Planning from Pixels

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The author propose a representation learning method based on predictive information. They compress a start state and an action sequence to predict the following state. Since the latent space is factorized between state and action sequence, it can be used as an abstract action space to accelerate model-free algorithms.

Strengths:
- While the state representation is a simple successor representation, the action abstraction is a simple method that seems novel.
- The multi-step return is a nice way of handling variable horizons in the context of temporally abstract actions.
- It is nice to see that the representation learning method can accelerate learning not just from pixels but also when learning from low-dimensional inputs.
- The method description and overall writing is very clear.

Weaknesses:
- Doesn't the multi-step return render the update on-policy, since the reward sequence is tied to the data collecting policy? If so, it might be worth to apply off-policy corrections from the literature. If not, this should be explained in Section 3.2.
- A comparison across more domains would be desirable. While there are 6 visual tasks, they share only two environments. The paper could be strengthened by comparison on standard benchmarks such as Gym or DMControl. I'm willing to raise my score when these or comparable results are added.
- I could not find a clear description of how the hyper parameters of baseline methods were selected, so it is unclear how much of the benefit comes from tuning.

Comments:
- Equation numbers are missing on page 4.
- An assumption of the work is that the pixel observations are Markovian. Maybe I missed this in the paper, but was there any frame stacking that would make this hold at least approximately?
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper presents DynE, a self-supervised approach for learning dynamics-aware state and action representations. DynE learns an encoding of individual states and action sequences, assigning nearby embeddings to those that have similar outcomes. This is achieved via a reconstruction objective that predicts the outcome of a sequence of "k" actions from a given state, along with losses that encourage compression in the learned latent representations. Additionally, a learned decoder allows for reconstruction of minimum-norm action sequences from the high-level latent action embedding. Combining DynE state and action embeddings with an actor-critic agent operating directly in the learned high-level action space leads to significant speedups in both from-scratch and transfer learning (on 2D and 3D OpenAI Gym tasks), leading to better data efficiency compared to model-free baselines. Additionally, the learned action and state embeddings lend themselves to better exploration and consistent value prediction, respectively.

The paper is very well written and the approach looks quite promising. A few comments:
1. The approach is well validated but additional ablation results can help quantify the effect of different components. For example, it would be useful to see the effect of varying "k", the number of actions to be encoded for generating the action embedding.  
2. A related paper that learns state representations that are physically consistent and dynamics-aware is this work:
Jonschkowski, Rico, et al. "Pves: Position-velocity encoders for unsupervised learning of structured state representations." arXiv preprint arXiv:1705.09805 (2017).
Here the state representation is learned to implicitly encode physical consistency via self-supervised losses that mimic constraints such as controlability, inertia, conservation of mass etc. Combining such additional self-supervised losses can help structure the state embedding learning further, albeit at the cost of introducing additional hyperparameters during optimization.
3. It would be useful to know what the actions are (and their dimensions) for the tasks considered in the paper.
4. The paper would benefit from a short discussion on the limitations of the proposed approach and potential to scale to more complicated tasks.
5. Fig. 5, bottom right: It is not clear why PPO (blue) performs significantly better on this task compared to the other 7DoF tasks considering that the thrower should be more complex than the pusher and striker. PPO also seems to match the data efficiency of DynE-TD3. Is this correct? 

Overall, I find the approach quite interesting and promising. I would suggest an accept.

Typos:
1. Intro, 2nd para, 2nd line, many samples to learn than a better one
2. Fig. 1, the pixel representation is very unintuitive
review_title: Official Blind Review #1

Meta-review: This paper studies how self-supervised objectives can improve representations for efficient RL. The reviewers are generally in agreement that the method is interesting, the paper is well-written, and the results are convincing. The paper should be accepted.
Decision: Accept (Poster)