Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Learn a hierarchy of latent models where each level takes in the latent state and a length n concatenation of actions and predicts the state after n steps, instead of simply predicting single step transitions. Information is passed from the layers predicting longer temporal distance to the lower levels. The representations are trained with l2 loss between the representations and the encoded observations. The representation is then used by SAC with h critics, corresponding to different levels of the hierarchy, and a policy representation that is based on the encodings learned by the levels.
strength_and_weaknesses: The reasoning that single step forward models fail to capture relevant information is a bit weak in the context of an MDP, where the markov property should ensure that all information is captured in a single state. 

It isn't clear exactly how the encoder is prevented from collapse, since if the encoder is encoded to zero state then the forward models would have zero loss.

While the results are promising and the writing is clear, the intuition behind what the different levels of the hierarchy might be encodinig. In particular, it would be nice ot have some visualization demonstrating how the higher levels capture different information from the lower ones, and which parts of the encoding the policy attends to. This might be more informative than indicating if task-relevant components are attended to, since it isn't clear why hskl would be especially effective at capturing task-relevant components. Otherwise, it is possible that a multi-head representation might get similar results, without the heads having temporally different meanings.

The experiments have many domains with only two levels, except for the falling pixels. It might be nice to look at a very long horizon tsk with a large number of levels.

clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear in terms of what it is doing, though the method is somewhat unintuitive. The writing does not detract from the paper. While methods doing multi-step dynamics modeling have been used in RL, having multiple encoders appears to be a novel contribution. The experiments are somewhat weak, since the domains are such that the number of layers remains low, and it is not clear what the layers are learning that is distinct between each other.
summary_of_the_review: I think this paper can be accepted because it appears to provide a novel, simple change to existing model-based state representation RL methods, and it is implemented well enough with sufficient experiments to constitute a contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper demonstrates how learning an ensemble of hierarchical models that predict latent representations at varying step-sizes into the future can be used for representation learning in model-free reinforcement learning. The proposed hierarchical technique is not used for planning, but rather the representations learned using their technique enable more efficient model-free reinforcement learning from pixels than baseline techniques. The paper provides compelling ablation results, baseline comparisons across 30 environments plus one newly introduced pixel-based control task, and releases all code needed to train their model and run the new environment.
strength_and_weaknesses: Strengths
* Clarity of writing and presentation. The paper is very easy to read. The motivation is clearly laid out. The method is clearly explained for either a casual reader (main text) or more detailed reader (mathematically laid out in the appendix). The results are sign-posted and follow sensible questions to ask given the setup of the approach.
* Statistics and reproducibility. The authors use interquartile means to show that their approach statistically outperforms recent baselines, as well as the various ablations of their model. The comparisons seem sound, and the diversity of experiments underscores the general utility of the hierarchical model-based representation learning technique. The authors also provide a link to code to ensure that results are reproducible. I commend the authors for their scientific openness and careful scholarship.
* Novelty and impact. The idea of using hierarchical models for reinforcement learning is not novel for model-based reinforcement learning, where the hierarchical models are used for planning. However, I am not aware of work that uses hierarchical models in model-free reinforcement learning as a way of learning better representations that can ignore distractors. I thought this was a nice twist, and this paper would invite more investigation from others looking at hierarchical models for model-based RL to think about using the same approaches for generic representation learning in model-free RL instead.

Weaknesses
* Overall I really enjoyed reading this paper. However, I had one big question related to the framing of the method as being useful for sample efficiency. The presented approach is model-free, and results are presented after a *tiny fraction* of the number of environment steps often shown for model-free algorithms. The paper highlights the usefulness of their approach (HKSL) for exactly this kind of efficiency. BUT normally we would use model-based techniques to improve sample efficiency, which are absent from this paper. It would be particularly compelling if the presented method can outperform model-based techniques when environment samples are limited, or if the presented method still outperforms model-free baselines even in the limit of significant experience. 
  * To address this concern, could the authors provide results for one of two experiments:
    * Compare to a model-based reinforcement learning method that uses the learned hierarchical models for planning, or really any other model-based RL technique, for the low-sample regime? I would like to understand whether the improvements here in model-free actually rival what one would expect for model-based.
    * Compare results for massive amounts of experience as well (ie show the asymptotic behavior of all the presented baselines, not just their behavior after 50k or 100k steps). If HKSL, the presented method, still outperforms baselines in the limit of significant experience, that would ameliorate my concern about “why not just use model-based RL”.



clarity,_quality,_novelty_and_reproducibility: See my comments above. One other small fix for the authors:
* “?? shows the MSE and ± one standard deviation over the testing episodes using encoders” – ?? should be Figure 6 I think

summary_of_the_review: Overall I think this paper is clear, novel, potentially impactful, and represents sound and good science. I am recommending acceptance, but would reduce my score if the authors do not respond to my questions about model-based vs. model-free RL techniques.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main contribution of the paper is a hierarchical architecture for predicting observations at different levels of temporal coarseness, and its evaluation for RL from images.

The individual dynamic models for the different temporal coarseness levels are independent from each other and each consist of an encoder, a latent-dynamics model and a nonlinear projection network. Furthermore, except for the lowest level dynamic model, each model also contains a communication manager that summarizes all latent predictions within a trajectory segment and provides it to the latent dynamic model of the next lower level as additional input.

The models are evaluated using a modified SAC agent, which uses an example of Q function: one critic per temporal coarseness level. Furthermore, the policy gets all embeddings (1 per coarseness level) of the current observation as input. The dynamic model itself, however, is only used for representation learning, but not directly used during RL. The modified SAC agent is evaluated at the DMControl suite and on a new toy task "falling pixels". The result show that the proposed method can outperform suitable baselines in terms of sample efficiency.

strength_and_weaknesses: - The paper tackles an important problem on how to capture and predict the effects of the agent's actions at different temporal coarseness levels. (+)
- The method/architecture is sound and achieves good performance (+)
- The paper is well written (+)
- The results seem to be well reproducible, code published (+) 
- A few parts are a bit unclear (-)
- The paper does not require theoretical analysis or derivations, so the novelty is limited to the network architecture (-)
clarity,_quality,_novelty_and_reproducibility: Quality
--------
The quality of the paper is good. The approach is sound, the claims are substantiated and the evaluation is sufficiently thorough and considers reasonable baselines.
However, I think that the chosen coarseness levels have not been sufficiently well discussed and evaluated. For the DMC control task the paper only considers two levels. Only for the falling pixel environment more than two levels (3) have been evaluated. Given that the topic of the paper is a hierarchical architecture, it would be crucial to evaluate the effects of the number of chosen layers, and the chosen coarseness levels.


Clarity
--------
The paper is well written, and mostly clear. However, I have a few questions that I would like the authors to address:
1. What is the purpose of the projection layer? If we omit it in Eq. 1, the l2 loss would try to match the actual latent with the predicted latent, which is reasonable, although I see the problem of mode collapse (an encoder that always predicts the same loss would achieve result in zero loss). How does the projection layer alleviate this problem?
2. For the plots in Fig. 5 and 6: Which encoder was used for computing the HKSL MSE? Did you concatenate the different embeddings (as in the policy input)? I would expect to use the corresponding embedding depending on how many steps ahead we want to predict.
3. There seem to be only 18 different colors in Fig. 7 (right), although there should be 20 trajectories. Why?

Originality
-------------
The proposed architecture is novel and interesting. Although there are no theoretical justification for the specific choices, the architecture is sounds and its evaluation thereby interesting.

Reproducibility
--------------------
The paper seems to be well reproducible, as the code is published and in a good state.
The procedure for tuning the temporal coarseness and the number of hierarchies should be discussed.
summary_of_the_review: The paper is well written and of high quality, with sufficient novelty.
However, I outlined a few issues that I would like the authors to address:
1. Ablations of critical hyperparameters (number of hierarchies, chosen coarseness) are missing.
2. There are some issues regarding the clarity
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach towards data-efficient reinforcement learning that leverages a hierarchy of latent forward models to train representations at different levels of temporal abstraction. The models are designed to take in encoded observations at a given timestep t and a sequence of actions (where sequence length >= 1) and can predict future encodings of observations at different timescales depending on the coarseness of the corresponding model. Additionally there is a communication network that passes information from forward models at slower temporal scales to more fine-grained temporal scales. The models are trained using squared error between predictions and "true" latents, and are effectively used to shape the learning of observation representations; the representations from different levels are fed into the actor to predict the final action output by the policy. The approach (combined with SAC) is tested on several continuous control tasks from the DMControl suite along with variations such as distractors, and changes to camera pose or object colors; an additional task of "falling pixels" that potentially requires reasoning at different temporal abstractions is also used as a test environment. Comparisons are made against baselines which use different objectives to shape representations on top of SAC. Overall it performs better than the compared baselines at low sample complexity (100k transitions) thereby being more data-efficient. A few ablations and analysis of the learned representations are provided.
strength_and_weaknesses: Strengths:
1. The paper provides an interesting approach towards training a hierarchy of forward models (albeit at explicitly specified levels of temporal abstraction) and shows that the resulting method performs well on a standard set of benchmarks. 
2. The addition of the communication module to communicate information from higher to lower levels is novel and seems to help improve performance on the tested domains.

Weaknesses:
1. The key novelty of the approach is on the addition of a hierarchy of forward models but there is no clear ablation in the paper on the sensitivity of the approach to the choice of hyper parameters of the hierarchy. Specifically there is no ablation across the number of levels used (h), and the number of steps skipped by a given level (n^l). The only relevant ablation provided was with [h=1] or [all n=1]. The appendix mentions that these parameters were chosen after a short search -- would be great if all the results across this search are presented. This would provide more empirical evidence to the stability and generality of the provided approach which is currently hard to evaluate.
2. Table 3 of the appendix presents hyperparameters used with HKSL. There are 5 different sets of parameters for the 7 tasks presented in the paper -- parameters varied include the trajectory length (k), number of levels (h), number of skipped steps (n^l) and learning rate. Most prior work (e.g. DrQ, CURL) uses atmost one or two sets of hypers for a variety of tasks. Is there a particular reason to tune the hypers for HKSL? Relating to the previous question, is it due to the sensitivity of HKSL to hypers? Would be good to clarify this.
3. On the topic of hypers, how does "k" affect the number of sequential predictions a model needs to make? For "Finger Spin" and "Reacher Easy" k = 3 but n_l = [1, 3]. Does this mean that the higher level forward model (n_1 = 3) does only a single forward prediction? Please clarify this in the text.
4. All the results presented are only for 100k env steps. Most prior work reports results on 100k and 500k env steps as well as asymptotic performance; is there a reason why this is not done in the proposed approach? It would be useful to add these results as it would be good to know what the asymptotic performance of the method is, not just whether the data efficiency in the low sample complexity regime is good. 
5. All the presented baselines use either some form of representation loss or single-step prediction losses. Is there a reason to not compare to a baseline that uses multi-step prediction losses, or some form of reconstruction objective (e.g. Dreamer)? While reconstruction approaches can potentially fail on the distracting control suite they might do quite well on some of the analysis results presented (such as MSE prediction error of object positions). Would be a useful comparison to add.
6. Re: Fig 7, it was mentioned that the data used for this comparison was collected using random policies. Why is this data interesting to consider for this particular analysis? Wouldn't it make sense to look at trajectories that are temporally consistent in some form for this comparison? It is also mentioned in the analysis that "This suggests that c considers the transition of the level below it when deciding what information to share", how do the results from Fig 7 (left) show this? From my perspective it only shows that the error scales proportionally to "t", not that the communicator (c) uses knowledge from the lower-level (which it doesn't have access to) to decide which information to communicate. It would be great if this point is explained better.
7. Out of the tasks tested in the paper only the "Falling Pixels" task introduced in the paper explicitly requires some form of reasoning across temporal scales. It would be helpful if the approach was tested on atleast one other task (e.g. from Atari?) where temporal abstraction of some form was necessary.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and the approach is reasonably well explained. While some of the notation is a bit dense, Fig 1 makes it easier to understand the overall approach. Some of the details of the implementation are hidden in the appendix (Table 3), would be good to explicitly call this out. And some statements (e.g. see point 6 in Weaknesses) don't fall out directly from the results; would be good to explain these better.

Quality & Novelty: The specific use of hierarchical models to learn representations for sample efficient RL is somewhat novel but the results are only marginally better than related work, and some key ablations, asymptotic performance results & baselines need to be included to quantify the contributions better.

Reproducibility: The draft provides a reasonable amount of details and builds on top of existing prior work so in my opinion it can be reproduced with a reasonable amount of effort. 
summary_of_the_review: Overall, while the approach presents a novel application of hierarchies of forward models to shape representations for data-efficient RL, the results are a bit weak. Some ablations, asymptotic performance results and analysis of the sensitivities of the approach are missing. As such I would recommend to reject this paper in its current form.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors introduce a hierarchical k-step auxiliary task for representation learning to capture structure at multiple temporal resolutions to accelerate model-free RL from pixel-based inputs.  As motivation, they note that forward models are one kind of auxiliary representation learning objective that could increase data efficiency, but observe that 1-step forward models may be less relevant than temporally abstract forward models, if the tasks are long-horizon.  

The reviews for this paper were ultimately all borderline, with 3/4 reviewers leaning weakly positive and the remaining reviewer leaning weakly negative.  More specifically, reviewers tended to find the paper clear though not necessarily very well motivated, and with some arbitrary design choices having been made (variously described as unintuitive, not theoretically motivated, etc.).  Reviewer WeT7 notes that it is odd that the task didn't really require long horizons, despite that being part of the motivation.  Reviewer X4PZ requested a model-based approach as a baseline.  The authors included DreamerV2 as a baseline.  Reviewer 6JA8 originally identified lack of ablations of number of hierarchy layers as an omission.  The authors added this ablation and that, along with other clarifications and the added baseline, prompted the reviewer to update their score (3-->5).  

justification_for_why_not_higher_score: Overall, starting with the motivation, following through the design choices and the task choices, there is something not entirely adequate about how the design choices are motivated.  Due to the many moving parts that potentially required tuning, I'm not convinced that without the clarity of motivation, the approach is likely to be sufficiently impactful to justify acceptance.
Decision: Reject