Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript proposes to reduce the intensive computation and memory requirement in reinforcement learning trainings by freezing the parameters of lower layers early. Besides, the authors also propose to store the low-dimensional latent vectors rather than the high-dimensional images in the replay buffer for experience replay. The effectiveness of the proposed techniques is evaluated on DeepMind Control environments and Atari. The motivation for this work is strong, and the results are impressive. However, the proposed technique is described in a very general way without clearly defined applicable conditions and specific design methods. Below are detailed comments and questions.

1: the main idea is to freeze lower layers of CNN encoders. However, for a certain network with various structures, is there any applicable conditions or limitations? How to choose the number of layers to freeze? The proposed technique needs to rigorous definition and explanation. 

2: It seems the reduction comes from the freezing of lower layers. I am wondering what is the computation/memory requirement breakdown in terms of layers? Is it always the case that lower layers consume a significant amount of computation and memory? If it is not the case, can LeVER still be effective?

3: this paper also proposes to store latent vectors instead of high-dimensional images. There again lacks detailed description and explanation of the applicable conditions and to what extent we can reduce the dimension of the latent vectors.

review_title: The experimental results show impressive improvement, but the proposed technique lacks rigorous definition and explanation. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: - Summary
    - This paper presents a method for compute- and memory-efficient reinforcement learning where the visual encoder is frozen partway into training.  After freezing latent vectors are stored in the replay buffer instead of images (and any existing images are replaced by them).  This leads to both better compute and memory utilization.
    - The authors demonstrate their method by comparing to Rainbow on Atari and CURL on DM Control.  On DM control, their method reduces the compute by a considerable margin.  On Atari, the results are less clear cut, but the compute cost is reduced.
    - When they also impose a memory constraint, the effectiveness of their method is further increased.
- Strengths
    - Elegant and "obvious in hindsight" (a good thing) idea, meaning it will likely have broad applicability
    - While the authors only tested it on off-policy methods, it is clearly also applicable to on-policy methods that use a rollout storage (PPO, PPG, IMPALA, V-MPO, A2C, etc.)
    - Good FLOPs vs. Reward results on DM Control
- Weaknesses
    - The memory constrained results seem very contrived.  60 MB is a tiny amount of memory and even 9.0 GB of (presumably) CPU memory isn't that prohibitive.
        - Perhaps if wall-clock time was plotted in addition to samples, the smaller memory footprint of LeVER would mean the replay buffer can be stored on the GPU and training would be much faster since many expensive CPU -> GPU transfers would be eliminated?
    - The CNN is frozen all at once instead of frozen iteratively. Raghu 2017 and Figure 6c suggest that the early layers could be frozen much earlier, although this may increase the memory usage initially since CNNs typically increase the memory size of the feature map in lower layers.
    - T_f seems like yet another hyper-parameter to tune.  In theory, SVCCA (or PWCCA from Morcos 2018) could be used to choose when to freeze (if the representation of the shallowest unfrozen layer didn't change in the last K steps, freeze it).  There is a nontrivial cost to computing either of those so it  
- Suggestions for improvement
    - I very much like the idea of this paper, but I think the chosen application is making the idea look less convincing (i.e. freezing the CNN isn't really that impactful when the CNN and observations are tiny).  I urge the authors to try this for visual navigation (i.e. PointGoal Navigation in Habitat/AI2 Thor/etc), where deeper CNNs, e.g. ResNet18, and higher resolution images, e.g. 256x256, are used.
    - One other potentially benefit of LeVER is the ability to increase the batch size during training (as in Smith 2017).  This could perhaps increase its effectiveness further?
    - In figure 6a, there should also be a CURL + LeVER from Scratch line.  Currently two variables are changing.
    - One paper that should be cited is Fang 2019.  They do a very similar thing as LeVER out of necessity.
- Overall
    - While I like this paper and think the idea has a lot of potential, I don't think it is quite ready for publication yet.  I urge the authors to try their idea in a setting with a larger CNN and higher resolutions and to see if there is a way to find T_f without it being "yet another hyper-parameter".
- References
    - Smith 2017: https://arxiv.org/abs/1711.00489
    - Morcos 2018: https://arxiv.org/abs/1806.05759
    - Fang 2019: https://arxiv.org/abs/1903.03878

## Post Rebuttal

I thank the authors for their responses. The results with a larger CNN and increased batch size help show the benefit of the method further.  I still believe the presentation of the method would be considerably stronger if results were presented in a setting with larger CNNs and higher resolution.

review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: Significance:
The paper proposes to reduce memory and computation demands of an image based RL by exploiting early convergence of the convolutional encoder. While the approach is quite intriguing, I find it hard to see the approach being general and thus having a significant effect on the RL community. 

Pros:
The paper provides an interesting approach in order to save memory and computational footprint in image-based deep RL. The method is based on an observation that the convolutional encoder converges faster then the actor and critic networks.
The authors provide an extensive comparison and ablation study that covers different domains (DMControl and Atari). The ablation study shades more light on some of the properties for the training dynamics of an image based model free RL algorithm (attention map, layer convergence)
Cons:
I’m a bit skeptical about the generality of this approach. Freezing the encoder’s weights prematurely prevents the encoder to adequately encode priorly unseen images (out of distribution), this in turn will hurt the agent’s performance down the road. Note that in DMControl the method is only showcased on the simplest tasks (at least cheetah run should be included) where learning a good policy only takes about 100K env steps -- enough to collect sufficient data support to learn an almost optimal policy. Figure 6c adheres to my point here, as conv1 weights pretty much don’t change, suggesting convergence.
The task transfer (walker stand to walker walk) experiment is exactly the same as the one demonstrated in SAC-AE (https://arxiv.org/pdf/1910.01741.pdf, Section 5.3). I’m not sure what is the difference here besides using CURL instead of SAC-AE. Could the authors elaborate? Also the domain transfer experiment (App G) shows that the approach doesn’t really buy anything.
It seems that the approach requires storing 4 data augmented observations per env observation and the replay buffer size is equal to the number of training steps. I would like to point out that DrQ (https://arxiv.org/pdf/2004.13649.pdf) only needs a constant size replay buffer of 100K transitions, even if training requires 3M steps. Given that, I’m skeptical that LeVER would buy much in terms of memory and computation here. It would be nice to see a head to head comparison.
Results are demonstrated over 3 random seeds, which is too few to get any conclusive statistical evidence given the variance. A common practice is to use 10 seeds for DMControl and 5 seeds for Atari.


Quality:
While the technical contributions of the paper are limited in novelty and significance, and don’t meet the high acceptance bar of ICLR, I still think the paper is well done and could be a good workshop paper.


Clarity:
The paper in general is clearly written and well organized. I particularly appreciate the contribution bullet points and the experimentation roadmap.

------------------------------------------
Post rebuttal:

The authors has addressed several of my concerns regarding the method's generality and some experiments. While I'm raising my score to 5, I'm still not convinced that the paper proposed a valuable contribution to the community -- comparing RL algorithms in memory or compute footprints instead of the number interactions with the environment is not meaningful, especially when a simulator is in use. There are several much simpler things one can do to tradeoff compute or memory (for example re-render observations on the fly from stored low dimensional states). Thus, I'm voting for a rejection. 

review_title: A Memory and compute optimization method that replaces an image based replay buffer with a buffer that stores low dimensional learned representations.

final_score: rating: 7: Good paper, accept
review_content: ### Paper summary

This work proposes LeVER, a method that modifies general off-policy RL algorithms with a fixed layer freezing policy for early embedding layers (in this particular case, a few early layers of a CNN). As a direct consequence, the method enables to store embeddings in the experience replay buffer rather than observations, with a potential decrease in memory required, as well as providing a boost in clock time due to fewer gradient computations needed for every update. The method is benchmarked with a couple of off-policy RL algorithms against a few different environments.

### Good things

- The approach is extremely relevant to most of the RL community. We are training for longer periods of time and generating significantly more data than we did a few years ago, so any method that enables to increase training efficiency is extremely welcomed.
- The method is simple, but clever, and the manuscript quite nicely details the steps taken to improve learning stability (which can arise due to the obvious possibility of bad model freezes).
- The coverage of related work in the literature throughout the manuscript is excellent, and provides enough pointers for the reader to understand how the manuscript is placed in it.
- The experimental setting clearly states hypotheses and questions that will be answered.
- Section 5.4 convincingly argues that the freezing method is empirically justified.

### Concerns

This is a good paper, so generally I don't have any strong negative thoughts, however I think it would be good to report how the method does when different choices are made with respect to how much of the network is frozen.
That is, in the proposed experiment setting the choices were reasonable but nonetheless a little arbitrary, so knowing a little bit more about learning dynamics with this approach would probably make the paper stronger and more robust for future readers.

### Questions:

- I wonder whether the authors would shed more details on the transfer learning setting (e.g. whether the transfer capacity changes wrt. changes in method hyperparameters such as freezing time, different saved embedding functions, etc.), and whether the results do generally show up in more environments/algorithms.
- The reduction in variance after the freezing is interesting; I wonder if the plots could show all the single runs, and whether the authors have any explanations for this somewhat consistent (!) change in learning behaviour.

review_title: Simple idea, good manuscript.

Meta-review: 
This paper presents approach to improve compute and memory efficiency by freezing layers and storing latent features. The approach is simple and provide efficiency. However, there are concerns as well. One big concern is that the experiments are not on realistic settings for example real world images and the current CNN is too simple. Overall, the reviewers are split. The AC agrees with some of the reviewers that for a paper like this experiments on more realistic setting will make it significantly stronger. 

Decision: Reject