Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: - This paper presents a method for compute- and memory-efficient reinforcement learning where the visual encoder is frozen partway into training.  After freezing, latent vectors are stored in the replay buffer instead of images (and any existing images are replaced by them).  This leads to both better compute and memory utilization.
- The authors demonstrate their method by comparing to Rainbow on Atari and CURL on DM Control.  On DM control, their method reduces the compute by a considerable margin.  On Atari, the results are less clear cut, but the compute cost is reduced.
- When they also impose a memory constraint, the effectiveness of their method is further increased.

main_review: - Strength
    - Elegant and "obvious in hindsight" (a good thing) idea, meaning it will likely have broad applicability
    - While the authors only tested it on off-policy methods, it is clearly also applicable to on-policy methods that use a rollout storage (PPO, PPG, IMPALA, V-MPO, A2C, etc.)
    - Good FLOPs vs. Reward results on DM Control
    - Results are more influential with a deeper CNN
- Weaknesses
    - The memory constrained results seem very contrived.  60 MB is a tiny amount of memory and even 9.0 GB of (presumably) CPU memory isn't that prohibitive.
        - Perhaps if wall-clock time was plotted in addition to samples, the smaller memory footprint of SEER would mean the replay buffer can be stored on the GPU and training would be much faster since many expensive CPU -> GPU transfers would be eliminated?
    - The CNN is frozen all at once instead of frozen iteratively. Raghu 2017 and Figure 6c suggest that the early layers could be frozen much earlier, although this may increase the memory usage initially since CNNs typically increase the memory size of the feature map in lower layers.
- Overall
    - This is a useful idea that improves experiment time, giving researchers a faster iteration cycle.  The idea is quite generic so it should be easy for researchers to adopt.
- References
    - Smith 2017: https://arxiv.org/abs/1711.00489

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Stored Embeddings for Efficient Reinforcement learning (SEER) is a method to reduce the computational cost of value-based off-policy reinforcement learning, e.g. Q-learning, in settings with pixel-based observations. It uses a convolutional encoder whose lower layers are frozen early in agent training, as there has been previous work showing that a convolutional network's lower layers converge earlier on in training. Following an early period of joint training with the convolutional encoder, the agent afterwards stores only the encoded image features into its replay buffer instead of the raw pixels of the image. This enables potentially significant memory and compute savings as the image no longer needs to be re-encoded during off-policy learning. The paper presents empirical results demonstrating that final agent performance is not hindered by this procedure and that, when counting in terms of total FLOPs calculated or replay buffer size, SEER has an advantage over the naive image-based replay buffers most commonly used. A variety of experimental settings are considered, differing across domains (DMControl, Atari) and training procedure (Rainbow, SAC, CURL). 
main_review: The proposed method, SEER, can be seen as an application of Freezeout to value-based RL, combined with storing embeddings instead of images in the replay buffer after the lower layers are frozen. The method seems well-motivated and straightforward, leveraging the fact that the convolutional encoder trains quickly to avoid the need to re-encode images several times during learning. Considering that reinforcement learning is a considerably different setting to supervised learning, in that the input data distribution can change significantly over the lifetime of the agent, it was not immediately obvious that Freezeout can be applied without issues in this setting.

However, the authors have provided experimental results on pixel-based DMControl and Atari that seem to validate that freezing the lower convolutional layers early-on in training does not cause decreased performance in those domains. But there are still some remaining concerns I have regarding the generality of the proposed method that result in my score of weak reject:

1) A lot of the environments the method was evaluated on are highly regular in visual appearance, even throughout policy performance levels. I checked videos of the various Atari 2600 games, and only Krull has non-static backgrounds and a larger variety of visual representations for entities. The DMControl tasks are also very static across agent performance levels. I think it is important to test how this method would fare when there exists significant distribution shifts during training, e.g. an important new entity or visual mechanic appears at a certain policy performance. Would freezing the network before this critical performance level still make later policy learning improve?
2) Even not considering shifts in distribution, the visual understanding necessary for high performance in the environments tested is limited. As mentioned in the paper's section 6, it would be interesting to test SEER on more complex visual environments which require larger image encoders to extract useful semantics.
3) Maybe owing to the visual simplicity of the environments, the ablations reveal that the convolutional layers can be frozen early with minimal reduction in final performance. Have the authors tried using features from a randomly initialized network as a baseline.
4) What is the effect on wall-clock time of training an agent using SEER? What would the graphs look like if the x-axis was experiment run-time and all agents were trained on the same reference machine?

==========================================================================

Thanks for the response, the experimental results you described have addressed most of the concerns I had above. I think those results should be included in the camera-ready paper as an appendix section. I will raise my score to a 6.

limitations_and_societal_impact: The authors discuss some limitations of their method in section 6, mentioning the additional freezing time hyperparameter introduced by SEER.

There does not seem to be a potential for negative societal impact for this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper propose a compute and memory efficient pixel wise RL algorithm by early stoping the update of CNN encoder and store the latent vector to the replay buffer instead of the images. Experiments had been done on DM Control and Atari with baseline algorithm CURL.

main_review: Strengths,

The paper is well organized and easy to follow.
The code, data, and instructions needed to reproduce the main experimental results is provided.
The compute cost is lower than baselines.
The memory cost is smaller.


Concerns and Questions,

1. The computation and the memory cost is not the very important issue in RL. The most concerns are the sample efficiency, since interact with the environment usually cost benefit loss or even damage accidents during the exploration. So although the result showed with FLOPs significantly outperformed the baseline, the result is not quit surprising, since the SEER no longer needs to compute the gradient of the encoder after freeze time. And the memory is not the concerns also, since the size of replay buffer is not the larger the better as discussed in “A Deeper Look at Experience Replay” and some other researches. 

2. Missed some related works that improves the sample efficiency in pixels state reinforcement learning. To name a few, PISAC, DRQ, SLAC, the performances of them are better than CURL. I was wondering whether freeze it at a previous time harms the performance of those algorithms. Even with CURL, as shown in Fig. 12, before the freeze line the performance is quite similar, but the performance drops after the freeze time in most task. 

3. It is hard to choice the time to freeze the encoder in practice. Seems like it needs more repetitions of the experiments to give a good choice of the freeze time? More experiments show the sensitive to the time of freeze of the algorithm would be better.



Suggestions and Discussions,
1. Before the encoder is freeze, the replay buffer still needs to store the images and if the replay buffer slots number is equal with the baseline settings, the performance is equal to the baseline in theory. After the freeze time, the replay buffer slots number grows more larger. So the benefit is seems like on the condition of the assumption that a larger replay buffer would brings performance improvement? Such assumption needs to be claimed and discussed more clearly in the paper. Further discussions and experiments on the different limitations of the memory cost would be helpful.

2. The part of transfer learning (Sec 5.4) is not quit clear to me. It seems like transfer learning is kind of a relatively independent experiment of what want to claim in this paper. If the paper want to claim that the SEER can bring benefit to transfer learning, more experiments has to be done to justify such benefit. If not, I think the paper without transfer learning experiment is enough for what the paper want to propose.


limitations_and_societal_impact: N/A
review_title: null

Meta-review: An interesting approach to addressing the problem of dealing with vision input in the reinforcement learning setting. Presented approach is more computationally efficient than its counterparts. The algorithm is also flexible enough and can be applied in the on-policy context, even though in the paper the authors focus on the off-policy setting. Well written paper. All the reviewers see the importance of the proposed method in dealing with the notoriously difficult problem of computationally-efficient vision-based RL.
Decision: Accept (Poster)