Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
Authors: Lili Chen
Kimin Lee
Aravind Srinivas
Pieter Abbeel
Venue: NeurIPS 2021 Poster (2021)
Keywords: computational efficiency
memory efficiency
reinforcement learning
deep learning

Abstract: Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games.

Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
Lili Chen'
Kimin Leel
Aravind Srinivas? OpenAA
Pieter Abbeel
UC Berkeley
Abstract
Recent advances in off-policy deep reinforcement learning (RL) have led im- pressive success in complex tasks from visual observations. Experience replay improves sample-effciency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensiona, inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RI. methods, to address these computa- tional and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training reduce memory
due to early convergence of their parameters. Additionally, requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling adaptive increase in the replay buffer capacity, useful technique in constrained-memen ments, show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games.
settings. In our experi-
Introduction
Success stories of deep reinforcement learning (RL) from high dimensional inputs such as pixels or large spatial layouts include achieving superhuman performance on Atari games [30, 37, 1]. grandmaster level in Starcraft II [50] and grasping diverse set of objects with impressive success rates and generalization with robots in the real world [21]. Modern off- off-policy RL algorithms [30, 15, 11, 12, 39, 22, 24] have improved the sample-eficienccy of agents that process high-dimensional pixel inputs with convolutional neural networks (CNNs; LeCun et al. 25) using past experiential data that is typically stored as raw observations in replay buffer [28]. However, these methods demand high memory and computational bandwidth, which makes deep RL inaccessible in several scenarios, such learning with much lighter on-device computation (e.g. mobile phones other light- edge devices).
For compute- and memory- -fficceet deep learning. several strategies, such as network pruning [13, 8], quantization [13, 17] and freezing [53, 36] have been proposed in supervised learning and unsupervised learning for various purposes (see Section 2 for more details). In computer vision, Raghu et al. [36] and Brock al. [5] showed that the computational cost of updating CNNs can be reduced by freezing lower layers earlier in training, and Han et al. [13] introduced a deep compression, which reduces the memory requirement of neural networks by producing a sparse network. natural language processing, several approaches [46, 42] have studied improving the computational efficiency of Transformers [49]. deep RL, however, developing compute- and memory-efficia techniques has received relatively little attention despite their serious impact the practicality of RL algorithms.
In this paper, we propose Stored Embeddings for Efficient Reinforcement Learning (SEER), simple technique to reduce computational overhead and memory requirements that is compatible with various
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

off- RL algorithms [10, 15, 39]. Our main idea is to freeze the lower layers of CNN encoders of RL agents early in training, which enables two key capabilities: (a) conpute-efficieney reducing the computational overhead of gradient updates in CNNs; (b) memory-effiiieny: saving memory by storing the low-dimensional latent vectors to experience replay instead of high-dimensional images. Additionally, we leverage the memory-efficiencg of SEER to adaptively increase replay capacity, resulting in improved sample-efficiency of off-policy RL algorithms in constrained-memory settings. SEER achieves these improvements without sacrificing performance due to early convergence of CNN encoders.
The main contributions of this paper are as follows:
We present SEER, a compute- and memory--eficceee technique that can be used in conjunction with most modern off-policy RL algorithms [10, 15].
We show that SEER significantly reduces computation while matching the original performance of existing RL algorithms both continuous control tasks from DeepMind Control Suite [45] and discrete control tasks from Atari games [2].
We show that SEER improves the sample-efficiency of RL agents in constrained-memoyy settings by enabling an increased replay buffer capacity.
2
Related work
Off-policy deep reinforcement learning. The most sample-efficient RL agents often use off-policy RL algorithms, a recipe for improving the agent's policy from experiences that may have been recorded with a different policy [44]. Off-policy RL algorithms are typically based Q-Learning [51] which estimates the optimal value functions for the task at hand, while actor-critic based off- off-policy methods [27, 38, 10] are also commonly used. In this paper we will consider Deep Q-Networks (DQN: Mnih et al. 30), which combine the function approximation capability of deep convolutional neural networks (CNNs; LeCun et al. 25) with Q-Learning along with the usage of the experience replay buffer [28] as well as off-policy actor-critic methods [27, 10], which have been proposed for continuous control tasks.
Taking into account the learning ability of humans and practical limitations of wall clock time for deploying RL algorithms in the real world, particularly those that learn from raw high dimensional inputs such as pixels [21], the sample-inefficiency of off-policy RL algorithms has been a research topic of wide interest and importance [23, 20]. To address this, several improvements in pixel- based off-policy RL have been proposed recently: algorithmic improvements such as Rainbow [15] and its data-efficient versions [48]: using ensemble approaches based on bootstrapping [34, 26]; combining RL algorithms with auxiliary predictive, reconstruction and contrastive losses [18, 16, 33, 52, 39, 40]; using world models for auxiliary losses and/or synthetic rollouts [43, 9, 20, 12]; using data- aata-augmentations on images [24, 22].
Compute-efficient techniques in machine learning. Most recent progress in deep learning and RL has relied heavily on the increased access to more powerful computational resources. To address this, Mattson et al. [29] presented MIPerf,, fair and precise MI. benchmark to evaluate model training time on standard datasets, driving scalability alongside performance, following a recent focus on mitigating the computational cost of training ML models. Several techniques, such as pruning and quantization [13, 8, 4, 17, 46] have been developed to address compute and memory requirements. Raghu et al. [36] and Brock et al. [5] proposed freezing earlier layers to remove computationally expensive backward passes in supervised learning tasks, motivated by the bottom-up convergence of neural networks. This intuition was further extended to recurrent neural networks [31] and continual learning [35], and Yosinski et al. [53] study the transferability of frozen and fine-tuned CNN parameters. Fang et al. [7] store low-dimensional embeddings of input observations in scene memory for long-horizon tasks. We focus on the feasibility of freezing neural network layers in deep RL and show that this idea can improve the compute- and memory-efficiene of many off-policy algorithms using standard RL benchmarks.
2

---

Forward
Actor
Critic
Forward Backward
Actor
Critic
Backward
Latent vector (dim=50)
Latent vector (dim=50)
Encoder
Encoder
Replay buffer
8484
Replay buffer
(a) SEER hefore freezing.
(b) SEER after freezing.
Figure I: Illustration of our framework. (a) Before the encoder frozen, all forward and backward passes are active through the network, and we store images in the replay buffer. (b) After freezing, we store latent vectors in the replay buffer, and remove all forward and backward passes through the encoder. We remark that more samples can be stored in the replay buffer due to the relatively low dimensionality of the latent vector.
Background
We formulate visual control task as partially observable Markov decision process (POMDP: Sutton & Barto 44, Kaelbling et al. 19). Formally, at each timestep t, the agent receives high-dimensional observation or, which is an indirect representation of the state 5t, and chooses an action de based on its policy The environment returns reward r, and the agent transitions the next observation 0++1- The return R₂ = Ex= r is the total accumulated rewards from timestep with discount factor Y [0. 1). The goal of RL is to learn policy that maximizes the expected return trajectories. By following the common practice DQN [30], we handle the partial observability of environment using stacked input observations, which are processed through the convolutional layers of an encoder fv-
Soft Actor-Critic. SAC [10] is an off-policy actor-critic method based on the maximum entropy RL framework [55], which encourages robustness to noise and exploration by maximizing a weighted objective of the reward and the policy entropy. To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement. At the soft policy evaluation step, a soft Q-function, which is modeled neural network with parameters is updated by minimizing the following soft Bellman residual:
Co ()) - -E, ~B (Qo((olo)).aa)-
-Buetiees (Ql art1) - a log To (ol(atilf
where 7: (Ot, ae, Tt, 0(+1) is transition, B is replay buffer, 0, y are the delayed parameters, and a temperature parameter. the soft policy improvement step, the policy æ with parameter is updated by minimizing the following objective:
C EB,-BBar [a log a (arlf Qo(fv(ot),a:))
Here, the policy is modeled as a Gaussian with mean and covariance given by neural networks.
Deep Q-learning, DQN algorithm [30] learns a Q-function, which is modeled as neural network with parameters 0, by minimizing the following Bellman residual:
0a (0, = ETT,B (oolUsta)).aa) ox) Ti max QTl Qo((E(Ot1))a)))
where T. = (0r, út, Ft, 0t++) transition, a replay buffer, and 0, T are the delayed parameters. Rainbow DQN integrates several techniques, such double Q-learning [47] and distributional DQN [3]. For exposition, we refer the reader Hessel al. [15] for more detailed explanations of Rainbow DQN.

---

4 SEER: Stored Embeddings for Efficient Reinforcement Learning
In this section, we present SEER: Stored Embeddings for Efficient Reinforcement Learning, which can be used in conjunction with most modern off-policy RL algorithms, such SAC [10] and Rainbow DQN [15]. Our main idea is to freeze lower layers during training and only update higher layers, which eliminates the computational overhead of computing gradients and updating in lower layers. We additionally improve the memory-efficienee low-dimensional latent vectors in the replay buffer instead of high-dimensional pixel observations. See Figure I and Appendix ?? for more details of our method.
of off-policy RL algorithms by storing
4.1 Freezing encoder for saving computation and memory
We process high-dimensional image input with an encoder f to obtain 2, = f& (ot), which is used as input for policy a and Q-function Qo as described in Section 3. In off-policy RL, we store transitions (oz, 0,,004 1. re) in the replay buffer B to improve sample-efficiency by reusing experience from the past. However, processing high-dimensional image input or is computationally expensive. To handle this issue, after T'y updates, we freeze the parameters of encoder vú, and only update the policy and Q-function. We remark that this simple technique can save computation without performance degradation because the encoder is modeled as deep convolutional neural networks, while shallow MLP is used for policy and Q Q-function. Freezing lower layers of neural networks also has been investigated in supervised learning based on the observation that neural networks converge to their final representations from the bottom-up, i.e., lower layers converge very early in training [36]. For the first time, we show the feasibility and effectiveness of this idea in pixel-based reinforcement learning (see Figure 7a for supporting experimental results) and present solutions to its RL-specific implementation challenges.
Moreover, order to save memory, we consider storing (compressed) latent vectors instead of high-dimensional image inputs. Specifically, each experience in B is replaced by the latent tran- sition (2,, at, rr), and the replay capacity is increased to C (see Section 4.2 for more de- tails). Thereafter, for each subsequent environment interaction, the latent vectors 2N = (o,) and 2 = (0r+ are computed prior to storing (Zr, ar 2,+1, Tx) in B. During agent updates, the sampled latent vectors are directly passed into the policy Tę and Q-function Qs. bypassing the encoder convolutional layers. Since the agent samples and trains with latent vectors after freezing, we only store the latent vectors and avoid the need to maintain large image observations in B.
4.2 Additional techniques and details for SEER
Data augmentations. Recently, various data augmentations [39, 24, 22] have provided large gains in the sample-effiiiency of RL. from pixel observations. However, SEER precludes data augmentations because we store the latent vector instead of the raw pixel observation. We find that the absence of data augmentations could decrease sample-efficieng, in some cases, e.g., when the capacity of B is small. To mitigate this issue, we perform K number of different data augmentations for each input observation o, and store K distinct latent vectors {z/ = f (AUG,, (or ])kk = 1....A K}. We find empirically that K 4 achieves competitive performance to standard RL algorithms in most cases.
Increasing replay capacity. By storing the latent vector in the replay buffer, we can adaptively increase the capacity (i.e.. total number of transitions), which is determined by the size difference between the input pixel observations and the latent vectors output by the encoder, with a few additional considerations. The new capacity of the replay buffer is
c = [ (TNRK)
where C' is the capacity of the original replay buffer, P is the size of the raw observation, L is the size of the latent vector, and K is the number of data augmentations. The number of encoders and determines the number of distinct latent vectors encountered for each
N is algorithm-specinn observation during training. For Q-learning algorithms N = 1, whereas for actor-critic algorithms N 2 if the actor and critic each compute their own latent vectors. Some algorithms employ a target network for updating the Q-function [30, 10], but we use the same latent vectors for the online and target networks after freezing to avoid storing target latent vectors separately and find that tying their

---

CURL
CURL SEER (ours)
1002
Carosleesngge
1200
Pegreoph
1000
Seachen-aky
1000
600
Chearb--n
400-
402
400
200
200
200
200
1-
5-10
12
2.2-10
3'--1
1,310 5,0H10
445-10
2212 4"30
6x70
1-10
2-10
2410
Figure 2: Learning curves for CURL with and without SEER, where the x-axis shows estimated cumulative FLOPs. The dotted gray line denotes the encoder freezing time t Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
(ours)
ChanyCinterr
1200
200
20.000
450
100
10
000
60
2 FLOPs
25 FLOPs
6-0-10
2+10
6-10
FLOP
FLOPs
40000
5000
Dberi
PladdPurrer
Sencucal
cc0-
2000
4000-
500-
5000
5000
450-
2000
s.c0
30e-
P00
2.5-2015
50-10
2.5-0015
50-1ghh
2550015
5.010"
Figure 3: Learning curves for Rainbow with and without SEER, where the x-axis shows estimated cumulative FLOPs. The dotted gray line denotes the encoder freezing time t Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
parameters does not degrade performance. 1 The factor of 4 arises from the cost of saving floats for latent vectors, while raw pixel observations are saved integer pixel values. We assume the memory required for actions and rewards is small and only consider only the memory used for observations.
Experimental results
We designed our experiments answer the following questions:
Can SEER reduce the computational overhead of various off-policy RL algorithms for both continuous (see Figure 2) and discrete (see Figure 3) control tasks?
Can SEER reduce the memory consumption and improve the sample-efficiency of off-policy RL algorithms by adaptively increasing the buffer size (see Figure and Figure 5)?
Can SEER be useful for compute-efficient transfer learning (see Figure 7a)?
Do CNN encoders of RL agents converge early in training (see Figure 8a and Figure 8b)?
5.1 Setups
Compute-efficienccy We first demonstrate the compute-efficiency of SEER on the DeepMind Control Suite (DMControl: Tassa al. 45) and Atari games [2] benchmarks. DMControl is commonly used for benchmarking sample-efficiency for image-based based continuous control methods. For DMControl experiments, we consider a wfate-o-the--ae model-free RL method, which applies contrastive learning (CURL; Srinivas et al. 39) to SAC [10], using the image encoder architecture from SAC-AE [52].
We remark that the higher layers of the target network not tied to the online network after freezing.
5

---

Scores at 45T FLOPs
Scores 500K environment steps (0.07GB)
Rainbow
Rainbow+SEER
Rainbow
Rainbow
Alien
992.0 + 152.7
1172.6 1239.0
1038.4 + 101.1
1134.6 1452.9
Amidar
144.0 27.4
250.5 147.4
121.0 31.2
165.3 147.6
BankHeist
145.8 61.2
276.6 198.1
161.6 #57.7
151.8 + 65.8
CrazyClimbe
21580.0 3514.6
28066.0 +4108.5
10498.0 + 1387.8
17620.0 +4418.
Krull
2799.5 468.1
3277.5 5 +440.5
2215.7 = 336.9
3069.2 #377.6
Qbert
2325.5 1152.7
4123.5 1385.5
2430.5 + 658.8
#1567.6
RoadRunner
10376.0 +: 2886.0
11794.0 +1745.3
10612.0 2059.3
13064.0 42489.2
Seaquest
402.8 + 48.4
561.2 100.5
262.8 19.1
336.8 +45.9
Table 1: Scores Atari games at 45T FLOPS corresponding Figure 3 and at 500K environment interactions in the constrained-memor setup (0.07GB) corresponding to Figure 4. The results show the mean and standard deviation averaged five runs, and the best results are indicated in bold.
Rrinbow with 3.5GB (upper bound)
Rainhow with 0.07GB
Rainhow SEER with 0.07GB (ours)
Ale
250
Andder
400
Donk bele
CraryCirbbrr
1900
200
300
70,000
1900
200
10000
000
2-107
24107
2430"
2010
24307 2-10"
2-10 34107
1000
4000
in 003
-00
3002-
2000
1800-
200
1500
2-10 SHIPP 1-107 Sesg
1H107 SHIEP
Seng
TH5T SHIST
100
1+
2050" Se1P EHIOP Seld!
BANINGOOS
Figure 4: Comparison of the sample-efficiency of Rainbow with and without SEER in constrained- memory (0.07 GB) settings. The dotted gray line denotes the encoder freezing time = Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
For evaluation, we compare the computational efficiency of CURL with and without SEER by measuring floating point operations (FLOPs). For discrete control tasks from Atari games, we perform similar experiments comparing the FLOPs required by Rainbow [15] with and without SEER. For all experiments, we use the hyperparameters and architecture data-efficient Rainbow [48].
Memory efficiency. We showcase the memory efficiency of SEER with set of constrained-memont experiments in DMControl. For Cartpole and Finger, the memory allocated for storing observations is constrained to 0.03 GB, corresponding to an initial replay buffer capacity C 1000. For Reacher and Walker, the memory constrained to 0.06 GB for an initial capacity of constrained-memen
2000. In this setting, we compare the sample-effiiienc of CURL with and without SEER. As an upper bound, we also report the performance of CURL without memory constraints, i.e., the replay capacity is set to the number of training steps. For Atari experiments, the baseline agent is data-efficient Rainbow and the memory allocation 0.07 GB, corresponding to initial replay capacity C' 10000. The other hyperparameters are the same as those in the compute-efficiency experiments. Before the encoder is freeze, the replay buffer still needs to store the images and the replay buffer slots number is equal with the baseline settings, the performance is equal to the bascline in theory. After the freeze time, the replay buffer slots number grows more larger. So the benefit seems like on the condition of the assumption that larger replay buffer would brings performance improvement? Such assumption needs to be claimed and discussed more clearly in the paper. Further discussions and experiments on the different limitations of the memory cost would be helpful.
We explain our procedure for counting the number of FLOPs in Appendix ??. The gain on wall-clock time discussed in Appendix ??.

---

1000
1300
600
900-
689
400
200
200
COOC COOO
500P
2007
1012
2012"
2012
19527
2007
2017
(a) Cartpole-swingup
(b) Finger-spin
(c) Reacher-easy
400
(d) Walker-walk
(e) Cheetah-run
Figure 5: Comparison of the sample-efficieney of CURL with and without SEER in constrained- memory settings. The dotted gray line denotes the encoder freezing time t t= Ty. The solid line and T=
shaded regions represent the mean and standard deviation, respectively, across five runs.
The encoder architecture used for experiments with CURL is used in Yarats et al. [52]. It consists of four convolutional layers with 3 3 kernels and 32 channels, with the ReLU activation applied after each conv layer. The architecture used for our Rainbow experiments is from van Hasselt et al. [48], consisting of a convolutional layer with 32 channels followed by convolutional layer with 64 channels, both with x kernels and followed by a ReLU activation. For SEER, we freeze the first fully-connected layer in CURL experiments and the last convolutional layer of the encoder in Rainbow experiments. We present the best results across various values of the encoder freezing time Ty. See Appendices ?? and ?? for more hyperparameters and Appendix ?? for source code.
5.2 Improving compute- and memory-efficiency
Experimental results in DMControl and Atari showcasing the computational efficiency of SEER are provided Figures and Figure CURL and Rainbow both achieve higher performance within significantly fewer FLOPS when combined with SEER in DMControl and Atari, respectively. Additionally, Table compares the performance of Rainbow with and without SEER at 45T (4.5e13) FLOPs. In particular, the average returns are improved from 145.8 to 276.6 compared baseline Rainbow in BankHeist and from 2325.5 to 4123.5 in Qbert. We remark that SEER achieves better computational efficiency while maintaining the agent's final performance and comparable sample- efficiency (see Appendix ?? for corresponding figures).
Experimental results in Atari and DMControl showcasing the sample-efficiency of SEER in the constrained -etetraiinnd-mmmryyy setup are provided Figure and Figure 5. CURL and Rainbow achieve higher final performance and better sample-efficiency when combined with SEER in DMControl and Atari, respectively. Additionally, Table compares the performance of unbounded memory Rainbow and constrained-memory (0.07 GB) Rainbow with and without SEER at 500K interactions. In particular, the average returns are improved from 10498.0 to 17620.0 compared to baseline Rainbow in CrazyClimber and from 2430.5 to 3231.0 0 in Qbert. Although and memory benefits of SEER in these experiments, we also highlight the computational gain of SEER in constrained-memoryy settings (effectively combining the benefits) in Appendix ??. For an ablation on the freezing time, see Appendix ??. These experimental results show the real-world applicability of SEER (see Appendix ?? for more details).
disentangle the computational
5.3 Freezing larger convolutional encoders
We also verify the benefits of SEER using deeper convolutional encoders, which are widely used in range of applications such as visual navigation tasks and favored for their superior generalization
7

---

QURL
PL.0ss
FLOPs
(a) Cartpole-swingup
(b) Walker-walk
Figure 6: Learning curves using IMPALA architecture, where the x-axis shows estimated cumulative FLOPs. The dotted gray line denotes the encoder freezing time = T. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
ability. Specifically, we follow the setup described in Section 5.1 and replace the SAC-AE architecture (4 convolutional layers) with the IMPALA architecture [6] (15 convolutional layers containing residual blocks [14]). Figure 5.2 shows the computational efficiency of SEER in Cartpole-swingee walk with the IMPALA architecture. CURL achieves higher performance within significantly fewer FLOPs when combined with SEER. We remark that the gains due to SEER are more significant because computing and updating gradients for large convolutional encoders is very computationally expensive.
and Walker-
5.4 Improving compute-efficiency in transfer settings
We demonstrate, as another application of our method, that SEER increases compute-efficiency in the transfer setting: utilizing the parameters from Task on unseen Tasks B. Specifically, we train a CURL agent for 60K environment interac- tions on Walker-stand; then, we only fine-tune the policy and Q-functions on unseen tasks using net- work parameters from Walker-stand. To save com- putation, during fine-tuning, we freeze the encoder parameters. Figure shows the computational gain of SEER in task transfer (i.e., Walker-stand to Walker-wall similar to Yarats et al. [52]), and domain transfer (i.e., Walker-stand to Hopper-hop) shown in Figure 7b. Due to the generality CNN features, we can achieve this computational gain using pretrained encoder. For the task transfer setup, we provide more analysis on the number of frozen layers and freezing time hyperparameter Ty in Appendix ??. While these transfer learning experiments compute-efficiengy experiments in Section 5.2, we believe this is an exciting additional application of SEER and that more comprehensive investigations in this direction would be interesting future work.
R.
P.ORs
(a) To Walker-walk
(b) To Hopper-hop
Figure 7: Comparison of the computational efficiency of agents trained from scratch with CURL and agents trained with CURL+SEER from Walker-stand pretraining. The solid line and shaded regions represent the mean and stan- dard deviation, respectively, across three runs.
relatively independent to the
5.5 Encoder analysis
In this subsection we present visualizations to verify that the neural networks employed in deep reinforcement learning indeed converge from the bottom up, similar those used in supervised learning [36]. Figure 8a shows the spatial attention map for two Atari games and one DMControl environment various points during training. Similar to Laskin et al. [24] and Zagoruyko & Komodakis [54], we compute the spatial attention map by mecn--ooling the absolute values of the activations along the channel dimension and follow with 2-dimensional spatial softmax. The attention map shows significant change in the first 20% of training, and remains relatively unchanged thereafter, suggesting that the encoder converges to final representations early in training. Figure 8b shows the SVCCA [36] score, a measure of neural network layer similarity, between a layer and itself at time / and + 10K. The convolutional layers of the encoder achieve high similarity scores with themselves between time i and / + 10K, while the higher layers of the policy and Q-network continue to change throughout training. In our DMControl environments we freeze the convolutional layers and the first fully-connected layer of the policy and Q-network (denoted fc fc... Although the

---

policy fcl continues to change, the convergence of the Q-network fcl and the encoder layers allow us achieve our computational and memory savings with minimal performance degradation.
We remark that while the encoder can be frozen early in RL. training, using a randomly initialized encoder is ineffective [41]. is important train encoders the task in order to learn useful features (as is done by widely used methods such as Srinivas et al. [39] and Laskin et al. [24]), but our finding that these encoders converge early task-specific training.
Alien
Amidar
Cartpole-
swingup
Otsenation
0scaning
cuiniing
50% mriningg roiving
CNN encoder
Policy
Qunetwork
SVEGA
High ceviiiiiiityy
(a) Spatial attention map
(b) SVCCA similarity scores
Figure 8: Visualizations of encoder features throughout training. (a) Spatial attention map from CNN encoders. (b) SVCCA [36] similarity scores between each layer and itself at time t and + 10K throughout training for Walker-walk task.
6 Discussion and Limitations
In this paper, proposed technique that reduces computation requirements for visual reinforcement learning, which we hope serves to facilitate a shift toward more compute-efficien RL. Here, we highlight other techniques for reducing training time. For experimentation in computationally intensive environments, Obando-Ceron & Castro [32] propose to use small- and medium-scale experiments, which could reproduce the conclusions of the Rainbow DQN paper in Atari games. For faster training time in a particular experiment, one can also lower the resolution of the input images. Figures 9a and 9b show that reducing the resolution by factor 2f from 100 100 to 50 x 50 (and scaling crops appropriately) produces significant compute-efficieet Control Suite without sacrificing performance, and emphasize that this technique can be combined with SEER for further improved efficiency. We remark that the additional gain from SEER is larger more complex environments (e.g., Walker) where learning requires more steps. However, we find that naive resolution reduction may not generally be applicable across environments and may require domain knowledge order to prevent excessive information loss. In Figures 9c and 9d show that resolution reduction by factor of 2, from 84 x 84 to 42 42, results in noticeably worse performance in several Atari games. contrast, SEER successfully improves compute-efficiency without sacrificing performance in these games (see Figure 3). Overall, SEER is highly generalizable across visual domains, and can be easily combined with other modifications.
gain in DeepMind
A limitation of our work is the introduction of hyperparameter for the freezing time t. While domain knowledge can be used to decide a reasonable range for and reduce the search space, interesting future direction would be to adaptively determine the freezing time using metric of convergence. We also do not show the application of SEER tasks which are more computationally expensive or even infeasible. We evaluate our method in DM Control and Atari because they are common RL benchmarks used many recent works on RL from pixels, but the full impact of SEER may be more easily seen in very visually complex and challenging tasks such as 3D navigation. We do foresee any negative societal impacts of our work, as it simply reduces training time of already existing algorithms.

---

221
C.lk Bhi0age 1881
RLOPS
(a) Caripole-swingup
(b) Walker-walk
(c) Alien
(d) Amidar
Figure 9: Evaluation of the compute-cfficieccy of CURL ((a) and (b)) and Rainbow ((c) and (d) with original and reduced (by factor of 2) resolutions. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
7 Conclusion
presented SEER, a simple but powerful modification of off-policy RL algorithms that significantly reduces computation and memory requirements while maintaining state-of-the-art performance. We leveraged the intuition that CNN encoders in deep RL converge their final representations early in training to freeze the encoder and subsequently store latent vectors to save computation and memory. In our experimental results, we demonstrated the compute- and memory efficciccf of SEER in various DMControl environments and Atari games, and proposed technique for compute-efficient transfer learning. With SEER, we highlight the potential for improvements in compute- and memory- efficiency in deep RL that can be made without sacrificing performance, in hopes of making deep RL more practical and accessible in the real world.
Acknowledgements
This research is supported in part by Open Philanthropy, ONR PECASE N000141612723, NSF NRI #2024675, and Berkeley Deep Drive. We would like to thank Kourosh Hakhamaneshi, Fangchen Liu, and anonymous reviewers for providing helpful feedback and suggestions. We would also like to thank Denis Yarats for the IMPALA encoder architecture implementation and Kai Arulkumaran for help with modifying the Rainbow DQN codebase.
References
[1] Badia, Adrià Puigdomènech, Piot, Bilal, Kapturowski, Steven, Sprechmann, Pablo, Vitvitskyi, Alex, Guo, Daniel, and Blundell, Charles. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, 2020.
[2] Bellemare, Mare G, Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[3] Bellemare, Marc G. Dabney, Will, and Munos, Rémi. A distributional perspective on reinforce- ment learning. International Conference Machine Learning, 2017.
[4] Blalock, Davis, Ortiz, Jose Javier Gonzalez, Frankle, Jonathan, and Guttag, John. What the state of neural network pruning? arXiv preprint arXiv:2003.03033 2020.
[5] Brock, Andrew, Lim, Theodore, Ritchie, James M, and Weston, Nick. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint (oXXiv/706.0444 2017.
[6] Espeholt, Lasse, Soyer, Hubert, Munos, Remi, Simonyan, Karen, Mnih, Volodymir, Ward, Tom, Doron, Yotam, Firoiu, Vlad, Harley, Tim, Dunning. Iain, Legg. Shane, and Kavukcuoglu, Koray. Impala: Scalable distributed deep-rl with importance weighted actor-lcarner architectures, 2018.
[7] Fang, Kuan, Toshev, Alexander, Fei-Fei, Li, and Savarese, Silvio. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 538-547, 2019.
10

---

[8] Frankle, Jonathan and Carbin, Michael. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[9] Ha, David and Schmidhuber, Jürgen. World models. arXiv preprint arXiv:1803. 2018.
[10] Haarnoja, Tuomas, Zhou, Aurick, Abbeel, Pieter, and Levine, Sergey. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning. 2018.
[11] Hafner, Danijar, Lillicrap, Timothy, Fischer, Ian, Villegas, Ruben, Ha, David, Lee, Honglak, and Davidson, James, Learning latent dynamics for planning from pixels. In International Conference Muchine Learning, 2019.
[12] Hafner, Danijar, Lillicrap. Timothy, Ba, Jimmy, and Norouzi, Mohammad. Dream to con- trol: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
[13] Han, Song, Mao, Huizi, and Dally, William J. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 2015.
[14] He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing. and Sun, Jian. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
[15] Hessel, Matteo, Modayil, Joseph, Van Hasselt, Hado, Schaul, Tom, Ostrovski, Georg, Dabney, Will, Horgan, Dan, Piot, Bilal, Azar, Mohammad, and Silver, David. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
[16] Higgins, Irina, Pal. Arka, Rusu, Andrei A, Matthey, Loic, Burgess, Christopher P. Pritzel, Alexander, Botvinick, Matthew, Blundell, Charles, and Lerehner, Alexander. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning. 2017.
[17] landola, Forrest N, Han, Song, Moskewicz, Matthew W, Ashraf, Khalid, Dally, William J, and Keutzer, Kurt. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.073660 2016.
[18] Jaderberg, Max, Mnih, Volodymyr, Czarnecki, Wojeiech Marian, Schaul, Tom, Leibo, Joel Z, Silver, David, and Kavukcuoglu, Koray. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
[19] Kaelbling, Leslie Pack, Littman, Michael L, and Cassandra, Anthony R. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
[20] Kaiser, Lukasz, Babaeizadeh, Mohammad, Milos, Piotr, Osinski, Blazej, Camphell, Roy H, Czechowski, Konrad, Erhan, Dumitru, Finn, Chelsea, Kozakowski, Piotr, Levine, Sergey, al. Model-based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
[21] Kalashnikov. Dmitry, Irpan, Alex, Pastor, Peter, Ibarz, Julian, Herzog, Alexander, Jang, Eric, Quillen, Deirdre, Holly, Ethan, Kalakrishnan, Mrinal, Vanhoucke, Vincent, al. Qt-opt: Scalable deep reinforcement learning for vision-bas based robotic manipulation. In Conference on Robot Learning, 2018
[22] Kostrikov, Ilya, Yarats, Denis, and Fergus, Rob. Image augmentation all you need: Regulariz- ing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[23] Lake, Brenden M. Ullman, Tomer D, Tenenbaum, Joshua B, and Gershman, Samuel J. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[24] Laskin, Michael, Lee, Kimin, Stooke, Adam, Pinto, Lerrel, Abbeel, Pieter, and Srinivas, Aravind. Reinforcement learning with augmented data. In Advances neural information processing systems, 2020.
11

---

[25] LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-222) 1998.
[26] Lee, Kimin, Laskin, Michael, Srinivas, Aravind, and Abbeel, Picter. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04933. 2020.
[27] Lillicrap. Timothy Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[28] Lin, Long-Ji. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4)2933-31, 1992.
[29] Mattson, Peter, Cheng, Christine, Coleman, Cody, Diamos, Greg, Micikevicius, Paulius, Patter- son, David, Tang, Hanlin, Wei, Gu- Bailis, Peter, Bittorf, Victor, et al. MIperf training benchmark. In Conference on Machine Learning and Systems, 2020.
[30] Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Mare G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[31] Morcos, Ari, Raghu, Maithra, and Bengio, Samy. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, 2018.
[32] Obando-Ceron, Johan and Castro, Pablo Samuel. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. arXiv preprint arXiv:2011.14826, 2020.
[33] Oord, Aaron van den, Li, Yazhe, and Vinyals, Oriol. Representation learning with contrastive predictive coding, arXiv preprint 1807v:1807 2018
[34] Osband, Ian, Blundell, Charles, Pritzel, Alexander, and Van Roy, Benjamin. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, 2016.
[35] Pellegrini, Lorenzo, Graffieti, Gabrile, Lomonaco, Vincenzo, and Maltoni. Davide. Latent replay for real-time continual learning. arXiv preprint arXiv:1912.01100, 2019.
[36] Raghu, Maithra, Gilmer, Justin, Yosinski, Jason, and Sohl-Dickstein, Jascha. Sveca: Singular vector canonical correlation analysis for deep understanding and improvement. In Advances in neural information processing systems, 2017.
[37] Schrittwieser, Julian, Antonoglou, loannis, Hubert, Thomas, Simonyan, Karen, Sifre, Lau- rent, Schmitt, Simon, Guez, Arthur, Lockhart, Edward, Hassabis, Demis, Graepel, Thore, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.0826. 2019.
[38] Schulman, John, Chen, Xi, and Abbeel, Pieter. Equivalence between policy gradients and soft q-leamnig. arXiv preprint arviv: / /704.06440 2017.
[39] Srinivas, Aravind, Laskin, Michael, and Abbeel, Pieter. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
[40] Stooke, Adam, Lee, Kimin, Abbeel, Pieter, and Laskin, Michael. Decoupling representation learning from reinforcement learning, 2020.
[41] Stooke, Adam, Lee, Kimin, Abbeel, Pieter, and Laskin, Michael. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
[42] Sun, Zhiqing, Yu, Hongkun, Song, Xiaodan, Liu, Renjie, Yang, Yiming, and Zhou, Denny Mobilebert: compact task-agnostie bert for resource-limited devices. In Annual Meeting of the Association for Computational Linguistics, 2020.
12

---

[43] Sutton, Richard S. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
[44] Sutton, Richard S and Barto, Andrew Reinforcement learning: An introduction. MIT Press, 2018.
[45] Tassa, Yuval, Doron, Yotam, Muldal, Alistair, Erez, Tom, Li, Yazhe, Casas, Diego de Las, Budden, David, Abdolmaleki, Abbas, Merel, Josh, Lefrancq, Andrew, et al. Deepmind control suite. arXiv preprint arXív: 1001:0000000009000 2018.
[46] Tay, Yi, Zhang, Aston, Tuan, Luu Anh, Rao, Jinfeng, Zhang, Shuai, Wang, Shuohang, Fu, Jie, and Hui, Siu Cheung. Lightweight and efficient neural natural language processing with quaternion networks. In Annual Meeting of the Association for Computational Linguistics, 2019.
[47] Van Hasselt, Hado, Guez, Arthur, and Silver, David. Deep reinforcement learning with double q q-learning. In AAA/ Conference on Artificial Intelligence, 2016.
[48] van Hasselt, Hado P. Hessel, Matteo, and Aslanides, John. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, pp. 14322- 14333, 2019.
[49] Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N, Kaiser, tukasz, and Polosukhin, Illia. Attention is all you need. In Advances neural information processing systems, 2017.
[50] Vinyals, Oriol, Babuschkin, Igor, Czarnecki, Wojeiech M, Mathieu, Michaël, Dudzik, Andrew, Chung, Junyoung, Choi, David H, Powell, Richard, Ewalds, Timo, Georgiev, Petko, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019.
[51] Watkins, Christopher JCH and Dayan, Peter. Q-learning. Machine learning, 8(3-4):279-292. 1992.
[52] Yarats, Denis, Zhang, Amy, Kostrikov, Ilya, Amos, Brandon, Pineau, Joelle, and Fergus, Rob. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1rXiv:1910.01741, 2019.
[53] Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod. How transferable are features in deep neural networks? In Advances in neural information processing systems, 2014.
[54) Zagoruyko, Sergey and Komodakis, Nikos. Paying more attention to attention: Improving the performance convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017.
[55] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
13


Reviews:
[{"review_id": "a1UAxxIMyM0", "reviewer": "Reviewer_1FST", "summary": "The paper propose a compute and memory efficient pixel wise RL algorithm by early stoping the update of CNN encoder and store the latent vector to the replay buffer instead of the images. Experiments had been done on DM Control and Atari with baseline algorithm CURL.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths,\n\nThe paper is well organized and easy to follow.\nThe code, data, and instructions needed to reproduce the main experimental results is provided.\nThe compute cost is lower than baselines.\nThe memory cost is smaller.\n\n\nConcerns and Questions,\n\n1. The computation and the memory cost is not the very important issue in RL. The most concerns are the sample efficiency, since interact with the environment usually cost benefit loss or even damage accidents during the exploration. So although the result showed with FLOPs significantly outperformed the baseline, the result is not quit surprising, since the SEER no longer needs to compute the gradient of the encoder after freeze time. And the memory is not the concerns also, since the size of replay buffer is not the larger the better as discussed in “A Deeper Look at Experience Replay” and some other researches. \n\n2. Missed some related works that improves the sample efficiency in pixels state reinforcement learning. To name a few, PISAC, DRQ, SLAC, the performances of them are better than CURL. I was wondering whether freeze it at a previous time harms the performance of those algorithms. Even with CURL, as shown in Fig. 12, before the freeze line the performance is quite similar, but the performance drops after the freeze time in most task. \n\n3. It is hard to choice the time to freeze the encoder in practice. Seems like it needs more repetitions of the experiments to give a good choice of the freeze time? More experiments show the sensitive to the time of freeze of the algorithm would be better.\n\n\n\nSuggestions and Discussions,\n1. Before the encoder is freeze, the replay buffer still needs to store the images and if the replay buffer slots number is equal with the baseline settings, the performance is equal to the baseline in theory. After the freeze time, the replay buffer slots number grows more larger. So the benefit is seems like on the condition of the assumption that a larger replay buffer would brings performance improvement? Such assumption needs to be claimed and discussed more clearly in the paper. Further discussions and experiments on the different limitations of the memory cost would be helpful.\n\n2. The part of transfer learning (Sec 5.4) is not quit clear to me. It seems like transfer learning is kind of a relatively independent experiment of what want to claim in this paper. If the paper want to claim that the SEER can bring benefit to transfer learning, more experiments has to be done to justify such benefit. If not, I think the paper without transfer learning experiment is enough for what the paper want to propose.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SgPkkVHJwc", "reviewer": "Reviewer_qa64", "summary": "Stored Embeddings for Efficient Reinforcement learning (SEER) is a method to reduce the computational cost of value-based off-policy reinforcement learning, e.g. Q-learning, in settings with pixel-based observations. It uses a convolutional encoder whose lower layers are frozen early in agent training, as there has been previous work showing that a convolutional network's lower layers converge earlier on in training. Following an early period of joint training with the convolutional encoder, the agent afterwards stores only the encoded image features into its replay buffer instead of the raw pixels of the image. This enables potentially significant memory and compute savings as the image no longer needs to be re-encoded during off-policy learning. The paper presents empirical results demonstrating that final agent performance is not hindered by this procedure and that, when counting in terms of total FLOPs calculated or replay buffer size, SEER has an advantage over the naive image-based replay buffers most commonly used. A variety of experimental settings are considered, differing across domains (DMControl, Atari) and training procedure (Rainbow, SAC, CURL). ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method, SEER, can be seen as an application of Freezeout to value-based RL, combined with storing embeddings instead of images in the replay buffer after the lower layers are frozen. The method seems well-motivated and straightforward, leveraging the fact that the convolutional encoder trains quickly to avoid the need to re-encode images several times during learning. Considering that reinforcement learning is a considerably different setting to supervised learning, in that the input data distribution can change significantly over the lifetime of the agent, it was not immediately obvious that Freezeout can be applied without issues in this setting.\n\nHowever, the authors have provided experimental results on pixel-based DMControl and Atari that seem to validate that freezing the lower convolutional layers early-on in training does not cause decreased performance in those domains. But there are still some remaining concerns I have regarding the generality of the proposed method that result in my score of weak reject:\n\n1) A lot of the environments the method was evaluated on are highly regular in visual appearance, even throughout policy performance levels. I checked videos of the various Atari 2600 games, and only Krull has non-static backgrounds and a larger variety of visual representations for entities. The DMControl tasks are also very static across agent performance levels. I think it is important to test how this method would fare when there exists significant distribution shifts during training, e.g. an important new entity or visual mechanic appears at a certain policy performance. Would freezing the network before this critical performance level still make later policy learning improve?\n2) Even not considering shifts in distribution, the visual understanding necessary for high performance in the environments tested is limited. As mentioned in the paper's section 6, it would be interesting to test SEER on more complex visual environments which require larger image encoders to extract useful semantics.\n3) Maybe owing to the visual simplicity of the environments, the ablations reveal that the convolutional layers can be frozen early with minimal reduction in final performance. Have the authors tried using features from a randomly initialized network as a baseline.\n4) What is the effect on wall-clock time of training an agent using SEER? What would the graphs look like if the x-axis was experiment run-time and all agents were trained on the same reference machine?\n\n==========================================================================\n\nThanks for the response, the experimental results you described have addressed most of the concerns I had above. I think those results should be included in the camera-ready paper as an appendix section. I will raise my score to a 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations of their method in section 6, mentioning the additional freezing time hyperparameter introduced by SEER.\n\nThere does not seem to be a potential for negative societal impact for this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "EWLd8TSwWtM", "reviewer": "Reviewer_KTu3", "summary": "- This paper presents a method for compute- and memory-efficient reinforcement learning where the visual encoder is frozen partway into training.  After freezing, latent vectors are stored in the replay buffer instead of images (and any existing images are replaced by them).  This leads to both better compute and memory utilization.\n- The authors demonstrate their method by comparing to Rainbow on Atari and CURL on DM Control.  On DM control, their method reduces the compute by a considerable margin.  On Atari, the results are less clear cut, but the compute cost is reduced.\n- When they also impose a memory constraint, the effectiveness of their method is further increased.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength\n    - Elegant and \"obvious in hindsight\" (a good thing) idea, meaning it will likely have broad applicability\n    - While the authors only tested it on off-policy methods, it is clearly also applicable to on-policy methods that use a rollout storage (PPO, PPG, IMPALA, V-MPO, A2C, etc.)\n    - Good FLOPs vs. Reward results on DM Control\n    - Results are more influential with a deeper CNN\n- Weaknesses\n    - The memory constrained results seem very contrived.  60 MB is a tiny amount of memory and even 9.0 GB of (presumably) CPU memory isn't that prohibitive.\n        - Perhaps if wall-clock time was plotted in addition to samples, the smaller memory footprint of SEER would mean the replay buffer can be stored on the GPU and training would be much faster since many expensive CPU -> GPU transfers would be eliminated?\n    - The CNN is frozen all at once instead of frozen iteratively. Raghu 2017 and Figure 6c suggest that the early layers could be frozen much earlier, although this may increase the memory usage initially since CNNs typically increase the memory size of the feature map in lower layers.\n- Overall\n    - This is a useful idea that improves experiment time, giving researchers a faster iteration cycle.  The idea is quite generic so it should be easy for researchers to adopt.\n- References\n    - Smith 2017: https://arxiv.org/abs/1711.00489\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: An interesting approach to addressing the problem of dealing with vision input in the reinforcement learning setting. Presented approach is more computationally efficient than its counterparts. The algorithm is also flexible enough and can be applied in the on-policy context, even though in the paper the authors focus on the off-policy setting. Well written paper. All the reviewers see the importance of the proposed method in dealing with the notoriously difficult problem of computationally-efficient vision-based RL.
Decision: Accept (Poster)