Compute- and Memory-Efficient Reinforcement Learning with Latent Experience Replay
Authors: Lili Chen
Kimin Lee
Aravind Srinivas
Pieter Abbeel
Venue:  (2021)
Keywords: reinforcement learning
deep learning
computational efficiency
memory efficiency

Abstract: Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Latent Vector Experience Replay (LeVER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements without sacrificing the performance of RL agents. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that LeVER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games. Finally, we show that LeVER is useful for computation-efficient transfer learning in RL because lower layers of CNNs extract generalizable features, which can be used for different tasks and domains.

Under review as a conference paper at ICLR 2021
COMPUTE- AND MEMORY-EFFICIENT REINFORCE- MENT LEARNING WITH LATENT EXPERIENCE REPLAY
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in off- off-policy deep reinforcement learning (RL) have led to im- pressive success in complex tasks from visual observations, Experience replay im- proves sample-effciincy by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this pa- per, present Latent Vector Experience Replay (LeVER), simple modification existing off-policy RL methods, to address these computational and memory requirements without sacrificing the performance of RL agents. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional la- vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained- memory settings. In our experiments, we show that LeVER does not degrade the performance of RL agents while significantly saving computation and memory across diverse set of DeepMind Control environments and Atari games. Finally, we show that LeVER is useful for computation-efficien transfer learning in RL because lower layers CNNs extract generalizable features, which can used for different tasks and domains.
INTRODUCTION
Success stories of deep reinforcement learning (RL) from high dimensional inputs such as pixels or large spatial layouts include achieving superhuman performance on Atari games (Mnih et al., 2015;
Schrittwieser al., 2019; Badia et al., 2020), grandmaster level in Starcraft (Vinyals et al., 2019) and grasping a diverse set of objects with impressive success rates and generalization with robots in the real world (Kalashnikov et al., 2018). Modern off-policy RL algorithms (Mnih et al., 2015;
Hessel et al., 2018; Hafner et al., 2019; 2020; Srinivas et al., 2020; Kostrikov et al., 2020; Laskin et al., 2020) have improved the sample-efficieng of agents that process high-dimensional pixel inputs with convolutional neural networks (CNNs; LeCun et al. 1998) using the past experiential data that is typically stored as raw observations form in a replay buffer (Lin, 1992). However, these methods demand high memory and computational bandwidth, which makes deep RL inaccessible in several scenarios, such as learning with much lighter on-device computation (e.g. mobile phones or other light-weight edge devices).
For compute- and memory-efficient deep learning, several strategies, such as network pruning (Han al., 2015; Frankle & Carbin, 2019), quantization (Han et al., 2015; landola al., 2016) and freezing (Yosinski et al., 2014; Raghu et al., 2017) have been proposed in supervised learning and unsupervised learning for various purposes (see Section 2 for more details). In computer vision, Raghu et al. (2017) showed that the computational cost of updating CNNs can be reduced by freez- ing lower layers earlier in training, and Han et al. (2015) introduced a deep compression, which reduces the memory requirement of neural networks by producing sparse network. In natural lan- guage processing, several approaches (Tay et al., 2019; Sun et al., 2020) have studied improving the computational efficiency of Transformers (Vaswani et al., 2017). In deep RL, however, devel- oping compute- and memory-efie memory-efficient techniques has received relatively little attention despite their serious impact the practicality of RI. algorithms.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose Latent Vector Experience Replay (LeVER), simple technique to reduce computational overhead and memory requirements that is compatible with various pollyy RL al- gorithms (Haarnoja al., 2018: Hessel et al., 2018; Srinivas al., 2020). Our main idea is to freeze the lower layers of CNN encoders of RI. agents early in training, which enables two key capabil- ities: (a) compute-efficieney: reducing the computational overhead of gradient updates in CNNs; (b) memory-efficieney; saving memory by storing the low-dimensional latent vectors to experience replay instead of high-dimensional images. Additionally, we leverage the memory-efficient of LeVER to adaptively increase the replay capacity, resulting in improved sample-efficieng of off- constrained-memor settings. LeVER achieves these improvements without sacrificing the performance of RL agents due to early convergence of CNN encoders.
policy RL algorithms
To summarize, the main contributions of this paper are as follows:
We present I LeVER, compute- and memory-efficient technique that can be used in conjunction with most modern off-policy RI. algorithms (Haarnoja et al., 2018; Hessel et al., 2018).
We show that LeVER significantly reduces computation while matching the original perfor- mance of existing RI algorithms on both continuous control tasks from DeepMind Control Suite (Tassa et al., 2018) and discrete control tasks from Atari games (Bellemare et al., 2013).
We show that LeVER improves the sample-efficenncy of RI. agents in constrained-memory set- tings by enabling an increased replay buffer capacity.
Finally, we show that LeVER is useful for computation-efficien transfer learning, highlighting the generality and transfcrability of encoder features.
2 RELATED WORK
Off-policy deep reinforcement learning. The most sample-efficient RL agents often use off-policy RL algorithms, recipe for improving the agent's policy from experiences that may have been recorded with a different policy (Sutton & Barto, 2018). Off-policy RL algorithms are typically based Q-Learning (Watkins & Dayan, 1992) which estimates the optimal value functions for the task at hand, while actor-critic based off-policy methods (Lillicrap et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018) are also commonly used. In this paper will consider Deep Q- Networks (DQN; Mnih et al. (015).which combine the function approximation capability of deep convolutional neural networks (CNNs; LeCun et al. 1998) with Q-Learning along with the usage of the experience replay buffer (Lin, 1992) as well as off-policy actor-critic methods (Lillicrap et al., 2016; Haarnoja al., 2018), which have been proposed for continuous control tasks.
Taking into account the learning ability of humans and practical limitations of wall clock time for deploying RL algorithms in the real world, particularly those that learn from raw high dimensional inputs such as pixels (Kalashnikov et al., 2018), the sample-inefficiency of off-policy RL algo- rithms has been a research topic of wide interest and importance (Lake et al., 2017: Kaiser et al., 2020). To address this, several improvements in pixel-based off- off-policy RL have been proposed recently: algorithmic improvements such as Rainbow (Hessel et al., 2018) and its data-efficient ver- sions (van Hasselt et al., 2019); using ensemble approaches based bootstrapping (Osband al., 2016; Lee et al., 2020); combining RL algorithms with auxiliary predictive, reconstruction and con- trastive losses (Jaderberg et al., 2017: Higgins al., 2017; Oord et al., 2018; Yarats et al., 2019; Srinivas et al., 2020; Stooke et al., 2020); using world models for auxiliary losses and/or synthetic rollouts (Sutton, 1991; Ha & Schmidhuber, 2018; Kaiser al., 2020; Hafner al., 2020); using data-augmenttaionns on images to improve sample-efficiency (Laskin et al., 2020; Kostrikov et al., 2020).
Campute-efficienn techniques in machine learning. Most recent progress in deep learning and RL has relied heavily on the increased access to more powerful computational resources. To address this, Mattson et al. (2020) presented MLPerf, a fair and precise MI. benchmark to evaluate model training time on standard datasets, driving scalability alongside performance, following a recent focus on mitigating the computational cost of training ML models. Several techniques, such pruning and quantization (Han et al., 2015; Frankle & Carbin, 2019; 9; Blalock et al., 2020; landola et al., 2016; Tay et al., 2019) have been developed to address compute and memory requirements. Raghu et al. (2017) proposed freezing earlier layers to remove computationally expensive backward passes in supervised learning tasks, motivated by the bottom-up convergence of neural networks. This intuition was
2

---

Under review as a conference paper at ICLR 2021
further extended to recurrent neural networks (Morcos et al., 2018) and continual learning (Pellegrini et al., 2019), and Yosinski et al. (2014) study the transferability of frozen and fine-tuned CNN parameters. Fang et al. (2019) store low-dimensional embeddings of imput observations in scene memory for long-horizon tasks. We focus on the feasibility of freezing neural network layers in deep RI and show that this idea can improve the compute- and memory-effciieney of many off- policy algorithms using standard RL benchmarks.
3 BACKGROUND
We formulate visual control task as a partially observable Markov decision process (POMDP; Sut- ton & Barto 2018; Kaelbling al. 1998). Formally, at each timestep t, the agent receives high- dimensional observation 0N, which an indirect representation of the state 8t, and chooses an action a, based on its policy The environment returns a reward T2 and the agent transitions to the next observation 0++1- The return R2 X o F is the total accumulated rewards from timestep I with discount factor y E [0, 1). The goal of RL return over trajectories. By following the common practice in DQN (Mnih et al., 2015), we han- dle the partial observability environment using stacked input observations, which are processed through the convolutional layers of an encoder fé.
learn policy æ that maximizes the expected
Soft Actor-Critic. SAC (Haarnoja et al., 2018) is an off-policy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which encourages the robustness to noise and exploration by maximizing a weighted objective the reward and the policy entropy. To update the parameters, SAC alternates between soft policy evaluation and soft policy improvement. At the soft policy evaluation step, a soft Q-function, which modeled neural network with parameters 0, is updated by minimizing the following soft Bellman residual:
L (0, 1b) = Er,-5 (QolFo(or)) ar) - Tt
(o(()))); a log n T (0((+11
where 7. = (or, Tr, 0t4 a transition, B is replay buffer, 0, 1 are the delayed parameters, and a a temperature parameter. At the soft policy improvement step, the policy π with its parameter is updated by minimizing the following objective:
SAC (Ø) = Eor [a
[a log no (a - Qo(fe(o)), at)].
(1)
Here, the policy is modeled as a Gaussian with mean and covariance given by neural networks to handle continuous action spaces.
Deep Q-learning. DQN algorithm (Mnih et al., 2015) learns a Q-function, which is modeled as neural network with parameters 8, by minimizing the following Bellman residual:
cn (0, = EET (QolFolo)) or) - max Qo Qol(p(ott)).a))))
(2)
where Tj = (or, aq Tt, 0c+1) a transition, is a replay buffer, and 0, & are the delayed parameters. Rainbow DON integrates several techniques, such as double Q-learning (Van Hasselt et al., 2016) and distributional DQN (Bellemare et al., 2017). For exposition, we refer the reader to Hessel et al. (2018) for more detailed explanations of Rainbow DQN.
4 LEVER: LATENT VECTOR EXPERIENCE REPLAY
In this section, we present LeVER: Latent Vector Experience Replay, which can be used in conjunc- tion with most modern off-policy RL algorithms, such as SAC (Haarnoja et al., 2018) and Rainbow DQN (Hessel et al., 2018). Our main idea is to freeze lower layers during training and only update higher layers, which eliminates the computational overhead of computing gradients and updating in lower layers. We additionally improve the memory -effnoiceffcccnncy of off- off-policy RL algorithms by storing diweennniionll latent vectors in the replay buffer instead of high-dimensional pixel observations. See Figure I and Appendix A for more details of our method.
3

---

Under review as a conference paper at ICLR 2021
Forward Backward
Actor
Critic
Forward Backward
Actor
Critic
Latent
Latent vector
(dim=50)
(dim=50)
Encoder
Encoder
Replay buffer
x 84 84
Replay buffer
(a) VeVER before freezing.
(b) LeVER after freezing.
Figure I: Illustration of our framework. (a) Before the encoder frozen, all forward and backward passes are active through the network, and we store images in the replay buffer. (b) After freezing, we store latent vectors in the replay buffer, and remove all forward and backward passes through the encoder. We remark that more samples can be stored in the replay buffer due to the relatively low dimensionality of the latent vector.
4.1 FREEZING ENCODER FOR SAVING COMPUTATION AND MEMORY
We process high-dimensional image input with encoder Sa obtain 2; (or) which is used as input for policy Te and Q-function Qa as described in Section 3. In off-policy RL, we store tran- sitions (0t, at, Tt) in the replay buffer B improve sample-efficincc by reusing experience from the past. However, processing high-dimensional image input o, is computationally expensive. To handle this issue, after T updates, we freeze the parameters encoder v and only update the policy and Q-function. We remark that this simple technique can save computation without perfor- mance degradation because the encoder is modeled as deep convolutional neural networks, while shallow MLP is used for policy and Q-function. Freezing lower layers of neural networks also has been investigated in supervised learning based on the observation that neural networks converge to their final representations from the bottom-up, i.c.. lower layers converge very carly in train- ing (Raghu et al., 7). For the first time, we show the feasibility and effectiveness of this idea in pixel-based reinforcement learning (see Figure 7(a) for supporting experimental results) and present solutions to its RL-specific implementation challenges.
Moreover, in order to memory, consider storing (compressed) latent vectors instead of high-dimensional image inputs. Specifically, each experience in B is replaced by the latent tran- sition (zt, at, Tt), and the replay capacity is increased to Ć (see Section 4.2 for more de- tails). Thereafter, for each subsequent environment interaction, the latent vectors 2N = (o.) and 1 10 are computed prior to storing (zr, Ge, R T2) in B. During agent updates, the sampled latent vectors are directly passed into the policy T and Q-function Qs. bypassing the en- coder convolutional layers. Since the agent samples and trains with latent vectors after freezing, we only store the latent vectors and avoid the need to maintain large image observations in B.
4.2 ADDITIONAL TECHNIQUES AND DETAILS FOR LEVER
Data augmentations. Recently, various data augmentations (Srinivas et al., 2020; Laskin et al., 2020; Kostrikov et al., 2020) have provided large gains in the sample-efficiency of RL from pixel observations. However, LeVER precludes data augmentations because we store the latent vector instead of the raw pixel observation. We find that the absence of data augmentations could decrease sample-efficiency some cases, c.g., when the capacity of B is small. To mitigate this issue, perform K number of different data augmentations for each input observation o and store K distinct latent vectors { fo (AUG, (o:)) = 1... K}. We find empirically that 4 achieves competitive performance to standard RL algorithms in most cases.
Increasing replay capacity. By storing the latent vector in replay buffer, we can adaptively increase the capacity (i.e., total number of transitions), which is determined by the size difference between the input pixel observations and the latent vectors output by the encoder, with few additional considerations. The new capacity of the replay buffer is
c Ō= 0= C* (INRT)
4

---

Under review as a conference paper at ICLR 2021
where C' is the capacity of the original replay buffer, P is the size of the raw observation, L is the size of the latent vector, and K is the number of data augmentations. The number of encoders N is aggorithm-specfii and determines the number of distinct latent vectors encountered for each observation during training. For Q-learning algorithms N = 1, whereas for actor-critic algorithms N = if the actor and critic each compute their own latent vectors. Some algorithms employ larget network for updating the Q-function (Mnih et al., 2015; Haarnoja et al., 2018), but we use the same latent vectors for the online and target networks after freezing avoid storing target latent vectors separately and find that tying their parameters does not degrade performance. The factor of 4 arises from the cost of saving floats for latent vectors, while raw pixel observations are saved as integer pixel values. We assume the memory required for actions, and rewards is small and only consider only the memory used for observations.
5 EXPERIMENTAL RESULTS
We designed our experiments answer the following questions:
Can LeVER reduce the computational overhead of various continuous (see Figure 2) and discrete (see Figure 3) control tasks?
RL algorithms for both
Can LeVER reduce the memory consumption and improve the sample-efficiency of off-policy RL algorithms by adaptively increasing the buffer size (see Figure 4 and Figure 5)?
Can LeVER be useful for computation-effn -effcciinnn transfer learning (see Figure 7(a))?
Do CNN encoders of RL agents converge early in training (see Figure 7(b) and Figure 7(c))?
5.1
SETUPS
Computational efficiency. We first demonstrate the computational efficiency of LeVER on the DeepMind Control Suite (DMControl; Tassa et al. 2018) and Atari games (Bellemare et al., 2013) benchmarks. DMControl commonly used for benchmarking sample-efficiency for image-based continuous control methods. For DMControl experiments, we consider a state-of-the-art model-free RL method, which applies contrastive learning (CURL; Srinivas et al. 2020) to SAC (Haarnoja et al., 2018), 8), using the image encoder architecture from SAC-AE (Yarats et al., 2019). For evalu- ation, we compare the computational efficiency of CURL with and without LeVER by measuring floating point operations (FL.Pss.? For discrete control tasks from Atari games, we perform simi- lar experiments comparing the FLOPS required by Rainbow (Hessel et al., 2018) with and without LeVER. For both our method and the baseline, we use the hyperparameters and encoder architecture of data-efficient Rainbow (van Hasselt et al., 2019). We train our method and the baseline for 500K timesteps as done in Srinivas et al. (2020) and Laskin et al. (2020).
Memory efficiency. We showcase the memory efficiency of LeVER with a set of constrained- memory experiments in DMControl. For Cartpole and Finger, the memory allocated for storing observations is constrained to 0.03 GB, corresponding to an initial replay buffer capacity C = 1000. For Reacher and Walker, the memory constrained to 0.06 GB for an initial capacity of C = 2000 due to the difficulty of learning in these environments. In this constrained-ment memory setting, we compare the sample-efficiency of CURL with and without LeVER. As an upper bound, we also report the performance of CURL without memory constraints, i.e., the replay capacity is set to the number of training steps. For Atari experiments, the baseline agent is data-efficient Rainbow and the memory allocation is 0.07 GB, corresponding to initial replay capacity C = 10000. The other hyperparameters are the same as those in the computational efficiency experiments.
The encoder architecture used for our experiments with CURL is used in Yarats et al. (2019). It consists of four convolutional layers with 3 x 3 kernels and 32 channels, with the ReLU activa- tion applied after each conv layer. The architecture used for our Rainbow experiments is from van Hasselt et al. (2019), consisting of a convolutional layer with 32 channels followed by convolu- tional layer with 64 channels, both with 5 5 kernels and followed by ReLU activation. For our method, we freeze the first fully-connected layer of the actor and critic in CURL experiments and
We remark that the higher layers of the target network are not tied to the online network after freezing. We explain our procedure for counting the number of FI.OPs in Appendix B.
5

---

Under review as a conference paper at ICLR 2021
CURL
CURL LeVER (ours)
1985
Getarleoningrr
1330
13.00
Mhakharsway
5000
600
Chuslahron
302
200
530
500
802-
202
200
230
200
6rggh 6rlo 16
"ari""
PLORS
PLORG
FLORG
RLON
R.ON
Figure 2: Learning curves for CURL with and without LeVER where the x-axis shows FLOPs. The dotted gray line denotes the encoder freezing time Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Rainbow
Rainbow
LeVER (ours)
1400
Alen
300
Arider
SunkHoa
40.00
Crany
1200-
200-
30 38.000
200
200
20,000
500
500-
Ruschal PLOPS
.00
Pieipo FLOPs
4x
PLOPS
PLOPs
400-
4000
600-
2000
500-
3000
3000
400
2000
2000
-100
1000
2000
2
-200
10
S-/01
9+1g
2.5-1ghh
100
2.5-1gh
50H10
F..PP
P.OPP
FLOPP
Figure 3: Learning curves for Rainbow with and without LeVER, where the x-axis shows FLOPs. The dotted gray line denotes the encoder freezing time Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
the last convolutional layer of the encoder in Rainbow experiments (see Appendix E for justifica- tion). We present the best results achieved by LeVER across various values of Ty, the number of training steps before freezing the encoder. The full list of hyperparameters is provided in Appendix G (DMControl) and Appendix H (Atari).
5.2 IMPROVING COMPUTE- AND EFFICRYYYF
Experimental results in DMControl and Atari showcasing the computational efficiency of LeVER are provided in Figures and Figure 3. CURL and Rainhow both achieve higher performance within significantly fewer FLOPs when combined with LeVER in DMControl and Atari, respectively. Ad- ditionally, Table 1 compares the performance of Rainbow with and without LeVER at 45T (4.5e13) FLOPs. In particular, the average returns are improved from 145.8 to 276.6 compared baseline Rainbow in Hank and from 2325.5 to 4123.5 in Qbert. We remark that LeVER achieves better computational efficiency while maintaining the agent's final performance and comparable sample- efficiency (see Appendix E for corresponding figures).
Experimental results Atari and DMControl showcasing the sample-efficiency of LeVER in the constrained-memon setup are provided in Figure 4 and Figure 5. CURL and Rainbow achieve higher final performance and better sample-efficiency when combined with LeVER in DMControl and Atari, respectively. Additionally, Table compares the performance of unbounded memory Rainbow and constrained-memoot (0.07 GB) Rainbow with and without LeVER at 500K environ- ment interactions. In particular, the average returns are improved from 10498.0 to 17620.0 compared to baseline Rainbow in CrazyClimber and from 2430.5 to 3231.0 in Qbert. Although we disentan- gle the computational and memory benefits of LeVER in these experiments, we also highlight the

---

Under review as a conference paper at ICLR 2021
Scores at 45T FLOPs
Scores 500K environment steps (0.07GB)
Rainbow
Rainbo++LeVER
Rainbow
Rainbow+LeVER
Alien
992.0 152.7
1172.6 +39..
1038.4 + 101.1
1134.6 +:52.
Amidar
144.0 27.4
250.5 +47.4
121.0 31.2
165.3 #47.6
BankHeist
145.8 61.2
276.6 +98.1
161.6 +57.7
151.8 = 65.8
CrazyClimber
21580.0 3514.6
28066.0 +4108.5
10198.0 :: 1387.8 8
17620.0 44418.1
Krull
2799.5 468.1
3277.5 +440.5
2215.7 336.9
3069.2 1377.6
PrivateEye
81.5 + 37.0
100.0 +:0.0
80.0 +400
80.0 10.0
Qbert
2325.5 + 1152.7
4123.5 +1385.5
2430.5 + 658.8
3231.0 11567.
Seaquest
402.8 + 48.4
561.2 100.5
262.8 + 19.1
336.8 45.9
Table Scores Atari games 45T FLOPs corresponding to Figure 3 and scores Atari games at 500K environment interactions in the constrained setup (0.07GB) corresponding to Fig- The results show the mean and standard deviation averaged five runs, and the best results are indicated in bold.
computational gain of LeVER in constrained-memory settings (effectively combining the benefits) in Appendix D.
Rainbow with 3.5GB (upper bound)
Roinbow with 0.07GB
Rainbow + LeVER with 0.07GB (ours)
250
Andder
400
Donk
CoscyClrbbrr
35,000
200-
300
1000
150
200
100
10,000
10
34.0" AHEOT Seip7
3410
19127 2010 3+107
AHiD! Seld"
Kiul
Ober:
Seasued!
3500
1963
5003
1902-
4003-
200-
2803-
1803-
1500
-203-
100
Figure 4: Comparison of the sample-efficiency of Rainbow with and without LeVER in constrained- memory (0.07 GB) settings. The dotted gray line denotes the encoder freezing time t = Ty. The solid line and shaded regions represent the mean and standard deviation. respectively, across five runs.
5.3 FREEZING LARGER CONVOLUTIONAL ENCODERS
We also verify the benefits of LeVER using deeper convolutional encoders, which are widely used in a range of applications such as visual navigation tasks and favored for their superior generalization ability. Specifically, we follow the setup described in Section 5.1 and replace the SAC-AE architec- ture (4 convolutional layers) with the IMPALA architecture (Espeholt et al., 2018) 8) (15 convolutional layers containing residual blocks (He al., 2016)). Figure 6(b) shows the computational efficiency of LeVER in harpple-swinggee and Walker-wal walk with the IMPALA architecture. CURL achieves higher performance within significantly fewer FLOPs when combined with LeVER. We remark that the gains due to LeVER are more significant because computing and updating gradients for large convolutional encoders is very computationally expensive.

---

Under review as a conference paper at ICLR 2021
CURE 1208
(a) Caripole-swingup
(b) Finger-spin
(c) Reacher-easy
300
1417
2010 5010" 2010" 5-1
1012 2010 SHT GHIE OIT OHIE
(d) Walker-walk
(e) Cheetah-run
Figure 5: Comparison of the sample-efficiene, of CURL with and without LeVER in constrained- memory settings. The dotted gray line denotes the encoder freezing time Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Cheatshhiiinn
Wallewwai
SUPP
CURL -ta (n512)) (1RLL B-ctie
TraracSore
PLOPs
Varying hatch sizes
(b) IMPALA architecture
Figure 6: (a) Left: Cheetah-run learning curves for CURL with and without LeVER, with batch sizes b=512 and b=128, respectively, where the x-axis shows samples. Right: Number of FLOPs used by each agent to achieve its final performance. (b) Learning curves for CURL using the IMPALA architecture with and without LeVER, where the x-axis shows FLOPs. The dotted gray line denotes the encoder freezing time t Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
5.4 IMPROVING SAMPLE EFFICIENCY WITH LEVER AND LARGER BATCH SIZES
In this subsection we show that we can combine LeVER with larger batch sizes to improve the sample-efficiency of RL agents. Larger batch sizes have been shown to improve agent performance in many settings, but require more compute since each gradient calculation and update is performed on more observations. We demonstrate that LeVER can mitigate these issues by showing results in Cheetah-run, a task known to achieve better performance with larger batch sizes. Figure 6(a) shows the sample-efficiency of CURL (batch 128) and CURL+LeVER (batch 512), and the cor- responding computational efficiency of each agent. CURL achieves better sample-efficiency when combined with VeR and the larger batch size, but does this within comparable compute budget. In contrast, CURL (batch 256) requires significantly more compute to achieve similar performance CURL+LeVER (batch 512).
5.5 IMPROVING COMPUTATIONAL EFFICIENCY IN TRANSFER SETTINGS
We demonstrate, another application of our method, that LeVER increases computational effi- ciency in the transfer setting: utilizing the parameters from Task A on unseen Tasks B. Specifically, train CURL. agent for 60K environment interactions Walker-stand; then, only fine-tune
8

---

Under review as a conference paper at ICLR 2021
Store
Cataale WYDE
DJR sudte horwereree o0
0UR
FLOPs
(a) Walker-stand to Walker-walk
(b) Spatial attention map
(c) SVCCA similarity scores
Figure 7: (a) Comparison of the computational efficiency of agents trained from scratch with CURL and agents trained with CURL+LeVER from Walker-stand pretraining. (b) Spatial attention map from CNN encoders.
(c) SVCCA (Raghu et al., 2017) similarity scores between each layer and itself at time and [+10K throughout training for Walker-walk.
the policy and Q-functions on unseen tasks (e.g.. Walker-walk and Cheetah-run) using network parameters from Walker-stand. To computation, during fine-tuning, we freeze the encoder pa- rameters. Figure 7(a) shows the computational gain of LeVER in task transfer (i.e., Walker-stand to Walker-walk similar Yarats et al. (2019)), and domain transfer (i.e., Walker-stand Cheetah-run and Walker-stand to Hopper-hop) is shown in Appendix C. Due to the generality of CNN features, we can achieve this computational gain using pretrained encoder. For the task transfer setup, we provide more analysis the number of frozen layers and freezing time hyperparameter T3 in Appendix C.
5.6 ENCODER ANALYSIS
In this subsection we present visualizations to verify that the neural networks employed in deep reinforcement learning indeed converge from the bottom up, similar those used in supervised learning (Raghu et al., 2017). Figure 7(b) shows the spatial attention map for two Atari games and one DMControl environment at various points during training. Similar Laskin al. (2020) and Zagoruyko & Komodakis (2017), we compute the spatial attention map by mean-pooling the absolute values of the activations along the channel dimension and follow with a 2-dimensional spatial softmax. The attention map shows significant change in the first 20% of training, and remains relatively unchanged thereafter, suggesting that the encoder converges to its final representations early in training. Figure 7(c) shows the SVCCA (Raghu al., 2017) score, a measure of neural network layer similarity, between a layer and itself at time t and t+10K. The convolutional layers of the encoder achieve high similarity scores with themselves between time and t+ t+10,, while the higher layers of the policy and Q-network continue to change throughout training. In our DMControl environments freeze the convolutional layers and the first fully-connected layer of the policy and Q-network (denoted fcl), Although the policy fel continues to change, the convergence of the Q- network fel and the encoder layers allow us achieve our computational and memory savings with minimal performance degradation.
CONCLUSION
In this paper, we presented LeVER, simple but powerful modification of off-policy RL algorithms that significantly reduces computation and memory requirements while maintaining state-of-the- art performance. We leveraged the intuition that CNN encoders in deep RL. converge to their fi- nal representations early training to freeze the encoder and subsequently store latent vectors to save computation and memory. In our experimental results, demonstrated the compute- and memory-efficiiecy of LeVER in various DMControl environments and Atari games, and proposed technique for computation-efficie improvements in compute- and memory- eeemor--effciiecyy in deep RL that can be made without sacrificing performance, in hopes of making deep RL more practical and accessible in the real world.
transfer learning. With VER, highlight the potential for

---

Under review as a conference paper at ICLR 2021
REFERENCES
Adria Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference Machine Learning, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal Artificial Intelligence Research, 47: 259- 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement learning. In International Conference on Machine Learning, 2017.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 538-547877 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 1803. 10/22, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning. 2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representaatinss 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014: 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In International Conference on Machine Learning, 2017.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: efficient densenet using learned group convolutions. In IEEE conference computer vision and pattern recognition, 2018.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: /602.00600 2016.
10

---

Under review as a conference paper at ICLR 2021
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
Jongheon Jeong and Jinwoo Shin. Training enns with selective allocation channels. Interna-
lional Conference on Machine Learning, 2019.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):9--13) 1998.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-basee robotic manipulation. In Conference on Robot Learning, 2018.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprini arXiv:2004. 10044.13444 13649, 2020.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in neural information processing systems, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2224, 1998.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04490 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patter- son, Hanlin Tang, Gu- Yeon Wei, Peter Bailis, Victor Bittorf, et al. MIperf training benchmark. In Conference on Machine Learning and Systems, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, 2016.
Lorenzo Pellegrini, Gabrile Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real-time continual learning. arXiv preprint arXiv: 1912.010 20199

---

Under review as a conference paper at ICLR 2021
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep understanding and improvement. In Advances in neural information processing systems, 2017.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08263 2019.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint 170v:1700.00440, 2017.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning, 2020.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic bert for resource-limited devices. In Annual Meeting of the Association for Computational Linguistics, 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 000900, 2018.
Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. In Annual Meeting of the Association for Computational Linguistics, 2019.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. AAAI Conference on Artificial Intelligence, 2016.
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in rein- forcement learning? In Advances in Neural Information Processing Systems, pp. 14322-14333, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292 1992.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1r0.011110.01741. 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, 2014.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
12

---

Under review as a conference paper at ICLR 2021
Appendix
A ALGORITHM
We detail the specifics of modifying off-policy RL methods with LeVER below. For concreteness, we describe LeVER combined with deep Q-learning methods.
Algorithm Latent Vector Experience Replay (DQN Base Agent)
1: Initialize replay buffer B with capacity C
2: Initialize action-value network Q with parameters and encoder f with parameters +
3: for each timestep & do
4: Select action: a, arg max, (o a)
5: Collect observation o++1 and reward from the environment by taking action
6: if S Ty then
7:
Store transition (or, az 0 rr) in replay buffer B
8:
else
9:
Compute latent states 22, 2++11 (or), (oc+1) Store transition (2s, at, 2t41, Tr) in replay buffer
10:
11:
end if
12:
// REPLACE PIXEL-BASED TRANSITIONS WITH LATENT TRAJECTORIES
13:
if t TTy then
Compute latent states {(2,, 2 2-1)) 1) ,.)) +- { fo((r+))) min(Trre)
14:
15:
Replace {(o,, at, ot+1, (t)]ee
min(Ty.0)
with latent transitions {(z, ae, 2t+1, re) miin(
16:
Increase the capacity of B to
17:
end
18:
// UPDATE PARAMETERS OF Q-NETWORK WITH SAMPLED IMAGES OR LATENTS
19:
for each gradient step do
20:
if < Ty then
21:
Sample random minibatch {(o), aj 0f)1, B
22:
Calculate target y/ =r y max Qo ()(fx(o)+1),a)
23:
Perform gradient step LPRM 1b) (2)
24:
else
25:
Sample random minibatch ((zj, ag 25 B Calculate target yj ry m Qa(zit1, Perform gradient step on (P) (2)
26:
27:
28:
end if'
29:
end for
end for
B CALCULATION OF FLOATING POINT OPERATIONS
We consider each backward pass to require twice as many FLOPs as a forward pass. Each weight requires one multiply-add operation in the forward pass. In the backward pass, it requires two multiply-add operations: at layer i, the gradient of the loss with respect to the weight layer and with respect to the output of layer (i 1) need to be computed. The latter computation is necessary for subsequent gradient calculations for weights at layer ()).
We use functions from Huang et al. (2018) and Jeong & Shin (2019) to obtain the number of op- erations per forward pass for all layers in the encoder (denoted E) and number of operations per forward pass for all MLP layers (denoted M). For concreteness, provide FLOPs breakdown by layer for the architectures we use in Table 2 and 3.
Thhis method for FL.OP calculation is used in htttp:://opeet cow/blog/a and-compute//
13

---

Under review as a conference paper at ICLR 2021
We denote the number of forward passes per iteration F, the number of backward passes per iteration B. and the batch size b. We assume the number of updates per timestep is 1. Then, the number of FLOPs per iteration before freezing at time = T is:
bF(E + M) + 2 + M)
For the baseline, FLOPs are computed using this formula throughout training.
LeVER reduces computational overhead by eliminating most of the encoder forward and backward passes. The number of FLOPs per iteration after freezing is:
bFM + 26BM + EKN
where K the number of data augmentations and N is the number of networks as described in Section 4.2. The forward and backward passes of the encoder removed, with the exception of the EKN term at the end that arises from calculating latent vectors for the current observation.
At freezing time = Ty, need compute latent vectors for each transition in the replay buffer. This introduces a one-time cost of (EK) N min(T,, C)) FLOPs, since the number of transitions in the replay buffer is min(Ty; C), where C the initial replay capacity.
Table 2: Forward pass FL.OPs breakdown by layer for DM Control experiments.
Layer
FLOPs
Layer
FLOPs
Layer
FLOPs
encoder:convl
4.3e6
critic:fel
1.9e6
actor:fel
1.9e6
encoder:conv2
1.4e7
critic:fe2
5.3e4
actor:fe2
5.2e4
encoder:conv3
1.3e7
critic:fe3
1.0e6
actor:fe3
1.0e6
encodercconv4
1.1c7
critic:fe4
1.0c3
actor:fo4
2.0c3
Table 3: Forward pass FLOPs breakdown by layer for Atari experiments.
Layer
FLOPs
Layer
FLOPSs
Layer
FLOPs
encodersconvl
8.2e5
value stream:fel
1.SeS
advantage stream:fel
1.5e5
encoder:con22
4.6e5
value stream:fc2
1.3e4
advantage stream:fe2
1.2e5
C ADDITIONAL TRANSFER EXPERIMENTS
Domain transfer. In Figure 7(a) show the computational efficiency of LeVER in a task transfer setting. Here we show in Figure that frozen encoder parameters can also be used in domain transfer tasks (i.e. Walker-stand Cheetah-run and Walker-stand Hopper-hop). In this setting, we only transfer the encoder parameters, whereas we transfer the entire network in task transfer.
Transfer setting analysis. Figure 7(a) we show the computational efficiency of LeVER on Walker-walk with Walker-stand pretrained for 60K steps, with four convolutional layers frozen. We provide analysis for the number of layers frozen and number of environment interactions be- fore freezing Ty in Figure 9. We find that freezing more layers allows for more computational gain, since we can avoid computing gradients for the frozen layers without sacrificing performance. Longer pretraining in the source task improves compute-efficiency in the target task; however, early convergence of encoder parameters enables the agent to learn good policy even with only 20K interactions before transfer.
We remark that Yosinski et al. (2014) examine the generality features learned by neural networks and the feasibility of transferring parameters between similar image classification tasks. Yarats et al. (2019) show that transferring encoder parameters pretrained from Walker-walk to Walker-stand and Walker-run can improve the performance and sample-efficiency of SAC agent. For the first time, we show that encoder parameters trained on simple tasks can be useful for computation-effin efficcien training in complex tasks and new domains.
14

---

Under review as a conference paper at ICLR 2021
CURL'LeVER
30
CURL from scratch
GURL from scratch
Store
Score
FLOPs
FLOPs
1011
(a) Walker-stand to Cheetah-run
(b) Walker-stand to Hopper-hop
Figure 8: Comparison of the computational efficiency of agents trained from scratch with CURL and agents trained with CURL+LeVER from Walker-stand pretraining.
Score
Score
CURL L.VER fmane Sayee
CURL ILAER frecce SOK
CURL, (LaVER: Suace
CUIS, (LeVEIc freace 40K
FLOPs
FLOPs
(a) Number of frozen layers.
(b) Freezing time hyperparameter Ty-
Figure 9: (a) Analysis on the number of frozen convolutional layers Walker-walk training from Walker-stand pretrained for 60K steps. (b) Analysis the number of environment steps Walker- stand agent is pretrained prior to Walker walk transfer, where the first four convolutional layers are frozen.
D COMPUTATIONAL EFFICIENCY IN CONSTRAINED-MEMONK SETTINGS
In our main experiments, we isolate the two major contributions of our method, reduced compu- tational overhead and improved sample-efficiency in constrained-memory settings. In Figures 10 and we show that these benefits can also be combined for significant computational gain in constrained-memor settings.
E SAMPLE-EFFICIENCY PLOTS
In section 5.2 we show the computational efficiency of our method in DMControl and Atari envi- ronments. We show in Figure 12 that our sample-efficiency is very close to that of baseline CURL (Srinivas et al., 2020), with only slight degradation in Cartpole-swingup and Walker-walk. In Atari games (Figure 13), we match the sample-efficiency baseline Rainbow (Hessel al., 2018) very closely, with no degradation.
F GENERAL IMPLEMENTATION DETAILS
LeVER can be applied to any convolutional encoder which compresses the input observation into a latent vector with smaller dimension than the observation. We generally freeze all the convolutional
15

---

Under review as a conference paper at ICLR 2021
CUNL PLANELA 2006
TLDRA
(a) Caripole-swingup
(b) Finger-spin
(c) Reacher-casy
Sore
OUMI 210R00N
(d) Cheetah-run
(c) Walker-walk
Figure 10: Comparison of CURL in constrained-mente settings with and without LeVER, where the x-axis shows FLOPs, corresponding to Figure 5. The dotted gray line denotes the encoder freezing time / = Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Rulioas di ErEO
Dooe
(a) Alien
(b) Amidar
(c) BankHeist
(d) CrazyClimber
3108
ribow IuEncn
(L0703)
(e) Krull
PrivateEye
(g)
Qbert
(h) Seaquest
Figure 11: Comparison of Rainbow constrained-memor settings with and without Le LeVER, where the x-axis shows FLOPs, corresponding to Figure The dotted gray line denotes the encoder freezing time f = Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
layers and possibly the first fully-connected layer. our main experiments, we chose to freeze the first fully-connected layer for DM Control experiments and the last convolutional layer for Atari experiments. We made this choice in order to simultaneously save computation and memory; for those architectures, if freeze earlier layer, (convolutional features) are too large for our method to save memory. In DM Control experiments, the latent dimension of the first fully-connected layer is 50, which allows roughly 12X memory
less computation, and the latent vectors
16

---

Under review as a conference paper at ICLR 2021
Soor
Franziore
(a) Cartpole-swingup
(b) Finger-spin
(c) Reacher-casy
Soor
(d) Cheetah-run
(c) Walker-walk
Figure 12: Comparison of the sample- sample-fficiency of CURL with and without L+VER, corresponding to Figure 2. The dotted gray line denotes the encoder freezing time t Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
(a) Alien
(b) Amidar
(c) BankHeist
(d) CrazyClimber
A LApioa
(e) Krull
(1) PrivateEye
(g) Qbert
(h) Seaquest
Rainbow with and without LeVER, correspond- ing to Figure 3. The dotted gray line denotes the encoder freezing time = Ty. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Figure 13:
Comparison of the sample-efficiene
gain. In Atari experiments, the latent dimension of the last convolutional layer is 576, which allows roughly 3X memory gain.
G DMCONTROL IMPLEMENTATION DETAILS
We use the network architecture in https://gstt wow/Missaaeeee www.ashii/canurr..s for our
CURL (Srinivas et al., 2020) implementation. We show a full list of hyperparameters in Table 4.
H ATARI IMPLEMENTATION DETAILS
We the network architecture in https: /t///gitte /gittab..../////// www./aattiin/aaiiaa.a...... for our Rain- bow (Hessel et al., 2018) implementation and the data-efficient Rainbow (van Hasselt et al., 2019) encoder architecture and hyperparameters. We show full list of hyperparameters in Table 5.
17

---

Under review as a conference paper at ICLR 2021
Table 4: Hyperparameters used for DMControl experiments. Most hyperparameter values are un- changed across environments with the exception of initial replay buffer size, action repeat, and learning rate.
Hyperparameter
Value
Augmentation
Crop
Observation rendering
(100, 100)
Observation down/upsampling
(84, 84)
Replay buffer size Figure Initial replay buffer size Figure 5
Number of training steps
1000 cartpole, swingup: cheetah. run; finger, spin
2000 reacher, casy; walker, walk
Number of updates per training step Initial steps Stacked frames Action repeat
1
1000
3
finger. spin; walker, walk
4 cheetah, run; reacher, easy
8 cartpole, swingup
Hidden units (MLP)
1024
Evaluation episodes
Evaluation frequency
2500 cartpole, swingup
10000 cheetah, run; finger, spin; reacher, easy; walker, walk
Optimizer
Adam
(8,, (2) Te. Q6) B2) (a) Learning rate (Sv. TT. Qo)
(.9, .999)
(.5,
2e cheetah, run
1e cartpole, swingup; finger, spin; reacher, easy; walker, walk
Learning rate (a)
le
Batch Size
512 cheetah,
cartpole, swingup: finger, spin; reacher, easy; walker, walk
Q function EMA т
0.01
Critic target update freq Convolutional layers Number of filters Non-linearity
2
4
ReLU
Encoder EMA т
0.05
Latent dimension
50
Discount
.99
Initial temperature
0.1
Freezing time Ty in Figure 2
10000 cartpole, swingup
50000 finger, spin; reacher, easy
60000 walker, walk
80000 cheetah,
Freezing time Ty in Figure 5
10000 cartpole, swingup 50000 finger, spin
30000 reacher, easy
80000 cheetah, run; walker, walk
18

---

Under review as a conference paper at ICLR 2021
Table 5: Hyperparameters used for Atari experiments. All hyperparameter values are unchanged across environments with the exception of encoder freezing time.
Hyperparameter
Value
Augmentation
None
Observation rendering
(84, 84)
Replay buffer size in Figure
Number of training steps
Initial replay buffer size in Figure 4
10000
Number of updates per training step
1
Initial steps
1600
Stacked frames
Action repeat
1
Hidden units (MIP)
256
Evaluation episodes
10
Evaluation frequency
10000
Optimizer
Adam
(81,32) + (fe. Qo)
1e - .999)
Learning rate (f)
Learning rate (ox)
0.0001
Batch Size
32
Multi-step returns length
20
Critic target update freq Convolutional layers
2000
2
Number of filters
32, 64
Non- Non-linearity
ReLU
Discount
.99
Freezing time Ty in Figure
50000 Alien; Amidar; BankHeist; Krull; Seaquest
100000 CrazyClimber; PrivateEye; Qbert
Freezing time T3 Figure
50000 Amidar; BankHeist; Krull
100000 Alien; CrazyClimber; Qbert
150000 PrivateEye; Seaquest
19


Reviews:
[{"review_id": "yzzcbIs1TlC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Significance:\nThe paper proposes to reduce memory and computation demands of an image based RL by exploiting early convergence of the convolutional encoder. While the approach is quite intriguing, I find it hard to see the approach being general and thus having a significant effect on the RL community. \n\nPros:\nThe paper provides an interesting approach in order to save memory and computational footprint in image-based deep RL. The method is based on an observation that the convolutional encoder converges faster then the actor and critic networks.\nThe authors provide an extensive comparison and ablation study that covers different domains (DMControl and Atari). The ablation study shades more light on some of the properties for the training dynamics of an image based model free RL algorithm (attention map, layer convergence)\nCons:\nI’m a bit skeptical about the generality of this approach. Freezing the encoder’s weights prematurely prevents the encoder to adequately encode priorly unseen images (out of distribution), this in turn will hurt the agent’s performance down the road. Note that in DMControl the method is only showcased on the simplest tasks (at least cheetah run should be included) where learning a good policy only takes about 100K env steps -- enough to collect sufficient data support to learn an almost optimal policy. Figure 6c adheres to my point here, as conv1 weights pretty much don’t change, suggesting convergence.\nThe task transfer (walker stand to walker walk) experiment is exactly the same as the one demonstrated in SAC-AE (https://arxiv.org/pdf/1910.01741.pdf, Section 5.3). I’m not sure what is the difference here besides using CURL instead of SAC-AE. Could the authors elaborate? Also the domain transfer experiment (App G) shows that the approach doesn’t really buy anything.\nIt seems that the approach requires storing 4 data augmented observations per env observation and the replay buffer size is equal to the number of training steps. I would like to point out that DrQ (https://arxiv.org/pdf/2004.13649.pdf) only needs a constant size replay buffer of 100K transitions, even if training requires 3M steps. Given that, I’m skeptical that LeVER would buy much in terms of memory and computation here. It would be nice to see a head to head comparison.\nResults are demonstrated over 3 random seeds, which is too few to get any conclusive statistical evidence given the variance. A common practice is to use 10 seeds for DMControl and 5 seeds for Atari.\n\n\nQuality:\nWhile the technical contributions of the paper are limited in novelty and significance, and don’t meet the high acceptance bar of ICLR, I still think the paper is well done and could be a good workshop paper.\n\n\nClarity:\nThe paper in general is clearly written and well organized. I particularly appreciate the contribution bullet points and the experimentation roadmap.\n\n------------------------------------------\nPost rebuttal:\n\nThe authors has addressed several of my concerns regarding the method's generality and some experiments. While I'm raising my score to 5, I'm still not convinced that the paper proposed a valuable contribution to the community -- comparing RL algorithms in memory or compute footprints instead of the number interactions with the environment is not meaningful, especially when a simulator is in use. There are several much simpler things one can do to tradeoff compute or memory (for example re-render observations on the fly from stored low dimensional states). Thus, I'm voting for a rejection. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tUccLitY8jG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\n    - This paper presents a method for compute- and memory-efficient reinforcement learning where the visual encoder is frozen partway into training.  After freezing latent vectors are stored in the replay buffer instead of images (and any existing images are replaced by them).  This leads to both better compute and memory utilization.\n    - The authors demonstrate their method by comparing to Rainbow on Atari and CURL on DM Control.  On DM control, their method reduces the compute by a considerable margin.  On Atari, the results are less clear cut, but the compute cost is reduced.\n    - When they also impose a memory constraint, the effectiveness of their method is further increased.\n- Strengths\n    - Elegant and \"obvious in hindsight\" (a good thing) idea, meaning it will likely have broad applicability\n    - While the authors only tested it on off-policy methods, it is clearly also applicable to on-policy methods that use a rollout storage (PPO, PPG, IMPALA, V-MPO, A2C, etc.)\n    - Good FLOPs vs. Reward results on DM Control\n- Weaknesses\n    - The memory constrained results seem very contrived.  60 MB is a tiny amount of memory and even 9.0 GB of (presumably) CPU memory isn't that prohibitive.\n        - Perhaps if wall-clock time was plotted in addition to samples, the smaller memory footprint of LeVER would mean the replay buffer can be stored on the GPU and training would be much faster since many expensive CPU -> GPU transfers would be eliminated?\n    - The CNN is frozen all at once instead of frozen iteratively. Raghu 2017 and Figure 6c suggest that the early layers could be frozen much earlier, although this may increase the memory usage initially since CNNs typically increase the memory size of the feature map in lower layers.\n    - T_f seems like yet another hyper-parameter to tune.  In theory, SVCCA (or PWCCA from Morcos 2018) could be used to choose when to freeze (if the representation of the shallowest unfrozen layer didn't change in the last K steps, freeze it).  There is a nontrivial cost to computing either of those so it  \n- Suggestions for improvement\n    - I very much like the idea of this paper, but I think the chosen application is making the idea look less convincing (i.e. freezing the CNN isn't really that impactful when the CNN and observations are tiny).  I urge the authors to try this for visual navigation (i.e. PointGoal Navigation in Habitat/AI2 Thor/etc), where deeper CNNs, e.g. ResNet18, and higher resolution images, e.g. 256x256, are used.\n    - One other potentially benefit of LeVER is the ability to increase the batch size during training (as in Smith 2017).  This could perhaps increase its effectiveness further?\n    - In figure 6a, there should also be a CURL + LeVER from Scratch line.  Currently two variables are changing.\n    - One paper that should be cited is Fang 2019.  They do a very similar thing as LeVER out of necessity.\n- Overall\n    - While I like this paper and think the idea has a lot of potential, I don't think it is quite ready for publication yet.  I urge the authors to try their idea in a setting with a larger CNN and higher resolutions and to see if there is a way to find T_f without it being \"yet another hyper-parameter\".\n- References\n    - Smith 2017: https://arxiv.org/abs/1711.00489\n    - Morcos 2018: https://arxiv.org/abs/1806.05759\n    - Fang 2019: https://arxiv.org/abs/1903.03878\n\n## Post Rebuttal\n\nI thank the authors for their responses. The results with a larger CNN and increased batch size help show the benefit of the method further.  I still believe the presentation of the method would be considerably stronger if results were presented in a setting with larger CNNs and higher resolution.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "reSiiImfcIO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper summary\n\nThis work proposes LeVER, a method that modifies general off-policy RL algorithms with a fixed layer freezing policy for early embedding layers (in this particular case, a few early layers of a CNN). As a direct consequence, the method enables to store embeddings in the experience replay buffer rather than observations, with a potential decrease in memory required, as well as providing a boost in clock time due to fewer gradient computations needed for every update. The method is benchmarked with a couple of off-policy RL algorithms against a few different environments.\n\n### Good things\n\n- The approach is extremely relevant to most of the RL community. We are training for longer periods of time and generating significantly more data than we did a few years ago, so any method that enables to increase training efficiency is extremely welcomed.\n- The method is simple, but clever, and the manuscript quite nicely details the steps taken to improve learning stability (which can arise due to the obvious possibility of bad model freezes).\n- The coverage of related work in the literature throughout the manuscript is excellent, and provides enough pointers for the reader to understand how the manuscript is placed in it.\n- The experimental setting clearly states hypotheses and questions that will be answered.\n- Section 5.4 convincingly argues that the freezing method is empirically justified.\n\n### Concerns\n\nThis is a good paper, so generally I don't have any strong negative thoughts, however I think it would be good to report how the method does when different choices are made with respect to how much of the network is frozen.\nThat is, in the proposed experiment setting the choices were reasonable but nonetheless a little arbitrary, so knowing a little bit more about learning dynamics with this approach would probably make the paper stronger and more robust for future readers.\n\n### Questions:\n\n- I wonder whether the authors would shed more details on the transfer learning setting (e.g. whether the transfer capacity changes wrt. changes in method hyperparameters such as freezing time, different saved embedding functions, etc.), and whether the results do generally show up in more environments/algorithms.\n- The reduction in variance after the freezing is interesting; I wonder if the plots could show all the single runs, and whether the authors have any explanations for this somewhat consistent (!) change in learning behaviour.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "p2k9zx3hSMW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes to reduce the intensive computation and memory requirement in reinforcement learning trainings by freezing the parameters of lower layers early. Besides, the authors also propose to store the low-dimensional latent vectors rather than the high-dimensional images in the replay buffer for experience replay. The effectiveness of the proposed techniques is evaluated on DeepMind Control environments and Atari. The motivation for this work is strong, and the results are impressive. However, the proposed technique is described in a very general way without clearly defined applicable conditions and specific design methods. Below are detailed comments and questions.\n\n1: the main idea is to freeze lower layers of CNN encoders. However, for a certain network with various structures, is there any applicable conditions or limitations? How to choose the number of layers to freeze? The proposed technique needs to rigorous definition and explanation. \n\n2: It seems the reduction comes from the freezing of lower layers. I am wondering what is the computation/memory requirement breakdown in terms of layers? Is it always the case that lower layers consume a significant amount of computation and memory? If it is not the case, can LeVER still be effective?\n\n3: this paper also proposes to store latent vectors instead of high-dimensional images. There again lacks detailed description and explanation of the applicable conditions and to what extent we can reduce the dimension of the latent vectors.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
This paper presents approach to improve compute and memory efficiency by freezing layers and storing latent features. The approach is simple and provide efficiency. However, there are concerns as well. One big concern is that the experiments are not on realistic settings for example real world images and the current CNN is too simple. Overall, the reviewers are split. The AC agrees with some of the reviewers that for a paper like this experiments on more realistic setting will make it significantly stronger. 

Decision: Reject