Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation
Authors: Qingfeng Lan
Yangchen Pan
Jun Luo
A. Rupam Mahmood
Venue: Accepted by TMLR ()

Abstract: Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in a heavy memory burden, especially for onboard and edge devices with limited memory capacities. We propose memory-efficient reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both feature-based and image-based tasks while easing the burden of large experience replay buffers.

Published in Transactions on Machine Learning Research (04/2023)
Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation
Qingfeng Lan Deportment of Computing Science University of Alberta
gion3@nalberta.ca
Yangchen Pan
University of Orford
Jun Luo Huawei Nooh's 's Ark Lab
jun. lwol @haawei.comm com
A. Rupam Mahmood
wwwahhmoodd@ulabrtt..cco
Deportment of Computing Science
University of Alberta CIFAR AI Chair, Amii
Reviewed OpenReview: https: openreview. net/ forum? id= =SDCula/Br
Abstract
Artificial neural networks are promising for general function approximation but challenging to train on non-independent or non-identically distributed data due to catastrophic forgetting. The experience replay buffer, a standard component in deep reinforcement learning, is often used to reduce forgetting and improve sample efficiency by storing experiences in a large buffer and using them for training later. However, a large replay buffer results in heavy memory burden, especially for onboard and edge devices with limited memory capacities, We propose memory-efficint reinforcement learning algorithms based on the deep Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network. Compared to baseline methods, our algorithms achieve comparable or better performance in both caature-based and image-based tasks while easing the burden of large experience replay buffers.
1
Introduction
When trained on non-independent and non-identically distributed (non-lID) data and optimized with stochastic gradient descent (SGD) algorithms,, neural networks tend to forget prior knowledge abruptly, a
phenomenon known as catastrophic forgetting (French 1999 McCloskey Cohen 1989 It is widely observed
continual supervised lcarning (Hsu al. 2018 Farquhar & Gal 2018 Van de Ven Tolias 2019 Delange
al. 2021 and reinforcement learning (RL) (Schwarz et al.) 2018 Khetarpal et al. 2020 Atkinson al.,
2021 In the RL case, agent receives a non-Ill) stream of experience due to changes in policy, state
distribution, the environment dynamics, simply due to the inherent structure of the environment (Alt
ct al.
2019 This leads to catastrophic forgetting that previous learning is overridden by later training,
"ork partially done intern at Noah's Ark Lab, Huawei Canada.
Work done while Noah's Ark Lab, Huawei Canada.
1 Code release: bttpp::/giithn http://gtthu.ccoo//.....e com//lan///MaD

---

Published in Transactions on Machine Learning Research (04/2023)
0.5
0.5
-0.5
-0.5
boundary
Incul boundary
True function
Trus fundtion
Leamnd function Stage
Leamed fundtion after
function Stage
Loamed Stage
0.0
0.5
1.0
1.5
2.0
0.5
X
(a) Results trained with SGD.
(b) Results trained with SGD knowledge consolidation (our method).
Figure A visualization of learned functions after training as well as the true function. In Stage 1, we generate training samples (I, y), where E [0, 1] and y sin().)). In Stage 2, x E [1, and y = sin(Tx)). The blue dotted line shows the boundary of the input space for Stage 1 and 2.
resulting in deteriorating performance during single-task training (Ghiassian al. 2020 Pan et al. 2022a To mitigate this problem, Lin (1992) equipped the lcarning agent with experience replay, storing recent transitions collected by the agent in a buffer for future use. By storing and reusing data, the experience replay buffer alleviates the problem of non-IID data and dramatically boosts sample efficiency and learning stability. In deep RL, the experience replay buffer is a standard component, widely applied in value-based algorithms (Mnih al., 2013 2015 van Hasselt al.) 2016), policy gradient methods (Schulman al. 2015; 2017 Haarnoja al. 2018 Lillicap al.) 2016), and model-based methods (Heess al. 2015 Ha & Schrittwieser et al.) 2020).
Schmidhuber 2018
However, using an experience replay buffer is not an ideal solution to catastrophic forgetting. The memory capacity of the agent is limited by the hardware, while the environment itself may generate an indefinite amount of obscrvations. To store enough information about the environment and get good lcarning performance, current methods usually require a large replay buller (e-g., a bulfer of million images). The requirement. of large buffer prevents the application of RL algorithms to the real world since creates heavy memory burden, especially for onboard and edge devices (Hayes al.) 2019 Hayes & Kanan 2022). For example, Wang et al. (2023) showed that the performance of SAC (Haarnoja et al. 2018 decreases significantly with limited memory due to hardware constraints of real robots. /mmth et al. (2022) demonstrated that a quadrupedal robot can learn to walk from scratch in 20 minutes in the real-world. However, to be able to train outdoors with enough memory and computation, the authors had to carry a heavy laptop tethered to a legged robot during training, which makes the whole learning process less human-friendly. The world is calling for more memory-offiiient RL algorithms.
In this work, we propose memory-efficient RL algorithms based the deep Q-network (DQN) algorithm. Specifically, we assign new role to the target neural network, which was introduced originally to stabilize training (Mnih et al. 2015). In our algorithms, the target neural network plays the role of a knowledge keeper and helps consolidate knowledge in the action-value network through a consolidation loss. We also introduce a tuning parameter to balance learning new knowledge and remembering past knowledge. With the experiments both featuro-based and imago-basod environments, we demonstrate that our algorithms, while using an experience replay buffer at least 10 times smaller compared the experience replay buffer for DQN, still achieve comparable even better performance.
2

---

Published in Transactions on Machine Learning Research (04/2023)
Greedy Action
0-
DQN
20000
40000
60000
80000
100000
Greedy Action
1-
M
DQN(S)
20000
40000
60000
80000
100000
Greedy Action
1-
MeDQN(U)
20000
40000
60000
80000 100000
Step
Figure 2: The greedy actions of randomly sampled state for different methods during training in Mountain Car. There are 3 actions in lotal, and the optimal action is 1. Without large replay buller, DQN(S) keeps forgetting and relearning the optimal action, while DQN and MeDQN(U) suffer much less from forgetting.
2 Understanding forgetting from an objective-mismatch perspective
In this section, we first use a simple example in supervised learning to shed some light on catastrophic forgetting from objective-mismach perspective. Let D be the whole training dataset. We denote the true objective function on D as (o:),,, where 0 is set of parameters time-step t. Denote B, as subset of D used for training at time-step L. We expect to approximate Ln(0,) with Ln, (0.). For example, B, could be mini-batch sampled from D. It could also be sequence of temporally correlated samples when samples in D come a stream. When B, is IID (c.g., sampling B, uniformly at random from D), there no objective mismatch since (0.) = Sn.-D[Ln, (0.)] for a specific However, when B, non-IID, is likely that (O.) * BRB T.p, (0.)], resulting in the objective mismatch problem. The mismatch between optimizing the true objective and optimizing the objective induced by Be often leads catastrophic forgetting. Without incorporating additional techniques, catastrophic forgetting is highly likely when SGD algorithms are used to train neural networks given non-IID data the optimization objective is simply wrong.
To demonstrate how objective mismatch can lead to catastrophic forgetting, we provide regression experiment. Specifically, a neural network was trained to approximate a sine function y = (in() where x E [0, 2]. To get non-IID input data, we consider two-stage training. In Stage 1, we generated training samples (x. y), where x E [o, 1] and y = sin(TT)). For Stage 2, [1, 2] and y = sin(..). The network was first trained with samples in Stage 1 in the traditional supervised learning style. After that, continued training the network in Stage 2. Finally,
plotted the learned function after the end of training for each stage. More details are included in the appendix.
As shown in Figure aa), after Stage 1, the learned function fits the true function sin(Tx)) on x E [0, 1] almost perfectly. At the end of Stage 2, the learned function also approximates the true function on æ e [1,2] well. However, the network catastrophically forgets function values Icarned on æ E [0. 1], due to input distribution shift. As stated above, the objective function we minimize is defined by training samples only from one stage (i.e. x E [0, 1] or [1)2) that are not enough to reconstruet the true objective properly
3

---

Published in Transactions on Machine Learning Research (04/2023)
which is defined on the whole training set (i.e. [0, Much of the previously learned knowledge is lost while optimizing a wrong objective.
A similar phenomenon also exists in single RL tasks. Specifically, we show that, without a large replay buffer, DQN can easily forget the optimal action after it has learned it. We tested DQN in Mountain Car (Sutton & Barto 2018), with and without using a large replay buffer. Denote DQN(S) as DQN using a tiny (32) experience replay buffer. We first randomly sampled state S = [00.70167244,, 0.04185214] and then recorded its greedy action (i.e., arg max, Q(S, a)) through the whole training process, as shown in Figure 2
Note that
the optimal action for this state is 1. We did many runs to verify the result and only showed the single-run result here for better visualization. We observed that, when using a large buffer (i.e. 10K), DQN suffered not much from forgetting. However, when using small buffer (i.e. 32), DQN(S) kept forgetting and relearning the optimal action, demonstrating the forgetting issue in single RL tasks. MeDQN(U) is our algorithm which will be introduced later in Section As shown in Figure 2 McDQN(U) consistently chose the optimal action as its groedy action and suffered much less from forgetting, even though also used tiny replay buffer with size 32. More training details can be found in Section 5.3 and the appendix.
3 Related works on reducing catastrophic forgetting
The key to reducing catastrophic forgetting is to preserve past acquired knowledge while acquiring new knowledge. In this section, we will first discuss existing methods for continual supervised learning and then methods for continual RL. Since we can not list all related methods here, we encourage readers to check
recent surveys [Khttarpa et al. 2020 Delange ct al. 2021
3.1 Supervised learning
In the absence of memory constraints, rehearsal methods, also known as replay, are usually considered one
of the most effective methods in continual supervisod lcarning (Kemker et al., 2018 Farquhar & Gal 2018
Van de Ven & Tolias 2019 Delange et al. 2021). Concretely, these methods retain knowledge explicitly
by storing previous training samples (Rebuffi et al. 2017: Riemer et al.] 2018 Hayes et al. 2019 Aljundi
et al.) 2019a; Chaudhry et al. 2019 Jin et al. 2020). In generative replay methods, samples are stored
in generative models rather than in a buffer (Shin et al.] 2017 Kamra et al. 2017: Van de Ven & Tolias
2018 Ramapuram el al. 2020 Choi et al., 2021 These methods exploit a dual memory system consisting of a student and a teacher network. The current training samples from a data buffer are first combined with pseudo samples generated from the teacher network and then used to train the student network with knowledge distillation (Hinton et al., (014)). Some previous methods carefully select and assign subset of weights in large network to each task (aallya & Lazebnik 2018 Sokar al. 2021 Fernando et al. 2017
Serra et al. 2018 Masana et al. 2021
2019 Yoon et al. 2018 assign task-specific network Aljundi et al.] 2017). Changing the update rule is another approach to
Li et al.
to each task (Rusu et al. 2016
reducing forgetting. Methods such as GEM (Lopez-Paz & RRanzato 2017), OWM (Zeng et al. 2019), and
ODG (Farajtabar ct al. 2020)
protect obtained knowledge by projecting the current gradient vector onto
some constructed space related to previous tasks. Finally, parameter regularization methods (Kirkpatrick
et al. 2017 Schwarz et al.,
2018 Zenke et al.) [2017] Aljundi et al.) 2019b) reduce forgetting by encouraging
parameters stay close to their original values with regularization term so that more important parameters are updated more slowly.
3.2
Reinforcement learning
RL tasks are natural playgrounds for continual learning research since the input is stream of temporally structured transitions (Khetarpal et al. 2020p. In continual RL, most works focus on incremental task lcarning where lasks arrive sequentially with clear lask boundaries. For this setting, many methods from continual supervised learning can be directly applied, such as EWC (Kirkpatrick et al. 2017) and MER (Riemer et al.) 2018). Many works also exploit similar idcas from continual supervised learning to roduce forgetting in RI. For example, Ammar et al. (2014) and Mendez al. (2020) assign task-specific parameters to each task while sharing reusable knowledge base among all tasks. Mendez et al. (2022) and Isele & Cosgun (0188)
4

---

Published in Transactions on Machine Learning Research (04/2023)
use experience replay buffer to store transitions of previous tasks for knowledge retention. The above methods cannot be applied to our case directly since they require either clear task boundaries large buffers. The student- dual memory system is also exploited in multi-task RL, inducing behavioral cloning by function regularization between the current network (the student network) and its old version (the teacher network) (Rolnick et al. 2019 Kaplanis et al., 2019 Atkinson et al. 2021]. J. Note that continual supervised learning, there are also methods that regularize a function directly (Shin et al., 2017: Kamra et al. 2017 Titsias et al., 2020). Our work is inspired by this approach in which the target Q neural network plays the role of the teacher network that regularizes the student network (i.e., current Q neural network) directly.
In single RL tasks, the forgetting issuc is under-explored and unaddressed, as the issue is masked by using a large replay buffer. In this work, we aim to develop memory-efficien single-task RL algorithms while achieving high sample efficiency and training performance by reducing catastrophic forgetting.
4 MeDQN: Memory-efficient DQN
4.1 Background
We consider RL tasks that can be formalized as Markov decision processes (MDPs). Formally, let M P,,,P, be an MDP, where S is the discrete state space [ A is the discrete action space, P S x A X + [0, 1] is the state transition probability function, r S x A + R is the reward function, and E [0, 1) is the discount factor. For a given MDP, the agent interacts with the MDP according to its policy i which maps a state to a distribution over the action space. At cach time-step t, the agent obscrves state S. E S and samples an action A, from (--(.)). Then it observes the next state S.41 S according to the transition function P and receives scalar reward R++1 r(,, A.) R. Considering we define the return G, as the sum of discounted rewards, that is, G, = -I h RR++1 function of policy æ is defined as (r(8,a) a) E |, 8, A, a), V(s,a) Exxxx The agent's 's goal is to find optimal policy that maximizes the expected return starting from some initial states.
episodic task with horizon T,, 2 The action-value
As an policy algorithm, Q-learning [aatkins 1989) is an effective method that aims to learn the action- value function estimate Q : S x A - R of an optimal policy. Mnih et al. (2015) propose DQN, which uses a neural network to approximate the action-value function. The complete algorithm presented as Algorithm| 2 in the appendix. Specifically, to mitigate the problem of divergence using function approximators such as neural networks, a target network Q is introduced. The target neural network Q is a copy of the Q-network and is updated periodically for every Ctarget steps. Between two individual updates, Q is fixed. Note that this idea is not limited to value-based methods; the target network is also used in policy gradient methods, such as SAC (Haarnoja et al. 2018) and TD3 (Fujimoto et al. 2018
4.2 Knowledge consolidation
Originally, Hinton et al. (2014) proposed distillation to transfer knowledge between different neural networks effectively. Here we refer to knowledge consolidation as special case of distillation that transfers information from an old copy of this network (e.g. the target network Q, parameterized by 8-) to the network itself (e.g. the current network Q, parameterized by 0), consolidating the knowledge that is already contained in the network. Unlike methods like EWC (Kirkpatrick et al. 2017) and SI (Zenke et al.] 2017)) that regularize parameters, knowledge consolidation regularizes the function directly. Formally, we define the (vanilla) consolidation loss as follows:
V. = B(S.A)<p(.)) ((0)),A:0)
where p(s, a) is a sampling distribution over the state-action pair 8, a. To retain knowledge, the state-action space should be covered by p(s, a) sufficiently, such as p(s. = d= (s)) or p(s, a) = d= (s),(a), where π is
We consider discrete state spaces for simplicity. In general, our method also works with continuous state spaces.
5

---

Published in Transactions on Machine Learning Research (04/2023)
the e-greedy policy, dl is the stationary state distribution of n, and is uniform distribution over A. Our preliminary experiment shows that p(s,a) d (s)j((a is a better choice compared with p(s, a) d (a|s; so we use p(s, a) = d (s) in this work. However, the optimal form of p(s, a) remains an open problem and we leave it for future research. To summariie, the following consolidation loss is used in this work:
Teconsolid = Esndd
Σ LAEA
(Q(S, A:0) 0(5.
(1)
Intuitively, minimizing the consolidation loss can preserve previously learned knowledge by penalizing Q(s,;; ((),;;)) for deviating from Q(s, a; 0-) too much. In general, may also use other loss functions, such where and &t can be softmax policies induced by action values, that is, T(ass) x exp(Q(s, a; 0)) and a(als) x exp(Q(s, a; ())). For simplicity, we use the mean squared error loss, which also proves to be effective shown in our experiments.
the Kullback-Leiberr (KL) divergence (
Given a mini-batch B consisting of Iransitions T = (8,a,r,8)) 8'), the DQN loss is defined as
LDQN = E ( E(++ max Q a's',a':07) a(s,,)))))
TEB
We combine the two losses to obtain the final training loss for our algorithms
L = LDQN + XLconsolids
where À is a positive scalar. No extra network introduced since the target network used for consolidation. Note that LDQN helps Q network learn new knowlodge from B that is sampled from the experience replay buffer. contrast, Teonsalid is used to preserve old knowledge by consolidating information from Q to Q. By combining them with a weighting parameter X, we balance learning and preserving knowledge simultaneously Morcover, since Leonsolid acts as functional regularizer, the parameter f may change significantly as long the function values Q remain close to Q.
There is still one problem left: how to get d= (s)? In general, it is hard to compute the exact form of d (s).s. Instead, we use random sampling, as we will show next.
4.3 Uniform state sampling
One of the simplest ways to approximate (s) is with a uniform distribution on S. Formally, we define this version of consolidation loss as
L I. Esnu
Σ
- A:0-))²
LAEA
where is a uniform distribution over the state space S. Assuming that the size of the state space [S] is finite, we have Pr(S s) 1/|8| for any S. Together with d= S 1, we then have
Loonsotid
ds)(s) Σ
-) ((s,a;;)))
865
REA
-) ((s,
865 aEA
|S|L
(2)
Essentially, minimizing |S S minimizes an upper bound of Leonsotid- As long as L is small enough, we can achieve good knowledge consolidation with a low consolidation loss Leansotid- In the extreme case, Llonsoti = leads to Leonsolid = 0.
In practice, we may not know S advance. To solve this problem, we maintain state bounds SLOW and SHIGH the lower and upper bounds of all observed states, respectively. Note that both SLOW and

---

Published in Transactions on Machine Learning Research (04/2023)
A Algorithm 1 Memory-ellicient DQN with uniform state sampling
Initialize small experience replay buffer D (e.g., one mini-batch size)
Initialize state lower bound SLOW joe.... , oo] and upper bound SHIGH [-00,-., Initialize action-value function with random weights 0
Initialize target action-value function Q with random weights O
5: Observe initial state
6: while agent is interacting with the environment do
7:
Update state bounds: SLOW = min($Low 8), SHIGH = max(SHIGH,
Take action a chosen by e-greedy based on Q, observe T; s Store transition (s,,,r,s) in D and update state s t
10:
for every Cearrent steps do
11:
Get all transitions (+e one mini-batch) (sD, ag $ in D
12:
for i = to E do
13:
Compute DQN loss LDQN Sample a random mini-batch states 8g uniformly from (LLOw,SHIGH Compute consolidation loss L Loonsttid given sp
14:
15:
16:
Compute the final training loss: L LDQN + AL consolid
17:
Perform a gradient descent step with respect end for
18:
19:
end for
20: Reset Q == Q for every Crarges steps
21: end while
SHIGH are two state vectors with the same dimension as a state S. Assume S R", Initially, we set SLOW = [oo...., 2o] R" and SHIGH = [-,,,.., -o0 E R". For each newly received E R", we update state bounds with
SLOW min(s and SHIGH max
Here, both min and max are element-wise operations. During training, we sample pseudo-states uniformly from the interval [sLow, 8HIGH] to help compute the consolidation loss.
We name our algorithm that uses uniform state sampling as memory-efficient DQN with uniform state sampling, denoted as McDQN((U) and shown in Algorithm [ Compared with DQN, McDQN(U) has several changes. First, the experience replay buffer /) is extremely small (Line 1). In practice, set the buffer size to the mini-batch size to apply mini-batch gradient descent. Second, maintain state bounds and update them at every step (Line 7). Moreover, to extract as much information from a small replay buffer, we use the same data to train the Q function for E' times (Line 13- 13-19). In practice, we find that a small E (e.g..1-4) is enough to perform well. Finally, we apply knowledge consolidation by adding a consolidation loss to the DQN loss as the final training loss (Line 17).
4.4 Real state sampling
When the state space S is super large, the agent is unlikely to visit every state S. Thus, for a policy w, the visited state set uniform distribution over S far from a good estimation of d=. mini-batch) generated from uniform state sampling cannot cover S well enough, resulting in poor knowledge consolidation for the Q function. In other words, we may still forget previously learned knowledge (i.e. the action values over ST x A) catastrophically. This intuition can also be understood from the view of upper bound minimization. In Equation 2 as the upper bound of Leonsolid, |S is large when S is large. Even if U comsotid is minimized to a small value, the upper bound [ consalid may still be too large, leading to poor consolidation.
{s SId (s) 0} expected to be a very small subset of S. In this case,
small number of states (o.g., one
To overcome the shortcoming of uniform state sampling, we propose real state sampling. Specifically, previously observed states stored in state replay buffer D. and real states are sampled from D, for
7

---

Published in Transactions on Machine Learning Research (04/2023)
knowledge consolidation. Compared with uniform state sampling, states sampled from state replay buffer have a larger overlap with S", acting as better approximation of d=. Formally, we define the consolidation loss using real state sampling as
Σ (Q(S,A:0) (Q(S.A0))Q((SAA0)))) Q(S.A:)0)))
L L consolid = ES
In practice, we sample states from the experience replay buffer D. We name our algorithm that uses real state sampling as memory-efficiet DQN with real state sampling, denoted as MeDQN(RR). The algorithm description is shown in Algorithm in the appendix. Similar to McDQN(U), we also use the same data to train the Q function for E times and apply knowledge consolidation by adding a consolidation loss. The main difference is that the experience replay buffer used in MeDQN(R) is relatively large while the experience replay buller in MeDQN(U) is extremely small (i.e., one mini-batch size). However, as we will show next, the experience replay buffer used in MeDQN(() can still be significantly smaller than the experience replay buffer used in DQN.
5 Experiments
In this section, we first verify that knowledge consolidation helps mitigate the objective mismatch problem and reduce catastrophic forgetting. Next, propose strategy to better balance learning and remembering. We then show that our algorithms achieve similar or even better performance compared to DQN in both low-dimensional and high-dimensional tasks. Moreover, we verify that knowledge consolidation is the key to achieving memory efficiency and high performance with an ablation study. Finally, we show that knowledge consolidation makes our algorithms more robust to different buffer sizes. All experimental details, including ppper-paraeete settings, are presented in the appendix.
5.1 The effectiveness of knowledge consolidation
To show that knowledge consolidation helps mitigate the objective mismatch problem, we first apply it to solve the task of approximating sin(TT), as presented in Section 2 Denote the neural network and the target neural network as (a;; f) and f(x; 0-), respectively. The true loss function is defined as Itrue = [ () where y sin(TT)). We first trained the network (ax; 0) in Stage 1 (iee. x E [0, 1]) without knowledge consolidation. We also maintained input bounds [ILow, THIGH), similar to Algorithm ! At the end of Stage 1, we set 0- + 0. At this moment, both f(x; 0) and fz;;0) were good approximations of the true function for x E [0, 1]; and [ILOw, [HIGH] x [0. 1]. Next, we continue to train f(z; () with x E [1, 2]. To apply knowledge consolidation, we sampled & from input bounds [ILOw,HHIGH uniformly. Finally, we added the consolidation loss to the training loss and got:
L(l)= [1 [[f(=;((())))))) y) + Exgl (f ()
By adding the consolidation loss, L became a good approximation of the true loss function Lirues which helps to preserve the knowledge learned in Stage 1 while learning new knowledge in Stage 2, as shown in Figure bb). Note that in the whole training process, we did not save previously observed samples (x, y) explicitly. The knowledge of Stage 1 is stored in the target model f(x;8-) and then consolidated to f(x; 0).
Next, show that combined with knowledge consolidation, McDQN(U) is able to reduce the forgetting issue in single RL tasks even with tiny replay buffer. We repeated the RL training process in Section 2 for McDQN(U) and recorded its greedy action during training. As shown in Figure 2 while DQN(S) kept forgetting and relearning the optimal action, both DQN and MeDQN(U) were able to consistently choose the optimal action as its greedy action, suffering much less for forgetting. Note that both MeDQN(U) and DQN(S) use tiny replay buffer with size 32 while DQN uses a much larger replay buffer with size 10K. This experiment demonstrated the forgetting issue in single RL tasks and the effectiveness of our method to mitigate forgetting. More training details in Mountain Car can be found in Section 5.3 and the appendix.
8

---

Published in Transactions on Machine Learning Research (04/2023)
-200
Average Relum
-400-
Average Return
-200
-920
A-0.01
Am0.1
-000
-900
==2.0
4,4
-350
DON
=
McDON(U) (m=32)
-1000
Am0.01-4
20000
40000
60000
00000
100000
16'
10"
10'
Step
Log
(a) M))QN(U)
(b) The influence of buffer size m.
Figure 3: (a) Comparison of different strategies to balance learning and preservation for MeDQN(U) CorntainCar-vo. (b) A study of robustness to different buffer sizes m in MountainCar-v0. For MeDQN(U), m is fixed as the mini-batch size 32. DQN performs worse and worse as we decrease the buffer size while McDQN(U) achieves high performance with an extremely small buffer. All results were averaged over 20 runs, with the shaded area representing two standard errors.
5.2 Balancing learning and remembering
In Section 1.2 we claimed that À could balance between learning new knowledge and preserving old knowledge. In this section, used Mountain Car (Sutton & Barto 2018) as testbed to verify this claim. Specifically, fixed À was chosen from {0.01, 0,1, 4, 8J. The mini-batch size was 32. The experience replay bulfer size in MeDQN((U was 32. The update epoch F. = 4, the target network update frequency Ctarget =100, and the current network update frequency Ceurrent = 1. We chose learning rate from {1e-2, 2,e-2,32 3e-3, le 1e-3, 3e.33--44 le-4] and reported the best results averaged over 20 runs for different X, as shown in Figure B(a).
When small (e-g., A = or X 0.01), giving a small weight to the consolidation loss, McDQN(U) learns fast at the beginning. However, as training continues, the learning becomes slower and unstable; the performance drops. As we increase X, although the initial learning is getting slower, the training process becomes more stable, resulting in higher performance. These phenomena align with our intuition. At first, not much knowledge is available for consolidation; learning new knowledge is more important. A small X lowers the weight of Leonsotid in the training loss, thus speeding up learning initially. As training continues, more knowledge is learned, and knowledge preservation is vital to performance. A small fails to protect old knowledge, while larger helps consolidate knowledge more effectively, stabilizing the learning process.
Inspired by these results, we propose a new strategy to balance learning and preservation. Specifically, À longer fixed but linearly increased from small value Astart to a large value Aend- This mechanism encourages knowledge learning at the beginning and information retention in later training. We increased À from 0.01 to 4 linearly with respect to the training steps for this experiment. In this setting, we observed that learning is fast initially, and then the performance stays at a high level stably towards the end of training. Given the success of this linearly increasing strategy, we applied in all of the following experiments for MeDQN.
5.3 Evaluation in low-dimensional tasks
We chose four tasks with low-dimensional inputs from Gym Brockman et al. 2016 and PyGame Learning Environment [aasi] 2016): MountainCar-vo (2), Acrobot-vi (6), Catcher (4), and Pixelcopter (7), where numbers in pasentheses are input state dimensions. For DQN, we used the same training settings as in Lan et al. (2020). The mini-batch size was 32. Note that the buffer size for DQN was 10. 000 in all tasks. Moreover, we included DQN(S) as another bascline which used tiny (32) experience replay buffer. The buffer size the only difference between DQN and DQN(S). The experience replay buffer
and
9

---

Published in Transactions on Machine Learning Research (04/2023)
00630
20401
Okep
Stap
(a) Mar-tainCar-v0
(b) Acrobot-v1
(c) Catcher
(d) Pixclcopter
Figure& 4: Evaluation low-dimensional tasks. The results for MountainCar-vo and Acrobot-vl were averaged over 20 runs. The results for Catcher and Pixclcopter were averaged over 10 runs. The shaded arcas represent two standard errors. MeDQN(U) outperforms DQN in all four tasks, even though it uses an experience replay buffer with one Mini-batch size.
Stap
(a) Asterix (MinAtar)
(b) Breakout (MinAtar)
(c) Space Invaders (MinAtar)
(d) Seaquest (MinAtar)
Figure 5: Evaluation in high-dinensional tasks. All results were averaged over 10 runs, with the shaded areas representing two standard errors. McDQN(R) comparable with DQN even though it uses a much smaller experience replay buffer (10% of the replay bufler size in DQN).
size for MeDQN(() was 32. We set Astart 0.01 for MeDQN(U) and tuned learning rate for all algorithms. Except for tuning the update epoch E, the current network update frequency Ceurrent: and Xend, we used the same hyper-parameters and training settings as DQN for MeDQN(U). More details are included in the appendix.
For MountainCar-vl and Acrobot-vl, we report the best results averaged over 20 runs. For Catcher and Pixelcopter, we report the best results averaged over 10 runs. The learning curves of the best results for all algorithms are shown in Figure where the shaded area represents two standard errors. As we see from Figure when using small buffer, DQN(S) has higher instability and lower performance than DQN, which uses a large buffer. McDQN(U) outperforms DQN in all four games. The results are very inspiring given that MeDQN(U) only uses a biny experience replay buffer. We conclude that MeDQN(U) is much more memory-efficient while achieving similar or better performance compared to DQN in low-dimensional tasks. Given that MeDQN(U) already performs well while being very memory-efficient, we did not further test MeDQN(R) in these diveniinniional tasks.
5.4 Evaluation in high-dimensional tasks
To further evaluate our algorithms, chose four tasks with high-dimensional image inputs from Mi- nAtar (Young & Tian 2019): Asterix (10 x 10 4), Seaquest (10 x 10 10), Breakout (10 x 10 x 4), and Space Invaders (10 x 10 x 6), where numbers in parentheses are input dimensions. For DQN, we reused the lyper-parameters and neural network selting in Young & Tian (2019). The mini-batch size was 32. For DQN, the buffer size was 100, 000 and the target network update frequency Ctarget = 1, 000. For MRDQN(R), Crarget 1, 000 and the buffer size was 10% of the buffer size in DQN, i.c. 10, 000. The experience replay buffer size for McDQN(U) was 32. For McDQN(U), a smaller Crarget was better, and we set Ctarget 300. We set Astart- 0.01 for MeDQN(R) and MU)QN(U) and tuned learning rate for all algorithms. We tuned
10

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: An ablation study of knowlodge consolidation for McDQN(() with different buffer sizes, tested in Seaquest (MinAtar). All final returns were averaged over 10 runs, shown with one standard error.
Buffer Size
1e5
1e4
1e3
with consolidation
25.82 + 1.69
27.92 + 1.53
17.22 + 0.90
w.o. consolidation
20.10 + 0.71
19.78 + 1.19
16.04 + 0.60
Table 2: A study of robustness different buffer sizes, tested in Seaquest (MinAtar). All final returns were averaged over 10 runs, shown with one standard error.
buffer size
1e5
1e4
1e3
MeDQN(R)
25.82 1.69
27.92
1.53
17.22
0.90
DQN
20.48 0.75
21.38
0.80
15.04 + 0.66
the update epoch E, the current network update frequency Ceurvents and Xend for both McDQN(() and MeDQN(U), while reusing other hyper-parameters from DQN. More details are in the appendix.
The learning curves of best results shown in Figure 5 averaging over 10 runs with the shaded areas representing two standard errors. Notice that with small experience replay buffer, DQN(S) performs much worse than all other algorithms, including MeDQN(U), which also uses a small experience replay buffer of the same size as DQN(S). For tasks with a relatively lower input dimension, such as Asterix and Breakout, both MeDQN(R) and MeDQN(U) match up with or even outperform DQN. For Seaquest and Space Invaders which have larger input dimensions, McDQN(R) smaller experience replay buffer (10% of the replay buffer size in DQN). Meanwhile, the performance of MU) is significantly lower than MeDQN(R), mainly due to different state sampling strategies. As discussed in Section 4.4) when the state space S is too large as the cases for Seaquest and Space Invaders, states generated with uniform state sampling can not cover visited states well enough, resulting in poor knowledge consolidation. In this circumstance, storing and sampling real states is usually a better approach. Overall, we conclude that MeDQN(R) is more memory-efficient than DQN while achieving comparably high performance and high sample efficiency.
comparable with DQN even though uses a much
5.5 An ablation study of knowledge consolidation
In this section, we present an ablation study of knowledge consolidation in Seaquest. By setting Astart = Aend = 0 MeDQN(R), we removed the consolidation loss from the training loss and MeDQN(() was roduced DQN with E updates for cach mini-batch of transitions. Except this, all other training settings were the same as MeDQN(R) with consolidation. For example, we also tuned E and Ceurrent for MeDQN((R) without consolidation. The averaged final returns with standard errors presented in Table 1 which shows that MeDQN(R) performs better with consolidation than MeDQN(() without consolidation under different buffer sizes. These results proved that knowledge consolidation is the key to the success of McDQN(R); MeDQN(R) is not simply benefiting from choosing a higher or a larger Courrent- It is the use of knowledge consolidation that allows MeDQN(() to have a smaller experience replay buffer.
5.6 A study of robustness to different buffer sizes
Morcover, we study the performance of different algorithms with different buffer sizes on Seaquest (MinAtar), which has the largest input state space. In Table 2 we present the averaged returns al the end of training for (RDQN(R) and DQN with different buffer sizes. Clearly, MeDQN(R) is more robust than DQN to buffers at various scales. similar study in a low-dimensional task (MountainCar-vo) is shown in Figure 3(b). Note that for McDQN(U), m fixed as the biti- size 32. DQN performs worse and worse as we decrease the buffer size, while MeDQN(U) achieves high performance with tiny buffer.
11

---

Published in Transactions on Machine Learning Research (04/2023)
Battlezone
Double Dunk
Name This Game
40000
10000
30000
-2
5000
20000
10000
0
10
20
30
40
50
Frame (millions)
Phoenix
Q*bert
6000
15000
4000
10000
2000
5000
0
-2000
-5000
10
Frame (millions)
Frame (millions)
DQN (m=1e6)
DQN (m=1e5)
MeDQN(R) (m=1e5)
Figure 6: Return curves of various algorithus five Atari tasks over 50 million training frames, with different. buffer sizes m. Solid lines correspond to the median performance over 5 random seeds, and the shaded areas correspond to standard errors. Though using a much smaller buffer, McDQN(R) (m 1c5) outperforms DQN (m 1e6) in three tasks significantly (i..,, Name This Game, Phoenix, and Qbert) and matches DQN's performance in the other two tasks.
5.7 Additional results in Atari games
To further evaluate our algorithm, we conducted experiments in Atari games (Belemaree et al.) 2013). Specifically, we selected five representative games recommended by Aitchison et al. (0022), including Battlezone, Double Dunk, Name This Game, Phoenix, and Qbert. The input dimensions for all five games 84 x 84.
We compared our algorithm McDQN(R) and DQN with different buffer sizes. Specifically, we used the implementation of DQN y in Tianshou [Weng et al. 2022) and implemented MeDQN(R) based on Tianshou's 's DQN. For DQN, we used the default seper-parameters of Tianshou's DQN. For MeDQN, Astars = 0.01; Aend was chosen from (2,4); Tianshou's DQN. Except those hyper-parameters, other hyper-parameters of MeDQN the same as the default hyper-parameters of Tianshou's DQN.
We chose Courrent in {20, 40} while the default Courrent = 10
We train all algorithms for 50M frames (i.e., 12.5M steps) and summarize results across over 5 random seeds in Figure 6 The solid lines correspond to the median performance over 5 random seeds, while the shaded areas represent standard errors. Though using a much smaller buffer, MeDQN(()) (m = 1e5) outperforms DQN (m = 1e6) in three tasks significantly (i.e., Name This Game, Phoenix, and Qbert) and matches DQN's performance in the other two tasks. Overall, the memory usage of replay buffer reduced from 7GB 0.7GB without hurting the agent's performance, confirming our previous claim that MeDQN(R) more memory-efficient than DQN while achieving comparably high performance.
In the appendix, we report the actual memory usage and the training wall-clock time of training in four MinAtar games and five Atari games. Generally, MeDQN(U) and MeDQN(R) are more computation-effiics since they require less frequent updates due to less forgetting. Considering all above results, we suggest using MeDQN(U) for low-dimensional value-based control tasks and MeDQN(R) for high-dimensional value-based control tasks.
12

---

Published in Transactions on Machine Learning Research (04/2023)
6 Conclusion and future work
In this work, we proposed two memory--fficient algorithms based DQN. Our algorithms find good trade-off between learning new knowledge and preserving old knowledge. By conducting rigorous experiments, we showed that a large experience replay buffer could be successfully replaced by (real or uniform) state sampling, which costs less memory while achieving good performance and high sample efficiency.
There are many open directions for future work. For example, a pre-trained encoder can be used to reduce high-dimensional inputs low-dimensional inputs (Hayes al., 2019 2020 Chen al. (021), boosting the performance of MeDQN(U) on high-dinensional tasks. Combining classic memory-saving methods (Schlegel et al. 2017 with our knowledge consolidation approach may further reduce memory requirement and improve sample efficiency. Generative models (e.g., VAE (Kingma & Welling 2013) or GAN (Goodfellow et al. 2014 ) could be applied to approximate d= The challenge for this approach would be to generate rcalistic pseudo-states to improve knowledge consolidation and reduce forgetting. It is also worth considering the usage of various sampling methods for the knowledge consolidation loss, such as stochastic gradient Langevin dynamics methods (Pan et al., 2022b 2020). A combination uniform state sampling and real state sampling might further improve the agent's performance. Finally, extending our ideas policy gradient methods would also be interesting.
Acknowledgments
We gratefully acknowledge funding from the Canada CIFAR AI Chairs program, the Reinforcement Learning and Artificial Intelligence (RLAI) laboratory, the Alberta Machine Intelligence Institute (Amii), and the Natural Sciences and Engineering Research Council (NSERC) of Canada.
References
Joshua Achiam. Spinning up in deep reinforcement learning. 2018.
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. arXiv preprint arXiv:2210.0000 2022.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 2019a.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In International Conference on Learning Representations, 2019b.
Bastian Alt, Adrian Šošić, and Heinz Koeppl. Correlation priors for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Ouline multi-task learning for policy gradient methods. In International conference on machine learning, 2014.
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-rchearsal: Achieving deep reinforcement learning without catastrophic forgetting. Neurocompuling, 2021.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279. 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAT Gym. arXiv preprint or Xix: 01540, 2016.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. ar preprinl arXiv:1902.14486, 2019.
Lili Chen, Kimin Lee, Aravind Srinivas, and Picter Abbeel. Improving computational efficiency in visual reinforeement learning via stored emboddings. Advances in Neural Information Processing Systems, 2021.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual-teacher class-incremental learning with data- free generative replay. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Colin Cooper. On the rank of random matrices. Random Structures Algorithms, 2000.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytclaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Mehrdad Farajtabar, Navid Azizan, Alex Molt, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, 2020.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning, ar preprint arXiv: 905.09733, 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution channels gradient descent in super neural networks. arXiv preprinl Xrx: 170100874, 2017.
Robert M French. Catastrophic forgetting in connectionist. networks. Trends in cognilive sciences, 1999.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 2018.
Sina Ghiassian, Banafsheh Rafice, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020.
Ian Goodfellow, Jean Couget-Abadie. Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Aduances in Neural Information Processing Systems, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, In International Conference on Machine Learning, pp. 1861-1870, 2018.
Tyler L Hayes and Christopher Kanan. Online continual lcarning for embedded devices. arXie preprint 220: 10681, 2022.
Tyler Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In International Conference on Robotics and Automation, 2019.
Tyler I, Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European Conference on Computer Vision, 2020.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrp,, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Aduances in Neural Information Processing Systems, volume 28, pp. 2944-2951 2015.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning Workshop, 2014.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.124 12488, 2018.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient based memory editing for task-free continual learning. arXiv preprint arXi::006. 15294, 2020.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint vcXic:1100110300 2017.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. In International Conference on Machine Learning, 2019.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. arxiv preprint or orxiv:0012 13490, 2020.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint Xr11:1311.6611, 2013.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Qingfeng Lan, Yangchen Pan, Alona. Fyshe, and Martha White. Maxmin Q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 1992.
David Lopez- Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks to a single network by iterative pruning.
In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018.
Mare Masana, Tinne Tuytelaars, and Joost Van de Weijer. Ternary feature masks: zero-forgetting for task-incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
Jorge Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (04/2023)
Jorge A Mendez, Harm van Seijen, and Eric Eaton. Modular lifelong reinforcement learning via neural composition. In International Conference on Learning Representations, 2022.
Volodymyr Mnih, Koray Kavukcuogiu, David Silver, Alex Graves, and loannis Antonoglou. Playing Atari with deep reinforcement learning. In NIPS Deep Learning Workshop, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Toannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Yangchen Pan, Jincheng Mci, and Amir massoud Farahmand. Froquency-based scarch-control in dyna. In International Conference on Learning Represenlations, 2020.
Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online, In International Conference on Learning Representations, 2022a.
Yangchen Pan, Jincheng Mei, Amir massoud Farahmand, Martha White, Hengshuai Yao, Mohsen Rohani, and Jun Luo. Understanding and mitigating the limitations of prioritized experience replay. In Conference Uncertainty in Artificial Intelligence, 2022b.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neurocompul- ing, 2020.
Sylvestro-Alvisc Rebuffi, Alexander Kolesnikov, Goorg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Compuler Vision and Pottern Recognition, pp. 2001-20101 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Trina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Represenlalions, 2018.
David Rolnick, Arun Ahuja, Jomathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Newral Information Processing Systems, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaune Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 156::6606004611 2016.
Matthew Schlegel, Yangchen Pan, Jiecao Chen, and Martha White. Adapting kernel representations online using submodular maximization. In International Conference on Machine Learning, 2017.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar arXi::1707.066: 2017.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinske Barwinska, Yee Whye Teh, Razvan Pascamu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, 2018
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, 2018.
16

---

Published in Transactions on Machine Learning Research (04/2023)
Hamul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural informalion processing systems, 2017.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning walk in 20 minutes with model-free reinforcement learning. arXiv preprint or arxi:2208.. Xi::2208.77860, 2022.
Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. SpaceNet: Make free space for continual learning. Neurocomputing, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, second edition, 2018.
Norman Tasfi
PyGame learning
environment.
https://giichb..commttaass
PyGame- Envirommntt 2016.
Michalis K Titsias, Jonathan Schwarz, Alexander de Matthews, Razvan Pascanu, and Yee Whye Teh.
Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
Gido M Van de Ven and Andreas S Tolias. Generative replay with feedback connections as a general strategy for continual learning. orXiv preprint arXiv:1809.1006 2018.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1944.077 077944 2019.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference artificial intelligence, 2016.
Yan Wang, Gautham Vasan, and Rupam Mahmood. Real-time reinforcement lcarning for vision-based robolics utilizing local and remote computers. In International Conference on Robolics and Aulomation, 2023.
Chris Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, 1989.
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 2022.
Jachong Yoon, Eunho Yang, Jcongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Kenny Young and Tian Tian. MinAtar: An Atari-inspired testbed for more efficiont reinforcement learning experiments. arXiv preprint ar xrx:::1900 031060 2019.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
17

---

Published in Transactions on Machine Learning Research (04/2023)
A The pseudocodes of DQN and MeDQN(R)
Algorithm 2 Deep Q-learning with experience replay (DQN)
Initialize an experience replay buffer D
Initialize action-value function with random weights 0
Initialize target action-value function Q with random weights 0 0
Observe initial state
5: while agent is interacting with the environment do
6: Choose action a by c-groedy based on Q
7: Take action a, observe T, 8 Store transition (8, 0, r, 8') D and update state 8 <- 8
for every Cearrent steps do
10:
Sample random mini-batch of transitions (sB,ag,TB,$ s'g) from D
Get update target: YB + TB + max a'' 0))
11:
12:
Compute DQN loss: LDQN (YB - ((sB, aB; 0))²
13: Perform gradient descent step LDQN with respect to 0
14: end for
15:
Reset Q ==Q Q for every Ctarget steps
16: end while
Algorithm Memory-efficiet DQN with real state sampling
Initialize experience replay buffer D
2: Initialize action-value function Q with random weights 0
3: Initialize target action-value function & with random weights 0 = 0
4: Observe initial state
5: while agent interacting with the environment do Choose action a by c-groedy based on Q
Take action a, observe r; 8/ Store transition a,, in D and update state <-
for every Courrent steps do Sample a random mini-bas batch of transitions (sB, aB, TB, SB, from D
10:
11:
for i to E do
12:
Compute DQN loss LDQN Sample a random mini-batch of states 8n from Compute consolidation loss L Consotid given sg Compute the final training loss: L LDQN ALR consolid
13:
14:
15:
16:
Perform a gradient descent step L with respect to 0 end for
17:
18:
end for
19:
Roset Q = Q for every Ctarget steps
20:
end while
B The relation between knowledge consolidation and trust region methods
Trust region methods such TRPO (Schulman et al. 2015 and PPO (Schulman et al. 2017 use KL divergence to prevent large policy updates. In their implementations [Achiam 2018), the same states are used for TD learning and policy constraints, while we sample different states in our methods. Thus, policy constraints trust region methods are close to knowledge consolidation with real state sampling where the sampled states for policy constraints are the same as the states for TD learning. Idcally, we only want to update the Q values and policy of states used in TD learning: the rest of Q values and policy should remain the same. Using the states for TD learning and policy constraint cannot achieve this.
reduce forgetting,
18

---

Published in Transactions on Machine Learning Research (04/2023)
C Implementation details
C.1 Supervised learning:
approximating a sine function
The goal of this task is to approximate sine function sin(sa)), where E [0, To get non-IlD input data, we consider two-stage training. In Stage 1, we generated training samples (æ, y), where E [0, 1] and = sin For Stage 2, x E [1,2] and y= sin(xr)). The neural network was a multi-layer perceptron with hidden layers [32, 32] and ReLU activation functions. We used Adam optimizer with learning rate 0.01. The mini-batch size was 32. For each stage, we trained the network with 1, 000 mini-batch updales.
C.2 Low-dimensional tasks
For CountainCar-vD and Acrobot-vl, the depicted return in Figure was averaged over the last 20 episodes; the neural network was a multi-layer perceptron with hidden layers [32, 32]; the best learning rate was selected from fle- - 2, 3e 3, le 3, 3e - 4, 1e - 4} with grid search; Adam was used to optimize network parameters; all algorithms were trained for 1e5 steps.
For Catcher and Pixelcopter, the depicted return in Figure 4 was averaged over the last 100 episodes; the neural network
multi-layer perceptron with hidden layers [64, 64]; the best learning rate selected from (1e-3, 3, 3e 4, le - 4, 3e - 5, le-5) with grid search; RMSprop was used to optimize network parameters. In Catcher, algorithms were trained for 1.5c6 steps; in Pixelcopter, algorithms were trained for 2e6 steps.
All curves were smoothed using an exponential average. The discount factor was 0.99. The mini-batee bani-batch size was 32. For first 000 exploration steps, we only collected transitions without learning. e-greedy was applied as the exploration strategy with decreasing lincarly from 1.0 to 0.01 in 1, 000 steps. After 1, 000 steps, fixed to 0.01. For MeDQN, Astart 0.01; Aend was chosen from (1,2, 4); was selected from {1, 4). We chose Ceurrent in {1,2, 4, 8,8 Other hyper-parameter parameter choices are presented in Table 3
Table 3: The Hyper-parameter
different algorithms for MountainCar-v0,, used in Figure 4(a).
DQN DQN(S) MeDQN(U)
learning rate
le-2
le-3
le-3
experience replay buffer size
le4
32
32
4
Aend
/
4
Ctargat
100
100
100
Courrent
8
1
Table 4: The hyper-parameters of different algorithms for Acrobot-v1, used in Figure 4b).
Hyper-parameter
DQN DQN(S) MeDQN(U)
learning rate
le-3
3e-4
3e-4
experience replay buffer size
le4
32
32
E
1
Xend
2
Ctorget
100
100
100
Courrent
1
1
1
C.3
tasks:
MinAtar
The depicted return Figure was averaged over the last 500 episodes, and the curves were smoothed using an exponential average. In Seaquest (MinAtar), all algorithms were trained for 1e7 steps. Except that, all algorithms trained for 5e6 steps in other tasks. The discount. factor 0.99. The mini-batch size was 32.
19

---

Published in Transactions on Machine Learning Research (04/2023)
Table 5: The hyper-parameters of different algorithms for Catcher used Figure (c).
Hyper-parameter
DQN DQN(S) MeDQN(U)
learning rate
lc-4
lc-5
3c-5
experience replay buffer size
le4
32
32
/
4
Aend
4
Ctarget
200
200
200
Ceurrent
2
Table 6: The hyper-parameters of different algorithms for Pixelcopter, used in Figure
Hyper-parameter
DQN
DQN(S) MeDQN(U)
lcarning rate
30-5
lc-5
3e-4
experience replay buffer size
led
32
32
E Aend
1
/
1
Crarget
200
200
200
Ceurrent
8
For the first 5, 000 exploration steps, we only collected transitions without learning. e-greedy was applied as the exploration strategy with c decrcasing lincarly from 1.0 0.1 in 000 steps. After 5, 000 stops, c was fixod to 0.1. Following Young & Tian (211)), we used smooth L1 loss in PyTorch and centered RMSprop oplimizer with 0.95 and 0.01. The best learming rate chosen from (3e - 3, 1e 3, 3e 4, le 4, 3e 5} with grid scarch. We also reused the settings of neural networks. For McDQN, Astart = 0.01; Xend was chosen from f2,4); E was selected from {1,2}. We chose Ceurrent in {4, 16, 32}. Other hyper-pan parameter choices are presented in Table 7 10
Table 7: The hyper-parameters of different algorithms for Asterix (MinAtar), used in Figure 5().
Hyper-parameter
DQN DQN(S) MeDQN(U) MeDQN(R)
learning rale experience replay buffer size
le-4
3e-5
1e-3
3e-4
le5
32
32
le4
E
2
2
Aend
2
2
Otarget Ccurrent
1000
1000
300
1000
1
1
16
8
Computation and memory usage
For experiments, used computation resources (about CPU core years and 0.25 GPU core year) from
the Digital Research Alliance of Canada.
Table 11 presents a complete comparison of buffer sizes for different algorithms. Table 12 presents a
comparison of actual memory usage of training different algorithms for Seaquest (MinAtar). Table 13 presents
a comparison of actual memory usage of training different algorithms Atari games. Table 14 presents a
comparison of actual wall-clock time of training different algorithms for MinAtar games. Table presents
comparison of training speed of different algorithms for Atari games.
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table 8: The hyper-parameters of different algorithms for Breakout (MinAtar), used in Figure 5 b).
Hyper-parameter
DQN
DQN(S)
MeDQN(U) MeDQN(R)
learning rate experience replay buffer size
le-3
3e-5
3e-3
3e-4
le5
32
32
le4
E
/
Xend
Ctarget
1000
1000
300
1000
Ceurrent
1
Table 9: The hyper-parameters different algorithms for Space Invaders (MinAtar), used in Figure 5 cc.
Hyper-parameter
DQN DQN(S) MeDQN(U) MeDQN(R)
learning rate
3e-4
3e-5
1e-3
3e-4
experience replay buffer size
1e5
32
32
le4
E
Xend
Ctarget
1000
1000
300
1000
Caurrent
1
32
Table 10: The hyper-parameterr of different algorithms for Seaquest (MinAtar), used in Figure d).
Hyper-parameter
DQN DQN(S)
MeDQN(U) MeDQN(R)
learning rate experience replay buffer size
le-4
3e-5
3e-3
3e-4
1c5
32
32
lc4
E
Aend
Ctarget
1000
1000
300
1000
Ceurrent
1
32
Table 11: Comparison of buffer sizes low- and high-dimensional tasks: MinAtar. Compared DQN which large experience replay buffer, MeDQN is much more memory-efficient while achieving comparable performance.
Task
DQN DQN(S)
MeDQN(U) MeDQN(R)
low-dimeensinnal
10K
32
high-dimensional: MinAtar
100K
32
32
10K
high-dimensional:
Atari
1M
100K
21

---

Published in Transactions on Machine Learning Research (04/2023)
Table 12:
Comparison of actual memory usage of different algorithms Seaquest (MinAtar). Here, the actual memory usage refers to the resident set size (including experience replay buffer, neural networks, etc.), obtained with Python package psulil.
DQN DQN(S) MeDQN(U) MeDQN(R)
Memory (MB)
1400
240
250
280
Table 13: Comparison of actual memory usage of different algorithms in Atari games. Here, the actual memory usage refers to the resident set size (including experience replay buffer, neural networks, etc.), obtained with Python package psutil.
DQN (m 1e6)
DQN (m 1e5)
MeDQN(R) (m 1e5)
Memory (GB)
8.9
2.9
3.1
E
Why uniform state sampling works
The intuition behind uniform state sampling is that sometimes we do not necessarily need real states (i.e. states sampled from d²)) to achieve good knowledge consolidation. Although randomly uniformly generated states may not be meaningful, they are enough to induce perfect knowledge consolidation (i.e. Leonsolid = 0) in some cases, For example, considering linear function approximations that Q(s, a; 0) = x =o and Q(s, a; a(ssa;;0)) =)= 0 where 0 E IR", f E IR"" and x = (s) E IR.. To achieve perfect knowledge consolidation (Lconsolid 0), it is required to find = e by minimizing Leonsolid. Let {.1. Z2,..., .n be n points randomly uniformly A and set yi = x. A- for every Denote X [.i; ; and Y = "',,,",, ",m] The
sampled from problem can be reformulated as finding such that X _ Y. It is known that the random matrix X is full rank with a high probability (Cooper 2000p. In this case, we get the optimal 0 with 0 (XT)-Iy and thus achieve perfect knowledge consolidation (Iconsolid = 0). Note that {21, 22, are all randomly generated; they are not necessarily to be real state-action pairs sampled from real trajectories.
Table 14: The wall-clock training time (in hours) of different algorithms on four MinAtar games for results shown in Figure MeDQN(U) and MeDQN(R) take less time to train and are more computation-efficcent than DQN.
Task
DQN DQN(S) McDQN(U) McDQN(R)
Asterix (MinAtar)
3.83
3.68
1.17
1.78
Breakout (MinAtar)
3.83
3.68
1.17
2.67
Space Invaders (MinAtar)
4.08
3.83
0.75
3.33
Seaquest (MinAtar)
9.91
8.25
1.33
4.67
22

---

Published in Transactions on Machine Learning Research (04/2023)
Table 15: The training speed (step/s) of different algorithms five Atari games for results shown in Figure 6 MeDQN(R) faster to train and more computation- efficiient than DQN.
Task
DQN (m 1e6)
DQN (m 1e5)
MeDQN(R) (m 1e5)
Battlezone (Atari)
230
230
330
Double Dunk (Atari)
230
230
350
Name This Game (Atari)
240
240
350
Phoenix (Atari)
270
270
320
Qbert (Atari)
240
240
280
23


Reviews:
[{"review_id": "Gbxhwdgi4I", "reviewer": "Reviewer_h7DU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:  \n\n1. The paper is well organized and clearly written. It is generally easy to follow and the readers are well-informed. The two examples in Figures (1) and (2) clearly illustrate the ‘catastrophic forgetting’ in reinforcement learning. \n\n2. Memory constraint is a less-explored problem in our deep RL community. The findings presented by the paper may be valuable for future research on memory-efficient RL. \n\n3. Related works on reducing ‘catastrophic forgetting’ are adequately cited and discussed.  \n\n\nWeakness: \n\n1. The reviewer found the paper is not well-motivated. It is unclear to the reviewer in what real-world situation do we need to train an RL agent in an edge device. Giving more examples and discussion may help the reader better understand the motivation. \n\n2. The reviewer has some concern on the proposed uniform state sampling method. It seems that the uniform state sampling method only works in environments with very low-dimensional numberical observation / state. However, when the observation is low-dimensional, why should one worry about the memory issue of the replay buffer? \n\n3. The reviewer found the experimental results very limited. Besides the low-dimensional simple control tasks, the authors only report results on four out of more than fiftyAtari tasks. It is unclear if the proposed approach could scale to more complex tasks. In addition, the main topic of the paper is ‘memory efficiency’. However, there is no memory usage comparison provided in the experimental section.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies training reinforcement learning agents in a low-memory budget setting. Specifically, The authors pointed out that the replay memory of off-policy RL algorithms is critical for preventing ‘catastrophic forgetting.’ In this paper, the authors aim to alleviate the ‘catastrophic forgetting’ without using a large replay buffer. To address the problem, the authors proposed to distill knowledge from a target Q network into a current Q-network. The proposed approach is evaluated on low-dimensional control games and a subset of Atari games. The experimental results show that the proposed approach achieves competitive performance with a much smaller replay buffer. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following changes might help strengthen the paper:\n\n1. Better motivate the setting of low-memory RL. Giving more examples and discussion may help the reader better understand the motivation. For instance, in what kind of real world problems a large replay buffer would be an issue. \n\n2. Reporting experimental results on more Atari games or environments with higher dimensional observation. For instance, embodied AI environments such as robo thor, have high-dimensional and multimodal observation. Environment with high-dimensional observation could better justify the need of memory-efficient RL. \n\n3. Reporting the memory usage of the proposed methods and baselines. \n\n4. It is unclear to the reviewer how to apply the proposed uniform state sampling to states that contain discrete tokens. For instance, suppose the observation consists of description in text. How to define the min and max? How to sample a reasonable pseudo-state? Please clarify. \n", "broader_impact_concerns": "The reviewer has no concern on the ethical implications of this paper.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QyTCKKu733", "reviewer": "Reviewer_xdVG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- originality: the idea can be considered novel\n- quality: hyperparameters discussed, extensive experiments and ablations, good explanation of things, many technical details reported, overall good quality. \n- clarity: overall clear writing\n- significance: the problem is important, reducing memory usage will allow more practical usage of DRL especially deploying to portable devices. The results seem to be good too. \n\n**Weaknesses**\n- Overall the paper is nicely written. My main concern is on related work, there is no comparison to any other techniques in the literature to address the forgetting problem. While the proposed method works very well compared to the naive baseline, how does it compare to other related works? \n- Additionally, I have some questions: \n- in the paper it is shown the proposed method has faster computation, authors argue this is due to less frequency in updating the networks. I'm not sure I am convinced because the proposed method takes multiple updates and computes additional loss which will bring in extra computation. More detailed explanation on why it is faster is required here. \n- Some other questions and suggested change see next section. \n\n**Summary**\nOverall I like the paper, but I think there should be comparison to not just the very naive baselines, otherwise it is difficult to know how it compares to other related works. The other issues are fairly minor. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Authors point out the problem that a large buffer can use up a lot of memory, making it difficult to deploy DRL to portable devices. \n- Authors propose a new technique to reduce forgetting in the training process to allow better performance with a smaller buffer. The core idea is to have a consolidation loss, which essentially moves current Q estimates towards target Q network estimates. Two algorithmic variants are proposed, MeDQN(U) uses a very small buffer and sample uniformly from a bounded state space, MeDQN(R) uses a larger buffer (but still much smaller than DQN baseline) and sample from the buffer. \n- Extensive experiments are ablations are provided for a number of tasks. Authors show that the proposed method can achieve much stronger performance compared to baseline when buffer size is small. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add comparison to related techniques that might help tackle forgetting, test them with smaller buffer sizes to see how they compare to your proposed method. Just comparing to the baseline does not tell us how good the proposed method is against other methods in the literature. \n- Authors claim the proposed method also works for continuous setting. This is not convincing, we all know the continuous setting can be more complex in many ways, do you have theoretical or empirical evidence to show it also works in continuous setting? If not should change the argument.  \n- first mentioned Figure 2 (which has MeDQN(U)) before section 3, and MeDQN(U) is not defined or explained until much later in the paper (4.3). Would be great if you add some additional explanation early on to help readers understand. \n- End of page 5, why does functional regularization allows for optimizing θ in bigger parameter space? Is this knowledge from another paper? ", "broader_impact_concerns": "N/A ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YPcdjseBAf", "reviewer": "Reviewer_qCbE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is overall well written\n- The method is interesting and simple to implement\n- The experimental evaluation matches the claims of the paper\n\nWeaknesses\n- Some parts could have been written more concisely to focus on the core aspect of the contribution while potentially leaving some parts for the appendix (e.g. It seems possible to summarize more concisely the main ideas from sections 4.3 and 4.4).\n- The experiments are sufficient in my opinion but one could always argue about more environments", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a technique to improve learning in the context of reinforcement learning, in particular to allow using a smaller replay buffer without losing (much) in performance. The algorithm is based on the deep Q-network algorithm. The algorithm introduces a \"consolidation loss\" that reduces forgetting from the target Q-network to the current Q-network. The algorithm is compared to the baseline without the suggested improvement and show that it achieves comparable or better performance with smaller experience replay buffers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (suggestion to strengthen work) The positioning of the work with respect to the related work could be improved: the difference with methods such as EWC (Kirkpatrick et al., 2017) and SI (Zenke et al., 2017) are mentioned. In the related work, it is however not fully clear whether there are other algorithms that regularize the function directly in the context of (continual) supervised learning. If not, why is the contribution focused on the context of RL only? Is yes, could it be viewed as an extension to RL of existing techniques?\n- (suggestion to strengthen work) Rewrite more concisely Sections 4.3 and 4.4. It seems that the overall idea could be summarized in just a few lines for each sections. It is a bit unclear why some of the elements are explained in great lengths, while also introducing some notations in ways that might bring more confusion than values to the paper. For instance, the definition of S_{LOW} and s_{HIGH} is not very clear and probably not needed in my opinion. Some technical details could be put in the appendix and the most important parts could be summarized in the paper.\n- (suggestion to strengthen work) Concerning the environments, it is tested on 4 low dim environments and 4 high-dim ones (Atari). Overall 8 environments is already good according to me (more is even better, e.g. 6-8Atari games would already give a broader view). But given the compute requirements, one alternative (or something in addition) to having more environments is to clarify why/how these environments were selected. Were they at random in the set of Atari games or chosen for some specific reasons? This can be important to judge whether we can expect that the same performance would likely be obtained on other environments or whether we can expect that the improved performance is limited to specific types of environments.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3WiSthUmPw", "reviewer": "Reviewer_uL1k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* Simplicity of the proposed method: the proposed idea is easy to implement (but at the same time, novelty is a bit limited).\n\n* Clear writing\n\nWeaknesses\n\n* Novelty: The core part of this work (i.e., knowledge consolidation) is not a very new idea and the uniform sampling method doesn't sound like a general solution. I'm not sure if this proposed idea can be working on more complex domains with realistic images (https://aihabitat.org/) or complex dynamics (humanoid). \n\n* Current results on simple domains are not very convincing (see requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Utilizing a replay buffer is one of the important practical techniques in deep off-policy RL. However, keeping the replay buffer increases the memory requirement. This paper proposes a memory-efficient method for off-policy RL algorithms. The main idea is using a small replay buffer and adding consistency regularization (called knowledge consolidation). By adding knowledge consolidation, the authors prevent the catastrophic forgetting issue. For knowledge consolidation, the authors propose two sampling schemes: uniform sampling which generates random samples from uniform distribution, and real state sampling utilizes previous states in the buffer. The authors showed that the proposed method achieves good performances with small replay buffer on simple OpenAI Gym envs and MiniAtar envs. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Evaluation on more complicated domains: in order to show the (significant) benefit of this simple idea, more evaluations on complicated domains are required b/c MiniAtari and OpenAI Gym are too simple. I think testing the ideas on domains where a big replay buffer is required (e.g., more high-resolution images-- https://aihabitat.org/-- or requiring many samples due to complexity--humanoid) can make this work more convincing.\n\n* Extension to continuous action space. ", "broader_impact_concerns": "There are no particular concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Neural networks are state-of-the-art Q-function approximators in off-policy RL. When training these networks, experience replay is used to reduce forgetting and stabilize learning. This work proposes a new approach to experience replay, where the optimized Q network is regularized by the target Q network over samples from the experience replay buffer. The proposed approach is evaluated on several low-dimensional control problems and Atari games. The authors demonstrate that their approach requires a much smaller experience replay buffer to perform comparably to the original approach.

This work addresses an important problem and the reviewers recognize it. The approach is also simple and thus can be easily used by others. The reviewers disagree whether the paper demonstrated sufficiently good performance on more complex tasks. Specifically, two reviewers ask for:

* More Atari games, some of which are arguably hard for DQN-based algorithms.

* Continuous control tasks.

To reach a middle ground, I suggest the following:

* This paper will be **accepted with minor revision**.

* The authors will reduce the scope of their claims, including potentially changing the title, so that they better reflect the demonstrated value of the paper. I will review this and ask the reviewers to confirm. Please highlight the changes.
Decision: Accept (poster)