Beyond Target Networks: Improving Deep $Q$-learning with Functional Regularization
Authors: Alexandre Piché
Joseph Marino
Gian Maria Marconi
Christopher Pal
Mohammad Emtiyaz Khan
Venue: ICLR 2022 Submitted (2022)
Keywords: Q learning
regularization
deep Q learning

Abstract: A majority of recent successes in deep Reinforcement Learning are based on minimization of square Bellman error. The training is often unstable due to a fast-changing target $Q$-values, and target networks are employed to stabilize by using an additional set of lagging parameters. Despite their advantages, target networks could inhibit the propagation of newly-encountered rewards which may ultimately slow down the training. In this work, we address this issue by augmenting the squared Bellman error with a functional regularizer. Unlike target networks', the regularization here is explicit which not only enables us to use up-to-date parameters but also control the regularization. This leads to a fast yet stable training method. Across a range of Atari environments, we demonstrate empirical improvements over target-network based methods in terms of both sample efficiency and performance. In summary, our approach provides a fast and stable alternative to replace the standard squared Bellman error.

Under review as a conference paper at ICLR 2022
BEYOND TARGET NETWORKS: IMPROVING DEEP Q- LEARNING WITH FUNCTIONAL REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Much of the recent successes in Deep Reinforcement Learning have been based on minimizing the squared Bellman error. However, training often unstable due to fast-changing target Q-values, and target networks are employed to stabilize training by using an additional set of lagging parameters. Despite their advan- tages, target networks can inhibit the propagation of newly-encountered rewards which may ultimately slow down training. In this work, we address this issue by augmenting the squared Bellman error with funetional regularizer. Unlike target networks, the regularization we propose here is explicit and enables us to use up-to-date parameters well control the regularization. This leads to a faster yet more stable training method. Across range of Atari environments, we demonstrate empirical improvements
target-network based methods in terms of both sample efficiency and performance. In summary, our approach provides a fast and stable alternative replace the standard squared Bellman error.
INTRODUCTION
In practice, Deep Q-learning (DQL) methods demonstrate instability due to being trained with off- policy data, funetion-approximation, and bootstrapping (Sutton & Barto (2018) Van Hasselt et al.) 2018). The Bellman error, which is used to train DQL methods, is computed with target Q-values estimated by a Deep Neural Network (DNN) with constantly changing parameters which results in instability. Target networks were proposed as a solution to stabilize training by performing an implicit regularization. They do so by using additional set of parameters which can be updated in two different ways. The first way, which a popular method for visual tasks such as Atari, is to periodically update the parameters to estimate the target Q-values (Mnih et al.) 2013). The second way, popular in robotics and control tasks, is to parametrize the target network using a moving average of the parameters (Lillicrap et al. 2015).
Stabilizing the squared Bellman error using target networks parametrized by an additional set of weights has become standard practice in DQL algorithms achieving otate-of-the-art performance
in a variety of difficult tasks (Mnih et al.] 2013): Lillicrap et al.] [0015] Abdolmaleki et al.] 201 2b1
Haarnoja al. 2018a| Fujimoto al. 2018 Hausknechht & Stone 2015 Van Hasselt al. 2016).
Despite the impressive performance of target networks, using additional sets of parameters to esti- mate the target Q-values can be problematic. First, due to lagging parameters, newly encountered rewards may not be quickly propagated through the state space, ultimately slowing down the train- ing. Second, while this can be partially addressed by using a moving-average (Polyak averaging) of parameters, such averaging procedure can be seen as a type of weight- reuggh-regulrrization which too can slow down the updates. The weight-regulaarzztton do not necessarily improves stability (Benjamin et al., 2018). The goal of this paper is to address these issues by using up-to-date Q-values in the square Bellman error and using explicit Q-values regularization by functional regularization.
known be insufficient for DNNs and
We propose an alternative to the squared Bellman error based on functional regularization which, as we show, enables us to go beyond the traditional target networks. Our approach allows us to use the up-to-date parameters to compute the Bellman error while stabilizing the training by functionally regularizing against periodically updated prior network (see Fig. I Using up-to-date parameters addresses the issue of slow propagation of newly encountered rewards. Since the regularization is explicit, it enables us tune the regularization level for given task, which is not possible with

---

Under review as a conference paper at ICLR 2022
Squareed
FR Lass
Siguaeed Bellman Rinor
12
2
R
SOCDO
hand iodt update
DECCI COO00
epepa hard fooh update ostet
GOODO
sharred
SIDCO
(a) The Bellman error with Target Networks.
(b) The Functionally Regularized Bellman error using our Prior Network approach.
Figure I: (a) Target networks use an additional set of parameters, 0 (pink), to estimate target values which are then used in the Bellman error. The target network stabilizes the Q-value estimates, but is not up-to-date. (b) In our approach, we use the up-to-date 0 to estimate the Q-value for the Bellman error (see the rightmost block), and add a x weighted Functional Regularization (FR) loss to stabilize the current Q-value estimate (parameterized by 0) by comparing to the old Q-values (parameterized by 0).
standard approaches that rely on implicit regularization based on target networks. We validate our new objective on the Four Rooms environment (Sutton et al. 1999) with DNNs function approx- imation and show that we can approximate the true value function and learn quickly. Finally, we demonstrate that DQL methods with the FR Bellman error outperform popular algorithms based on the Bellman error on a subset of the Atari suite (Bellemare et al. 2013) which is the most common benchmark for DQL methods.
BACKGROUND
Preliminaries We consider the general case of Markov Decision Process (MDP) defined by A, R, u.). where S and A respectively denote the finite state and action spaces, P: S x SxAAx S- R represent the environment transition dynamics, where P(-ls, a) is the distribution of the next state taking action a in state 8. R: SxA - IR denotes the reward function, E [0, 1) is the discount MDP to represent the sequential interaction
factor, and μ is the initial state distribution. We between an agent and the environment, where at each round the agent observes its current state S,, selects an action a which results in a reward R(s,, ax) and the next state S,-1 P(-ss, at) The reinforcement learning (RL) objective involves defining an agent that maximizes the expected discounted sum of rewards E... Ee r R(s,, (with Sy ~ μ1), by means of policy π raass) that given a state selects in expectation the best action. Bellman optimality equations are defined as:
Q' (s. a) R(s, + Ep
R(s, ai) =s au
(2) where the optimal policy is defined as n² (als) 5(a arg max (s a)). 8(-) is the Dirac delta function, and we use π to evaluate the expectations Eqs. I and 2 (see (Watkins) 1989) Note that the value function can be obtained (> max (S,, a).
-R(ss a) + y a' P.x' (s',
Q-value Estimation We can estimate the Q-value for each state-action pair by formulating a regression problem, e.g. treating the right hand side of Eq. (T) the target. However, generally do not have access to the optimal policy or to samples from the optimal return. For- tunately, possible learn from arbitrary policy by using the Q-learning algorithm (Watkins) 1989). y. Specifically, we can minimize the one-step Mean Squared Bellman Error for given transi- tion a,,, St41) (Sutoon 1988):
1/
[Q(s++1, ac+1)] - Q(se, ai)
(()) = = R(ss, ar) +
(3)
target Q-value
2

---

Under review as a conference paper at ICLR 2022
where <aass) 5(a arg max, ((ssa)), and the loss is then averaged over multiple transitions.
This approach of using future value estimates as regression targets is referred to as bootstrap- ping. Sometimes, the target is expressed using the Bellman operator TQ(s,, a,) R((s,, ar) + 11 (Q($++1, A(41).
Function Approximation The original Q-learing algorithm by Watkins (1889). estimates each state-action pair separately, is a prohibitive task in real-world applications. Instead, it is possible approximate Q by means of a parametrized function Qo (e.g.. deep network), which can be learned by minimizing Eq. (3) with an iterative algorithm such as gradient descent. Unfortunately. combining off-policy data, function approximation, and bootstrapping makes learning unstable and potentially diverges (Van Hasselt et al. 2018 Sutton & Barto) 2018; Achiam et al.) 2019). The problem arises when the parameters of the Q-network are updated to better approximate the Q- value of state-action pair at the cost of worsening the approximation of other Q-values, including the ones used as targets.
One approach to stabilize learning involves estimating the regression targets with a targel network Qø. a copy of Qe where 0 is periodically updated only after certain number of iterations (Mnih al. 2013). This causes regression targets no longer directly depend on the most recent estimate of O. With this, the Mean Squared Projected Bellman Error (we will refer to this loss as simply the Squared Bellman Error as we only use functional approximation) loss for given state transition is:
gtarget (0) R(s;, a.) + + 11.0:1111111pp (Qg(5(+1, a++1)] Qe at)
(4)
A common strategy to stabilize learning is to update the target parameters only after a fixed number of training iterations i.e., periodic update (Mnih et al. 2013). In between these updates, the target using the lagging parameters, and the newly encountered rewards are not immediately reflected in the estimated target Q-values. The delay in the reward propagation can, however, slow down training. For instance, consider Markov chain environment of N states with deterministic transitions, in which all states have reward of O except the terminal, right-most state which has a reward 1. Consider using tabular Q-values and the corresponding target Qvvalues, which are updated every H training steps. Even with access to all the state-action pairs, this scheme will require NH H training steps to propagate the right-most reward through the chain, i.e., to the state N steps away. We can observe that the learning speed would be direct function of H.
An alternative hard-updatinn is the use of a moving average of the weights to parametrize the target Q-network as done with Polyak's 's averaging ("the soft update") (Lillicrap et al. 2015). In this case, the target network's 's weights are updated as 0 < (1 (1-13 T) + 70, with 5 E (0, 1). However, is not used to estimate the target Q-values, thus, regression targets no longer up-to- and learning is slowed.
Weight Regularization in DQN Soft updates can be seen as simple weight w regularization where the update 0 <- 0 is delayed by adding an L2 regularizer T 0 Essentially, we do not want the value of 0 abruptly change. This approach
improving stability common in non- stationary time-series analysis (Brown) 1959; Holt et al. 1960; Gardnnr Jr 1985). online learning (Cesa-Bianchi & Lugosi 2006). and continual learning (Kirkpatrick al. 2017 Nguyen et al.) 2017). Recent attempts to go beyond target networks employ preconditioning methods which can also be seen as
methods (Knight Lemnrr 2018 Achiam al. 2019). For ex- ample, natural-gradient descent used in Knight & Lemner employs the Kullback-Leible (KL) for Gaussian distributions N (o a), 1) over the Q-values q, giving rise to a quadratic weight- regularizer. Another approach of Achiam et al. (209)) uses a different preconditioner which can also The computation in these approaches is challenging
be seen as a quadratic Bright-regularizattonn and so far have only been applied to small networks.
A more serious problem with weight regularization is its ineffectiveness in DNNs. For neural net- works, the network outputs depend on the weights in complex way and the exact value of the weights may not matter much. What ultimately matters is the network outputs (Benjamin et al. 2018), and it is better to directly regularize those.
Deep Q-learning algorithms Deep Q -entwwrrk (DQN) Mnih et al. 2013) has been the first DQL algorithm to solve Atari games from images. To learn the Q-values, DQN minimizes the squared
3

---

Under review as a conference paper at ICLR 2022
Bellman error where the target Q-values are estimated by target network with periodically up- dated set of parameters. Polyak DQN is a DQL algorithm that also minimizes the squared Bellman error but uses target network using& moving average of the parameters. Double DQN decouples the action selection from the action evaluation by using different network to do both and to provide a target during training and counter the maximum bias. It has been shown effective way stabilize DQN and prevent
(Van Hasselt et al. 2016).
3 DEEP Q-LEARNING WITH FUNCTIONAL REGULARIZATION
In practice, deep Q-learning methods are known to be unstable. to not approximate the true value function, and to sometimes even (soft-) ddiverge (Van Hasselt 2010 Van Hasselt et al., 2016 2018). A multitude of solutions have been proposed to stabilize training, but in this work will mainly focus on the stabilizing role played by target networks.
Target networks are parametrized by additional set of lagging parameters. Intuitively, since the lagging parameters are used to estimate the target Q-value, they can be interpreted as a form of implicit prior, preventing the Q-value estimates from changing too quickly. In this section, we use this insight to introduce the functional regularized squared Bellman error, a regularized loss to stabilize training while using up-to-date target values. In this process, we help formalize the notion of target networks as forms of functional priors.
3.1 FUNCTIONALLY REGULARIZED SQUARED BELLMAN ERROR
Target networks have coupled roles: providing target Q-values used for learning, and stabilizing training. We can decouple these two roles by using up-to-date parameters estimate the target Q-values, and using functional prior to regularize the Q-values. Following Benjamin et al. (2018), we proposed using an L2 functional regularizer (FR) which penalizes the divergence between the current estimate of Q-value function (s a) and a lagging version of it Qz(s, a). This gives us the following loss function:
(R) - A(ss,, ar) stop-grad (04s Q0 ax) )²
(5)
- a ai)
ai)
2
where the up-to-date parameter 0 is used in the Squared Bellman Error along with FR loss to stabilize 0 by regularizing the current state-action pair only. The expectation above is taken over the transition and policy, and > 0 is the regularization parameter. By stop-grad, we indicate function that prevents gradients from propagating into the target Q-value estimate.
Critically, unlike Eq. 4). the target Q-value estimates supplied by the up-to-date Q-network, with the functional prior now separately serving to stabilize the estimate. This decoupling allows us quickly propagate newly encountered rewards and maintain stability. As with target network, can update the functional prior periodically to control the stability of the Q-value estimates. Overall, (FR arguably similar in complexity to (larget, requiring only an additional function evaluation for Q0(5 1 ) and an additional hyper-parameter K, which was not difficult to tune in all of our experiments (see Section 4). As will be discussed in the subsequent Section 3.2 target networks
implicitly make fixed trade-off between speed and stability, whereas k. in the FR squared Bellman error provides additional flexibility and can be task dependent.
The functional regularization between Qo and Qa can be seen as an approximation to the Kullback- Leibler (KI.) divergence between two Gaussian processes where we assume identity covariance for these two processes. Since the covariance matrices might not be the identity in practice, further improvement might be possible by estimating the covariances at the cost of additional computation, e.g. (Pan et al. 2020). This interpretation draws link with other popular regularization methods in policy optimization inspired by trust-region algorithms (Schulman et al. 2015a Abdolmaleki et al.) 2018b a) where the quantity of interest, the policy, is KL regularized towards its past values. In the same spirit, for value-bases methods, the FR Squared Bellman Error regularizes the Q function estimates towards their past values.
4

---

Under review as a conference paper at ICLR 2022
3.2 INTERPRETING TARGET NETWORKS THROUGH FUNCTIONAL REGULARIZATION
Given the similar regularization role accomplished by target networks and functional priors. it is reasonable assume that target networks can be interpreted through the lens of FR. To validate this assumption, we compared the gradients, Vol(O), arising from each case (Eq. (4) & Eq. 5 which allowed us to see how these learning procedures differed. For complete derivation, see
Appendix A To simplify notation, we define the Q-value estimate at time as Q Qa (8:, ar).
the corresponding expected change the Q-vauue estimate as _Q = Ep, (0(+1)))) |st ai) 2o and the reward as R(t) = R(s;, at). Then, for a given state-action pair, we can compare the two gradients:
Vel FR (0) = -R + x(26 + AQ% -000 Voo! + ==
- Vol
(6)
Replacing (t) with sQ and with y we recover the gradient of (0):
AQ
0
V thrget (R + 700% + o + =2 Q6")
(7)
By comparing the gradients, we see that, despite the differences in their loss formulations, target networks and FR result in nearly identical gradients, with two key exceptions. The first distinc- tion, underlined in orange, is the estimated change in the Q-value. With target networks, this term is supplied by the lagging network parameters, 0, and is therefore not up-to-date, whereas with FR, it is supplied by the up-to-date parameters. Thus, illustrating why FR can propagate updated value estimates faster. The second distinction, underlined in blue, is the weighting on the difference between the lagging and up-to-date Q-value estimates. With target networks, this is the discount factor, 7 whereas with FR, this is a separate hyper-parameter.
is usually part of the problem
definition and thus cannot be tuned for a specific task. Using the gradient in Eq. ( and defining
000 = Q² + AQ
(t)
we can derive a loss function, اiاne that yields equivalent gradients as target
but written in the FR form:
Tlarge (0) = R(st, + stop_grad (7t (00,0 Q0(
?
Q0(ss -aaaa--Qpp Qg at) 2
From this perspective, target networks effectively perform a special, restricted form of Gaussian FR, with p(Qe) .N(Qg. " i..., precision weight that is not under direct control. In contrast, FRR allows us to separately adjust this weight to trade-off between stability and learning speed. Overall, this derivation illustrates the similarities between the squared Bellman error with target networks and the FR squared Bellman error. It also highlights the advantages of using the FR squared Bellman error over square Bellman error combined with target networks: up-to-date target Q-value estimates and control over the regularization through K.
3.3 COMPARISON WITH POLYAK UPDATING
Polyak's updating common technique used to trade-off between speed and stability for control problems. Interestingly, if we assume the target network is estimate in weight space of the latest Q-network, we can show that Polyak's 's updates implicitly performs parameter regularization on the weight estimate. To better understand the implicit regularization, let consider situation in which after each gradient step (indexed by 1), we identified the target network as the solution to the problem:
1- 0 0,1
0. = min マ 11000 1-0-0,11 0
0,112
2T
whereby 0, is the latest set of weights of the Q-network and 0 is the previous instance of the target network, and 7 is the exponential averaging step size. Minimizing Ø is obtained by computing the gradient of the problem and equating it O, leading to 0,+1 = (1-7)0,+t -,0,0 which is exactly Polyak's 's updating. This derivation illustrates that Polyak's update is indeed form of weight regularization
5

---

Under review as a conference paper at ICLR 2022
(a) Success Rate
(b) EallT Qe Qallé
(c) E EgllVa(s) (
(d) Success Rate
16²
102
10"
0.
10"
10"
12
10"
002
0.
0.6
1.0
1.0
Gradienl Steps
165
Gradient Sleps
1ed
Gradient Steps
DQN Reg Weight 0.1 (Ours) Polyak DON Tau 0.01
DQN Target Update Perfod
Polyak DQN- Tau 0.05
DQN Target Update Period
Doep Mellow Mellow Temp
Figure 2: Four Rooms Comparison. We compare the performance of different Deep Q-learning algorithms. The reward position is kept fixed throughout training (upper-left room), but the agent's position is randomized the beginning of every episode to bypass exploration difficulties. In Figure (a), we can see that FR DQN (blue dashed) quickly and stably completes the task. The other methods struggle to get good combination of speed and stability, e.g., the two green methods either fast or stable. All methods minimize the squared Bellman error equally well, as shown in Figure (b), however our method approximates the true value function an order of magnitude more accurately than the other methods as shown in Figure (c). Figure (d) shows the stability of FR DQN where the performance not too sensitive changes in the regularization parameter (N).
with respect to the most recent target network weights. Weight regularization does not, however, guarantee that the output of the regularized network matches the previous target network. In fact, while this technique has found success in control problems (Lillicrap et al. [015] HHarnoja al., 2018 Sa Fujimoto et al. 20018) 8). periodically updating the parameters is usually preferred for complex DNN architectures (Mnih al. 20133) Hausknecht Stone 2015 Hessel et al. 2018: Kapturowski
et al.
2018 Parisotto et al. 2020).
4 RESULTS
We compared the performance of our objective (FR-Squared Bellman Error) and other algorithms minimizing the squared Bellman error and using target networks in the Four Rooms environment and the subset of Atari games (Bellemare et al., 2013) used by Mnih et al. (2013). The experimental code based on DQN Zoo (Quan & Ostrovski 2020), implemented Jax (Bradbury et al.) 2018)
and Haiku (Hennigan et al. 2020). Additional experimental details can be found in the Appendix.
4.1
FOUR ROOMS
4.1.1 EXPERIMENTAL SET-UP
In order to better understand the performance of the FR squared Bellman error, compared objective to a suite of algorithms and hyper-parameters minimizing the squared Bellman error and using target network in the Four Rooms environment (Sutton et al. 1999) with neural approxi- mation. Four Rooms is a grid world environment comprised of four rooms divided by walls with single gap in each wall allowing must reach goal state in the upper-left room near the right door, and the agent's ' initial position drawn randomly from all other cells excluding the walls position. The agent's position is given through a one-hot encoding. The agent is capable of taking actions in four directions: up, down, left and right. See Fig. 12 in the Appendix for graphical representation of the environment and the optimal discounted return.
agent to move between rooms. In this environment, the agent's
The advantage of using this environment over more complex ones such as Atari games, is that the underlying transition matrix is available, and the true optimal value function can be computed using tabular methods (which will be referred to as V*). We can then compare the agent's 's parametrized value estimate (s maxa Qa a) with the optimal one (*) to assess the accuracy of the agent's is Q- Q-values. The discount factor is 0.99, the reward is always except at the goal state where every action leads to reward of 1. Specifically all actions from the goal state receive reward of

---

Under review as a conference paper at ICLR 2022
pong
spaco_invaddars Invadier
20
10000
0000
1500
8000
6000
1250
Retu
6000
1000
4000
4000
730
2000
2000
-20
250
Frame
Seaquest
breakout
anduro
DON (Ours)
DON
3000
600
Polyax DON
Retum
Mellow
2000
4.00
0.5
1.0 Twh
0.5 Frame
1.0
0.5 Frama
1.0
Frame
Twh
1ed
Figure 3: DQN Atari Comparison. Performance curves for the subset Atari games from Mnih et al. (2033). The returns are averaged over 10 trials using the e-greedy policy with 0.05.
1. The episode terminates either once the agent reach the goal or after 500 steps. The environment implemented using Easy MDP (Ahmed] 2020). The Q-value is parametrized by Multi-Layers Perceptron (MLP) with 2 layers of 128 hidden units each. The shared syper-parameters (learning rate and batch size) were tuned on 5 seeds for Deep Q-Network (DQN) (Mnih al.) (013) to maximize success rate, and then use for all the other methods. The final performances are reported on 20 different seeds after 100k gradient steps (and as many transitions). All hyper parameters are available in Table
4.1.2 RESULTS
We compared the performance of the FR square Bellman error combined with the DQN algorithm (FR DQN) to DQN, Polyak DQN, and Deep Mellow (Kim et al. 2019). As observed in Fig. 2 (a), the target network update period greatly affects performance of algorithms minimizing the squared Bellman error. On the one hand, frequently updating the target network leads to faster initial per- formance since the reward is propagated through the space faster. On the other hand, updating the network too frequently can lead to instability as observed in Fig. 2 For example, T of 0.05 ini- tially outperforms of 0.01, but becomes unstable, and never reaches optimal performance. The FR squared Bellman error combined with the DQN algorithm (FR DQN) bypasses this issue, by using the previous Q-values estimates exclusively for regularization which allows to use the most up-to-date estimate for learning.
Additionally, Q-learning methods should converge the optimal value function V*, but in complex settings like Atari games, it is impossible to compute V* and monitor the distance to the optimum. In the Four Rooms setting, however, measuring the accuracy of their approximation is tractable. Interestingly. even if the Bellman error loss is roughly the same for FR DQN and Polyak DQN (Fig. 2 (b)), FR DQN approximates the true value function with much greater accuracy than Polyak DQN (Fig. 2 (c)). We also note that Deep Mellow does not recover the true value function. This failure is not surprising as Deep Mellow is biased by using the mellowmax operator instead of the max. This is of particular interest since the squared Bellman error is the metric that is usually monitored in practice and might not be an accurate measure of the accuracy of the Q-value.
4.2
ATARI
4.2.1
EXPERIMENTAL SET-UP
The Arcade Learning Environment (Bellemare et al. (003)) provides a set of benchmark Atari games for evaluating Reinforcement Learning algorithms. These games represent good benchmark as
7

---

Under review as a conference paper at ICLR 2022
pong
qpart
spece__ invaders
seaquuest
20
12500
25000
10000
10
10000
Retu
7500
7500
1500
5000
5000
1000
2500
2500
-20
Frame
anduro
broakout
boam, /ndor
1260
12500
1000
10000
FR Double DQN (Ouns)
Relum
7500
Double DON
5000
2500
0.5
1.0
0.5
1,0
0.5
1.0
Frame
Ted
Frame
Tex
Frama
1eh
Figure 4: Double DQN Atari Comparison. Performance curves for the subset of Atari games from Mnih et al. (Mnih et al. 2013). The baseline (dotted blue line) and 4 different FR weights (solid lines) average returns over 10 trials using e-greedy with = 0.05.
they possess rich visual observations and diverse reward functions. Complex DQL methods cur- rently hold thee--of-the-rrr results in this benchmark (Hessel et al.] 2018). But order to understand the effect of replacing the squared Bellman error with the FR squared Bellman error, we first com- pared our approach with the standard DQN (Mnih et al.] 2013) and Polyak DQN. Then we showed that we can easily combine the FR squared Bellman error objective with orthogonal improvements such as Double DQN
(Van Hasselt et al. 2016).
We stress that the algorithms based on the FR squared Bellman error, e.g., FR DQN and FR Double DQN, are simple modifications from the DQN Zoo library (Quan & Ostrovski 2020). We tuned the aper-parameter for each method (except DQN as they have already been tuned by the authors of the DQN Zoo package) by performing experiments on 2 seeds (only I set of global for each method have been selected,
used for every tasks). Once the best set of hyper-parameter we used 10 additional seeds (and exclude the 2 initial seeds) to compare the algorithms. All hyper- parameters we used are displayed in Table 2 in the appendix. We used the same suite of representa- tive environments from Mnnh al. (2013). Each seed took 48 hours run our resources for total of 14400 hours of GPU usage.
4.2.2 RESULTS
Comparison with DQN, Polyak DQN, and Deep Mellow The comparison between FR DQN, DQN and Polyak DQN Atari performance results are presented in Fig. [3 Across all environ- ments, FR DQN matches or exceeds the performance of DQN and Polyak DQN in terms of sample efficiency and final performance at 100M steps. FR DQN's ability use up-to-date target Q-values also plays significant role in improving performance over Target Networks. Polyak DQN and Deep Mellow are quite unstable and do not constantly match the performance of DQN. We hypothesise that the temperature of Deep Mellow has to be finely tune to each games in order to outperform DQN as reported in Kim et al. (2019).
Comparison with Double DQN Since the introduction of Target Networks, there has been nu- merous techniques to stabilize the training and improve the performance of DQL methods (Hessel et al. 2018). While it is outside of the scope of this paper to combine FR DQN with each and every DQN improvements, we show that it is fairly easy to combine the FR squared Bellman error with double DQN (Van Hasselt et al. 2016). The results presented in Fig, | clearly demonstrates that the FR squared Bellman error can be combined with DDQN to match or achieves even higher performances this subset of Atari games with the exception of Seaquest.
8

---

Under review as a conference paper at ICLR 2022
5 DISCUSSION
Multiple previous works have investigated how to improve value estimation through various con- straints and regularization. Farahmann et al. (2009) is perhaps the closest work to our own, func- tionally regularizing the L2 norm of the Q-value estimate, i.e., penalizing x| Qoll Farahmand al. (2009)'s 's approach can be interpreted as using a fixed Gaussian prior, whereas we used periodically updated set of parameters to provide a moving prior. However, penalising the magnitude of the Q- values would not prevent the algorithm to converge to V if x does not tend towards 0. [Shao et al. (2020) proposed adding an additional backward Squared Bellman Error loss to the next Q-value to stabilize training and remove the need for target network. Other works have sought to regularize the Q-value estimator by constraining parameter updates, c.g., through regularization (Farebrother al. 2018), conjugate gradient methods (Schulman et al.) 2015b), pre-conditioning the gradient updates (Knight & Lerner 2018] Accham et al. 2019), or using Kalman filtering (Shashua & Man- nor 2020). As argued in Sections 2 and functional regularization, as opposed parameter regularizations is more appropriate in combination with DNNs, because parameter regularization does not necessarily imply that the Q-value estimates remain stable during training.
Other previous works have sought to address orthogonal issues with DQL algorithms, and, in prin- ciple, these could be combined with the FR squared Bellman error. For instance, Kim et al. (2019) removed the need for target network on Atari games by using the mellow max operator as an alterna- tive to the max operator used in bootstrapping. This new operator is used with the squared Bellman error and could easily be used in combination with FR squared Bellman error. As mentionned above, Double Deep Q-learning (Van Hasselt 2010) Van Hasselt et al. 2016) addresses positive bias the value estimate (Thrun & Schwartz, 1993) by using two Q-networks, with each target network pro- viding target values for the other. Fujimoto et al. (Fujimoto et al., 2018) extend and apply this technique to continuous control, instead using the minimum of the two target networks in the target value.
CONCLUSION
In this paper, we proposed the FR Squared Bellman Error as an alternative to the Squared Bellman Error. The Bellman error requires target networks stabilize the training of DQL methods due to the non-stationary nature of the regression problem. Like the squared Bellman error combined with target networks, the FR squared Bellman error uses a lagging set of parameters to stabilize estimation. An important distinction however is that the lagging set is decoupled from target value estimation, instead taking the form of a functional prior. In this way, we can 1) use up-to-date target value estimates, thereby propagating reward information more quickly, and 2) separately control the degree of stabilization via the FR weight, k. Indeed, by comparing the gradients resulting from the squared Bellman error with target networks the FR squared Bellman error, networks implicitly perform a form of FR, but with lagging target values and a non-independent weight (x 7). This helps illuminate why target networks stabilize training in practice, while also highlighting their drawbacks.
noted that target
We also demonstrated the stability of FR DQN in the Four Rooms environment by showing that can quickly recover the optimal value function, and show that we achieve higher performance than DQN and Polyak DQN. Then we showed that FR DQN consistently match or outperform DQN and Polyak DQN on a subset of the Atari suite. Finally we showed that FR DQN can be combined with DQN to further improve performance. Thus. we conclude that FR Bellman error provides a conceptually simple, easy to implement, and effective objective for improving DQL methods
While the FR Bellman error requires an additional weighting parameter, Ki we noted that the perfor- mance does not change dramatically for small change in K. We see i as providing needed flexibility in setting the degree of regularization, unlike Target Networks, which cannot separately control this asppct. In our experiments, we used fixed к throughout training, however, future work could investigate methods for automatically adjusting x during training (Haamoja et al.) 2018b) or con- ditioning it on individual state-action pairs. Nevertheless, the FR Squared Bellman Error objective that we have presented here performs well across a wide range of environments, providing a drop-in replacement for the Squared Bellman Error.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint 1rXiv:/8/2.02256. 2018a.
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 180,00000 2018b.
Joshua Achiam, Ethan Knight, and Picter Abbeel. Towards characterizing divergence in deep q- learning, arXiv preprint 1rX.v:1903.08894, 2019.
Zafarali Ahmed. Easy MDPs implemented in gym like interface with access to transition dynam- ics., 2020. URL
com zafarali/emdaiee
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv: 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal
Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Kanderman-Milne and Qiao
Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. ctt wtt.c/////ooo///aaoooooo
Robert Goodell Brown. Statistical forecasting for inventory control. McGraw/Hill, 1959.
Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Amir Massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor, Regu- larized fitted q-iteration for planning in continuous-space markovian decision problems. In 2009 American Control Conference, pp. 725-730. IEFE, 2009.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint 1rXi::1810.00123, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Everette S Gardner Jr. Exponential smoothing: The state of the art. Journal of forecasting, 4(1): 1-28, 1985.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv:1812.05900, 2018b.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps.
arXiv preprint arviv: / 0507 2015.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL h wtt hta.deeeemmmmmmmmaaaaaaaaaaaaa dm-haiku
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
10

---

Under review as a conference paper at ICLR 2022
Charles C. Holt, Franco Modigliani, John F. Muth, and Herbert A. Simon. Planning Production, Inventories, and Work Force. Englewood Cliffs, 1960.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent ex- perience replay in distributed reinforcement learning. International conference on learning represenlations, 2018.
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Brabsska-arriinkk. Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352-3352, 2017.
Ethan Knight and Osher Lerner. Natural gradient deep q q-learning. arXiv preprint arXiv: 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02997 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rX1v:13/2.5602, 2013.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turne.. Variational continual learning. arXiv preprint arXiv: 1010.1068, 2017.
Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E Turner, and Mo- hammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. arXiv preprint ar arXiv:2004.1 14070, 2020.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pp. 7487-7498. PMLR, 2020.
John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020. URL
com detpa//deeind///ee/e hdgp/iniid//_mi//i///m/ 200
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1rxiv:1506.02438, 2015b.
Lin Shao, Yifan You, Mengyuan Yan, Qingyun Sun, and Jeannette Bohg. Grac: Self-guided and reuulregul actor-critic. arXiv preprint arXiv:2009.089) 2020.
Shirli Di-Castro Shashua and Shie Mannor. Trust region value optimization using kalman filtering. arXiv preprint
2019.
Shirli Di-Castro Shashua and Shie Mannor. Kalman meets bellman: Improving policy evaluation through value tracking, arXiv preprint arXiv:2002.07177 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
11

---

Under review as a conference paper at ICLR 2022
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(112):181- 211. 1999.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
Hado Van Hasselt. Double q-learning, Advances in neural information processing systems, 23: 2613-2621, 2010.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Mo- dayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02666 2018.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
A APPENDIX
A.1 Q-LEARNING WITH TARGET NETWORK
Letting R(()) = R(s;, at), Q(0) = Q(sr, at) and AQ = E(o!t1) 0
St. arl =
B1Q 1
|st, ai! 0
V o tunget (0) = (R + 10 -000 -
(RO0 + 10 ( + +40
(1)) -
(R(0 + +1 10 +1 (1 71 Vook
0
-(R19 + +1 + A0
(1) -(1 -(1-7)08 Vo
- -(R( + +~ (1 - 10- + (7 - 0 V
- -(R(9 +~ 20 + + V. +1 - 0 1
(R(0
(0 + AQ +7
0 VoOg 10 VoQ
A.2 Q-LEARNING WITH FUNCTIONAL REGULARIZATION
(O) ((R()) P.
10
(+t"))11+000-000000 1 Q²" VOQ +x(0z -000 VQ
-(R()) + AQ Q V + x 0 VoL
A.3 CONVERGENCE OF FUNCTIONALLY REGULARIZED VALUE ITERATION
Let us consider the value iteration algorithm for the state-action value function in the tabular setting.
Q++ + T
where (Q) = R + 7P Q It is known that this algorithm converges linearly as this update is a contraction. We show in this section that value iteration with functional regularization enjoys similar properties.
Theorem (Convergence of functionally regularized value iteration). Value iteration with learning
rate n and with functional regularization E(Q, Q) i Q11 where Q is updated every T steps
the current Q value converges linearly for 2 0. 17 < 1+7+N and T log(1+2 x
log
12

---

Under review as a conference paper at ICLR 2022
Proof. For this proof, need to show that Q will converge to Q" For this split the problem in two 1) on one hand we show that Q converges to Q% (Q), the best FR-regularized Q value and 2) that Q (Q) converges to Q* depending on Q.
First step Q and Q% (Q): Our loss function is
I - 9 Q11
where Q a prior state-action value function that will be update every 7 steps. The update for FR-regularized loss would be
Q++1 Qt + -.
=(Q, Q))
Therefore, equating Qi Q++1, the fixed point is
(T( ((1+ K)1 PPT (
(R (R++kQ) x())
Let us look at the Euclidean distance
1Q411- (T(2)1
II - 17pp" (1 -1+))
S pllQ 07(1)
This is guaranteed to be a contraction for the spectral radius p = max| x' T (I (7(11 + R) yP")) < 1. As the spectral radius of pa is 1 2
theorem), we can deduce that I4y+# is sufficient to get the spectral radius 1 - (n((1 x)] -P=) bounded by 1.
Thus we have
1[Q++ QE(QN) < 0 1Q: -.(
Second step (Q) and Q": First let us show that (7(Q) = Q + (1 + -x(0 Q*)
We have (Q)
(R xQ) and Q (1 7P-)-11 R. Using -1---11
A A A)B-1 we get
(2) Q = (-)=))))-))))((((000--------)--- -R
R + ((1 + K.) - -
= ( xQ --Q2"")
Putting all together: Let us define the FR Bellman operator Q + TFRlQ) which consists into updating the prior in FR the current Q-value then doing T steps of FR update Q. We have TI TF = Q" as Q = R + P"Q"
1T 1
11u4T ((00001++00000) QF1
p T1Q - -0000000000000000===~-00=1>>1=10 Q"1l [|= Q:)1l ulIQ: Q
+ (1+ +1))11
With Spectral radius + K - -1)) S 1ータカル < 1 as > 0. Thus, for
T log(1 + 2 2
log 1+
log
log
have that (1*1)+))+)) therefore operator is contraction, thus the methods converges linearly.
13

---

Under review as a conference paper at ICLR 2022
Average Regret Function of the Environment Size
agent
FR DQN (Ours)
DQN
0.5
Average Regret
Deep Mellow
Polyak DQN
0.4
0.3
0.0
9
19
29
39
Size
Figure 5: We benchmark the different algorithms on different Four Room environment sizes. As increase the size of the Four Room environment, the reward becomes sparser and the task more difficult to complete. We observe that FR DQN scale more gracefully to larger environment sizes. We report the average regret over 500k iterations. See Table for the syper-paramete used for each algorithm.
Median Regret as a Funccion of the Environment Size
0.4
full agent
FR DQN (Ours)
DQN
Median Regret
Deep Mellow
Polyak DQN
0.0
19
39
Size
Figure 6: The average regret can be noisy since certain runs might not complete the task, e.g. 2 seeds do not complete the task for DQN on the size 19. Thus, we also report the median regret over 500k iterations. See Table I for the hyper-parameter used for each algorithm.
14

---

Under review as a conference paper at ICLR 2022
DQN
1.0
0.8
smooth reeurr
0.6
target _ecectt_____ee__ee period
0.4
10
25
0.2
50
100
250
100000
200000
300000
400000
500000
iteration
Figure 7: DQN with different target update period.
FR DQN (Ours)
1.0
0.8
smooth refur
0.6
reg weight
0.4
0.1
0.25
0.2
0.5
0.75
1.0
100000
200000
300000
400000
500000
iteration
Figure 8: FR DQN with different regularization weight.
15

---

Under review as a conference paper at ICLR 2022
Polyak DQN
1.0
0.8
smooth refurn
0.6
tau
0.001
0.4
0.005
0.01
0.2
0.05
0.5
100000
200000
300000
400000
500000
iteration
Figure 9: Polyak DQN with different parmacter 7.
Deep Mellow
1.0
0.8
smooth refur
0.6
0.4
mellow mellow_temp
25.0
0.2
50.0
100.0
250.0
100000
200000
300000
400000
500000
iteration
Figure 10: Deep Mellow with different temperature parameter.
16

---

Under review as a conference paper at ICLR 2022
Table Four Rooms Hyperparameters
Hyperparameter
DQN
Polyak DQN
FR DQN
Deep Mellow
Learning rate
le-3, le-4
le-4
le-4
le-4
Batch size
32, 512
512
512
512
Target Update Period
10, 25, 50, 100, 250
1
Prior Update Period
50, 500, 5000
5,- le-1, Se-2, 1e-2
[0.1. 0.9]
Mellow Temperature
25, 50, 100, 250 [128, 128]
Deep Neural Network
[128, 128] 0.05
[128, 128]
[128, 128) 0.05
0.05
0.05
agent DQN
agent DQN
agent = Polyak DQN
1.0
return
0.6
0.4
0.2
0.0
0.0
0.2
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.6
true
true_erra
true_errar error
Figure 11: Return.
B APPENDIX
B. FOUR ROOMS
B.2 ATARI
Most of the byper-parameters have been kept from the defaut DQN and Double DQN respectively.
Here is a list of the tuning that has been performed and the resulting best hyper-parameters.
Table 2: Atari Hyperparameters
Hyperparameter
Polyak DQN
FR DQN
FR Double DQN
Deep Mellow
T Target Update Period Prior Update Period
5e-5, 2.5e-5, 1e-5, 5e-6
100
4e4
4e4
K
0.25, 0.5. 0.75
0.75
Mellow Temperature
50, 75, 100
C ADDITIONAL EXPERIMENTAL DETAILS
D RETURN AND OPTIMAL VALUE FUNCTION ERROR
17

---

Under review as a conference paper at ICLR 2022
Figure 12: Value function of the Rooms environments estimated via tabular Q-learning
18


Reviews:
[{"review_id": "uj72wcnvxlA", "reviewer": "Reviewer_ugJ6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach is elegant and well-motivated.\n\nWriting is clear.\n\nThe results in the four rooms task show good performance: compared to DQN and Polyak DQN, the proposed approach can recover the optimal value function quickly (given that the optimal value function in this task can be computed using tabular methods in this task). The proposed approach can match and sometimes get better than DQN / Polyak DQN on Atari tasks. \n\n\nComments/concerns: \n\nMore detailed analysis on how FR DQN performs on tasks with very sparse rewards would be interesting (perhaps also with a relatively large action space), given that this is one of the motivations. Will FR DQN actually perform better when there're sparse rewards? \n\nHow does FR DQN compare to \"increasing the update frequency of target nerworks\"?\n\nMore principled / automatic ways of adjusting kappa in Eq. (5) would be interesting. The kappa value is likely a very important term. \n\nIt’s not immediately clear to me why on some tasks, FR DQN results in a much larger standard deviation (seqquest).\n\nIt's not clear to me why FR DQN needs such a \"target update period\" shown in Table 1 in the appendix. \n\nMinor: page 4 typo (\"sectionSection\"), page 6 typo (\"move between room-\"). ", "summary_of_the_paper": "Given that in Q-learning, the Bellman error (which we optimize) involves the target Q values with ever-changing parameters, two major approaches have been proposed to address the instability: (1) have a copy of a slightly old version of Q (i.e., periodically update the parameters to estimate target Q); (2) use a moving average of parameters. Learning is slow in both cases. For example for (1), suppose the Markov chain has N states and we update the target network once every H steps, then it would take NH steps for a sparse reward (right-most reward of the chain) to propagate to the beginning. Similarly, weight regularization in DQN is ineffective too. \n\nTherefore, the authors propose deep Q learning with functional regularization (FR DQN). See Eq. (5). \n\nEssentially, they replace the target network Q’ by the current network Q (and adding stop-grad); and they add the regularization term at the end, i.e., (Q - Q’)^2, so that the up-to-date parameter can be used in the Bellman error.\n", "summary_of_the_review": "The approach is well-motivated. The comparison to Polyak DQN is great. More experiments on complex tasks with relatively large action space, with sparse rewards will be great. A few minor concerns/questions in the main review. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "s1cD-MJba3E", "reviewer": "Reviewer_PwAu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea of using up-to-date parameters in the target network looks interesting. It was believed that the target network needs to be periodically updated for stability, so this paper brings a new perspective on the target network in DQNs.\n\n2. The authors showed empirically that the proposed method can outperform DQN baselines across a few different domains. The performance gap indeed shows the promise of the proposed method.\n\nWeakness:\n1. It's not convincing to me that the regularization term in Equation (5) can be interpreted as a KL divergence between two GPs. As the authors stated, \"mean functions are the Q-values at different state-action pairs and whose covariance function is the identity\". Given that Q_{\\bar{\\theta}} corresponds to a lagged version of Q_{\\theta_t}, the assumption may not hold, since covariance functions are not necessarily the same and equal to the identity matrix.\n\n2. As a part of the motivation, the authors mentioned a few times about the lack of propagating newly encountered rewards, due to the use of target networks. This statement is a bit problematic: Since rewards are randomly sampled from a replay buffer in DQNs, it's difficult to guarantee whether the used rewards can be newly encountered or not.  \n\n3. The experiments did show the advantages over DQN and DDQN baselines. However, it lacks the comparison against a very related work as follows, which also removes the need for target networks in DQNs. A direct comparison could strength this work empirically.\n\n     Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence, 2019.", "summary_of_the_paper": "This paper proposed a new version of deep Q-learning, by avoiding the use of target networks in Bellman errors. Instead, the authors added a regularization term to enforce the current Q-value estimates close to its lagged version. Some analysis was provided to draw the connection with previous work on functional regularization and Polyak updating. Finally, the authors showed in experiments that the proposed method achieved better or similar performance, when compared with DQN and DDQN baselines.", "summary_of_the_review": "This paper brings a new perspective for the use of target networks in DQNs, and the experiments show some promise. However, the interpretation on why it works has some flaws, and a more rigorous analysis could make it more convincing. Furthermore, the authors need to compare with the related work above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B9avANFk3bf", "reviewer": "Reviewer_yew3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow along. I liked reading the paper as its motivation is clearly lined out from the beginning and the claims are also supported by a theoretical justification. Target networks often provide an easy solution to get an RL agent running but properly tuning their update frequency remains difficult in practice. This is also due to the fact that all their implications are largely still unknown. Functional regularization of the prediction of the network presents an easy solution, whose implication becomes clear through the provided theory. The proposed solution should also be (despite not being mentioned by the authors) much more computationally efficient than both approaches that use target networks and related approaches that try to remove the need for target networks. That being said, the authors could also highlight this a bit more.\n\nFrom an experimental point of view, the used environments are sufficient. Atari serves well as a benchmarking environment and comparing the approach to DQN and DDQN also seems sufficient (first I wondered why the other improvements are not being used but those really should not affect each other, so it is better to keep it as simple as possible, DDQN makes sense; also, I wondered how results in DDPG should look like – however, the results should also hold here). Question: did you also use tuned DDQN hyperparameters from DQN Zoo for the experiments? This is not mentioned anywhere.\n\nHowever, what I am missing a bit is an experimental benchmark against related approaches that have been proposed (e.g., Shao et al. (2020) and Kim et al. (2019) which have both been discussed as similar approaches in the paper). I don’t expect them to be much worse or much better either. The proposed functional regularization remains much more intuitive and tractable and also more computationally efficient (which at least for me would the killer-argument to used FR over other that trade performance in sample efficiency over ease of implementation).\n\nSome minor comments:\n-\tp4: In practice, deep Q-learning methods are known to be unstable, [to] not approximate the true value function, and [to] sometimes even (soft-)diverge (Van Hasselt, 2010; Van Hasselt et al., 2016; 2018).\n-\tSec. 4.1.2: for the experiments in Fig. 2(a): do other DQN-variants that use other target network update periods lie in between those two graphs (10 and 100)?\n-\tSec. 4.1.2: \\tau=0.05 initially outperforms \\tau=0.01 […]. Does it reach the plateau in the end (looks like it will but not sure)\n-\tSec. 4.2.1: “Complex DQL methods currently hold state-of-the-art results in this benchmark (Hessel et al., 2018). As previously mentioned, complex DQL methods show state-of-the-art results in this benchmark task.” – this seems redundant. \n-\tCan you explain why in Sequest everything is different?\n-\tFig 2, d) Success rate is quite high for different kappa (in [0.7, 1.0]), but still somewhat “sensible”. In the last part of Sec. 6 Discussion they claim “[…] the performance does not dramatically change for small changes in kappa […].”; Also: It would have been nice to see convergence speed for different values of kappa (as opposed to only success rate)\n-\tp4: Typo “sectionSection 3.2”\n", "summary_of_the_paper": "The paper proposes a functional regularization scheme to constrain updates to the Q-value estimates to get target networks removed. The paper is theoretically justified and puts the scheme into the context of delayed target networks and polyak-averaging (that both instead introduce a lag to the Q-values estimated by the neural network). The paper provides experiments on some Atari benchmarks against a DQN and DDQN that use polyak-averaging and delayed target network updates.", "summary_of_the_review": "The paper is easy to read and provides a significant contribution to a very widely known problem. The solution can easily be integrated in existing applications and frameworks and is hence of broad interest to the community. While I am missing a few experimental benchmarks against approaches that address the same problem but differently, in my opinion the paper provides sufficient experimental evidence to judge the advantages and limitations of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "As-1Os_TMhD", "reviewer": "Reviewer_Gxgv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written and easy to follow. The idea of using the target network for “functional regularization” is simple (in a good sense) and well-motivated. The novelty is somewhat limited, but for empirical papers I think performance is more important than novelty. The authors include code that strengthens their arguments. Given that the paper presents no theorems, I think the sections which mathematically compare the proposed methods to previous ones could be made a little smaller. At any rate, my main concern is the empirical evaluation.\n\n## Four rooms\n\nThe four rooms environment is a simple toy environment. It is reasonable to use such an environment as a sanity check to e.g. show that some theoretical property holds. However, the proposed method lacks theoretical grounding in the form of theorems or guarantees and thus relies on empirical testing to demonstrate any virtues. I think that the four rooms environment is too simple for any empirical arguments, and that performance on it shows very little for an empirical method. \n\n\n## Atari\n\nThe first issue with the atari environment is the number of environments. The authors use 7 tasks considered by previous researchers and use 10 seeds for each of these. I think it would be better to use two seeds in each Atari environment (I think there are ~55).  This would not lead to statistically significant results on individual tasks, but I think it would lead to a better picture of the agents' performance once the authors average human-normalized scores across environments. Just using a few tasks raises the risk that the group of 7 tasks is cherry-picked (in the sense that there are many such groups to choose from) which undermines the results.\n\nThe second issue with the atari environment is the baselines. The DQN algorithm is over 5 years old at this time and has seen many improvements, so improving upon it does not advance the state-of-the-art. The authors state that “it is outside of the scope of this paper to combine FR DQN with each and every DQN improvement”. This is understandable, but what can be even more impactful is improving the performance of competitive methods. I would suggest that the authors consider ONE state-of-the-art baseline that is more competitive and show that their methods improve it. Suitable baselines might be spr (https://github.com/mila-iqia/spr) or rainbow (https://github.com/google/dopamine/tree/master/dopamine/agents/rainbow), both of which have excellent open-sourced code-bases. For such methods, the authors would also not need any parameter tuning.\n", "summary_of_the_paper": "In Q-learning, a target network is often used to define the Bellman loss. The authors propose to estimate the Bellman loss with the “online” network and use the target network for functional regularization. In practice, functional regularization means minimizing the square difference between the target and the online network. Empirically, the authors show that this can improve performance over 7 Atari tasks and the four rooms environment.\n", "summary_of_the_review": "The paper presents a straightforward and well-motivated idea for using online q-values for calculating the bellman loss. The paper is well written and easy to follow. Since there are essentially no theoretical contributions, the empirical evaluation is important but unfortunately has several problems. The authors consider the toy environment of four rooms, which I believe is too simple. The authors also consider 7 Atari environments, I think it would be better to consider all environments from the Atari suite. Most importantly, for the Atari environments, the baselines considered are very simple, and improving upon them does not show that the state-of-the-art is being advanced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple, reasonable, alternative to target networks.  Given the effectiveness of target networks, and the fact that they are still somewhat poorly understood, this is a good topic for consideration.

It is unfortunate that the paper did not have more depth, in terms of analysis and/or analytical experiments that expose the properties of the suggested approach, and the mechanism still seems heuristic (and inspired by the success of target networks, and similar) more than principled.  That said, the proposed mechanism does seem somewhat effective (even if performance differences are not very pronounced), and is clean and simple to implement.

This version of the paper is rejected because we believe the paper could be a lot better than it currently is.  If the proposed regularisation mechanism is really as good as the authors argue, then it should be possible (and hopefully even easy) to demonstrate this clearly in more settings (e.g., in more algorithms).  Alternatively or additionally, the authors could consider digging deeper into the understanding of the method.  For instance, the paper often argues that target networks slow down learning, but (naively?) one could argue the exact same point (in general) for regularisation: this will trade off stability for speed.  It could be that the proposed mechanism is indeed a better way to achieve this trade off, but this is currently argued heuristically and not really proven (either theoretically, or with sufficient empirical evidence)
(For what it is worth, I personally did not find Section 3.2 particularly enlightening, because it is known these TD algorithms are not actually gradient algorithms, and hence considering 'losses' and 'gradients' in this way does not convince me we are getting at an actual deeper understanding of the dynamics of these algorithms.)

I wholeheartedly encourage the authors to take the comments and suggestions to heart and use these to improve the paper (as they have already started to do during this reviewing cycle), because I believe that there could be quite a good paper on this topic.  I hope the authors can convince themselves and their readers more convincingly that this idea is an actual, lasting contribution to the literature.  Ultimately, if they can, this will make the paper more impactful.  So although I appreciate this decision will come as a disappointment, I hope the authors also see this as an opportunity to make a larger research impact.

In particular, I would encourage considering: 1) comparing to our current theoretical understanding of target networks (see, e.g., [1]); 2) considering the effect of multi-step updates (shown in, e.g., [2] and [3] to be quite effective); and 3) considering whether the proposed approach (or a variation thereof) could be understood as a more fundamental idea: could this update be derived from first principles?

[1] Shangtong Zhang, Hengshuai Yao, Shimon Whiteson (2021). Breaking the Deadly Triad with a Target Network.

[2] Hessel et al. (2017). Rainbow: Combining Improvements in Deep Reinforcement Learning. 

[3] van Hasselt et al. (2018). Deep Reinforcement Learning and the Deadly Triad.
Decision: Reject