Bridging the Gap Between Target Networks and Functional Regularization
Authors: Alexandre Piché
Valentin Thomas
Joseph Marino
Rafael Pardinas
Gian Maria Marconi
Christopher Pal
Mohammad Emtiyaz Khan
Venue: Accepted by TMLR ()

Abstract: Bootstrapping is behind much of the successes of deep Reinforcement Learning. However, learning the value function via bootstrapping often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization is still misunderstood. In this work, we show that they act as an implicit regularizer which can be beneficial in some cases, but also have disadvantages such as being inflexible and can result in instabilities, even when vanilla TD(0) converges. To overcome these issues, we propose an explicit Functional Regularization alternative that is flexible and a convex regularizer in function space and we theoretically study its convergence. We conducted an experimental study across a range of environments, discount factors, and off-policiness data collections to investigate the effectiveness of the regularization induced by Target Networks and Functional Regularization in terms of performance, accuracy, and stability. Our findings emphasize that Functional Regularization can be used as a drop-in replacement for Target Networks and result in performance improvement. Furthermore, adjusting both the regularization weight and the network update period in Functional Regularization can result in further performance improvements compared to solely adjusting the network update period as typically done with Target Networks. Our approach also enhances the ability to networks to recover accurate $Q$-values.

Published in Transactions on Machine Learning Research (09/2023)
Bridging the Gap Between Target Networks and Functional Regularization
Alexandre Piché
dezandrrlpiche@@@mall.com
ServiceNow Research Mila, Université de Montréol
Valentin Thomas
wtw..thomaas@mmail.ooomm com
Mila, Université de Montréal
Rafael Pardinas ServiceNow Research
rafael. rnfoll.anntina@@eeriiccoo...o
Joseph Marino DeepMond, London
Jloussmarino@@mmail.comm
Gian Maria Marconi
gmmarron@@gmailccmm
RIKEN Center for Advanced Intelligence Project
Christopher Pal
christopperr
Mila, Polytechnique Montréal
Canada CIFAR AI Chair
Mohammad Emtiyaa Khan
Khunyazz.hon@rrkeen..
RIKEN Center for Advanced Intelligence Project
Reviewed OpenReview: https: openreview. net/ forum? id= BFyoemrmqX
Abstract
Bootstrapping is behind much of the successes of deep Reinforcement Learning. However, learning the value function via bootstrapping often leads to unstable training due to fast-changing larget values. Target Networks are employed stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization still misunderstood. In this work, show that they act but also have disadvantages such as being inflexible and can result in instabilities, even when vanilla TD(0) converges. To overcome these issues, we propose an explicit Functional Regularization alternative that is flexible and a convex regularizer in function space and we theoretically study its convergence. We conduct an experimental study across a range of environments, discount factors, and off-policiness data collections investigate the effec- liveness of the regularization induced by Target Networks and Functional Regularization in terms of performance, accuracy, and stability. Our findings emphasize that Functional Regularization can be used drop-in replacement for Target Networks and result in performance improvement. Furthermore, adjusting both the regularization weight and the network update period in Functional Regularization can result. further performance improvements compared to solely adjusting the network update period as typically done with Target Networks. Our approach also cnhances the ability to networks recover accurate Q-values. The code is available here httph: attps://github. com/AleePiichefrr-tnlil
implicit regularizer which be beneficial in some cases,

---

Published in Transactions on Machine Learning Research (09/2023)
1 Introduction
Value functions are at the core of deep Reinforcement Learning (RL) algorithms. In order to learn a value function via regression, we need to estimate the target value the state. Estimating the target value using Monte Carlo roll-outs requires complete trajectorics and can have high variance. These issues make value estimation via Monte Carlo inefficient and umpractical in complex environments. Temporal Difference (TD) (Sutton 1988) algorithm addresses these issues by using bootstrapping (Sutton & Barto, 2018). Specifically, the value of state is estimated using the immediate reward and the discounted predicted value of its successor. Bootstrapping has potentially lower variance than Monte Carlo, particularly in the ofl off-policy settings, and does not require entire Irajectories to estimate the larget value. Bootstrapping can unfortunately be unstable as the target value is estimated with constantly updated parameters.
While stabilizing off-policy value function learning has been an active area of rescarch (Precup 2000 Precup et. al. 2001] Sutton et al.] 2008 2009). We focus on Targel Networks (TN) (Mnih et al. 2013) popular technique in deep RL which uses an additional set of lagging parameters to estimate the target value. Today TNs are central to most modern deep RL algorithms (Mnih ct al. 2013 Lillicrap et al.
2015 Abdolmaleki et al. 2018b Haarnoja et al. 2018 Fujimoto et al.; 2018 Hausknecht & Stone 2015 Van Hasselt et al. 2016 Hessel et al., 2018). Their popularity is due to the ease of implementation, little computation overhead, and their demonstrated effectiveness in a range of domains. Despite their popularity, the inner workings of TNs and how they stabilize the optimization of the value function remain unclear. While Van Hasselt et. al. (2018) empirically shows that TNs help but do not prevent divergence, Zhang et al. 2021p shows that variant of TNs can prevent divergence in simple cases.
In this work, we shed light on the regularization implicitly performed by TNs and how it is related to a regularization in the Q-function space whose strength is dictated by the discount factor 7, However, this rogularization cannot be written as the gradient a convex regularizor, as is commonplace in optimization. We prove that, because of this, TNs can unstabilize TD instead of making more stable. This leads us to propose a simple Functional Regularization (FR) alternative that has the advantages of provably ensuring TD remains stable while being more flexible than TN.
In our experimental study, we explored variely of environments, including the two-state MDP Tsitsiklis & Van Roy 1996), the Four Rooms environment (Sutton et al.) 1999), and the Atari suite (Bellemare et al. 2013), to assess the efficacy of regularization introduced by TN and FR in relation to performance, accuracy, and divergence. Our findings emphasize that Functional Regularization without regularization weight tuning can be used as a drop-in replacement for Target Networks without. loss of performance and can result performance improvement. Additionally, the combined use of the additional regularization weight and the network update period in FR can lead to enhanced performance compared to merely tuning the network update period for TN.
Contributions:
Target Network induces a rseuddo-regtlarizzaiion which effectively can slow down the Temporal Dif- ference (TD) optimization process. However, can also unstabilize TD, ie, make the iteration divergent while TD alone would converge.
Evidence: Explanation in Section and example in Section 4.1
2. There is continuous spectrum between FR and TD(0) and as such, for small enough FR has the same convergence properties as TD. Evidence: Proposition 3.2 and its proof in Appendix A.3
3. FR can result in improved performance when compared to TN and TD(0) in a variety of environ- ments, discount factors, and levels of *off-poliinness" Evidence: Demonstrated empirically on the Four Room environment (Section 4.2.2) and the Atari
suite (Section 4.3 Scction 4.4.2..
2

---

Published in Transactions on Machine Learning Research (09/2023)
2 Background
Preliminaries. We consider the general case of a Markov Decision Process (MDP) Puterran 2014 defined by p,r, 7: M), where S and A respectively denote the finite state and action spaces, p(s [s a represents the environment transition dynamics, ie the distribution of the next state taking action a state S. T: 8 x A + R denotes the reward function, Y E [0, 1) is the discount factor, and μ is the initial state distribution. For given policy, : S - ((A), starting state and action a, the value function Q" E IRIS x|Al is the expected discounted sum of rewards:
(s = E Σ
2'r((s, as) 5, - a
(1)
120
Additionally, the value function satisfies the Bellman equation
Bellman 1966
Q = + 7p 2 Q = TQ-
(2)
where R E RISH-A the reward vector and the state-action to state-action transition matrix Pr E [ISHA|IISHHAI where pa [(s, a(s, as, (s') = (a'ls') p(s and T RISHAAXIHHI + R (1-
is the Bellman operator.
Linear Function Approximation (LFA). The value function Q can be approximated via a linear func- tion. For fixed feature matrix & E RiSi-AAix,, our aim is to learn a paramcter vector of dimension p. w E RP, so that Qu &w approximates Q". We denote the of--policy sampling distribution by the diagonal matrix D E [RISI-AAA with diagonal entries d(s, a), positive and summing to 1. The update rule given by the TD(0) semi-gradient (Sutton & Barto 2018) is given by
Vh (w) = - D(R+7P**p Dw)
(3)
where the remainder of the return is estimated via bootstrapping, i.c., paew.
Deep Neural Networks (DNNs) Approximation. In complex environments, is difficult to adequately design features to approximate Q* with a linear function. Instead, non-linear functions, such as DNNs, are used to paramctrize Q. to automatically Iearn features and to improve the approximation to Q". But using DNNs to estimate Q using off-policy data and bootstrapping can be unstable and diverge (Sutton & Barto 2018 Van Hassolt et al. 2018). It is common to use lagging set of weights T to estimate the next value function to mitigate divergence (Mnih et al.) 2013 Lillicrap et al. 2015). The network parametrized by the set of lagging weights is commonly referred to as a target network. Despite the popularity of target networks, there are still gaps in our understanding of the impact it has on learning dynamics.
3 Functional Regularization as an Alternative to Target Networks
3.1 Understanding Target Network's implicit regularization
In this subsection, we will look more closely at the effects TNs have on the optimization. We place ourselves in the classical Linear Function Approximation (LFA) sctting (Sutton & Barto 2018) in order to be able to make theoretical contributions.
Target Networks MMii et al. 2013) are a popular approach to stabilize Temporal Difference learning. A periodically updated copy, Qm. of Q., is used to estimate the next state value. This causes the regression targets to no longer directly depend on the most recent estimate of w. With this, the Mean Squared Bellman Error for given state transition is
IN +1 1m
(4)
The semi-gradient Sutton & Barto 2018 denoted by V, of the expected squared Bellman error with TN can be decomposed (see prool in Appendix A
TN (w) = - D(R+>PPP* + 7 ww kw) + T DP
(5)
TD(0)
"Regalarizer"
3

---

Published in Transactions on Machine Learning Research (09/2023)
Written in this form, the effects of using TNs become clearer. The first term is the usual TD(0) update that we will not attempt to modify in this paper, and the second term can be interpreted as "regularization" term that encourages the weights w to stay close to the frozen weights w. This is in line with the intuition that TNs "slow down" or "stabilize" the optimization. However, this "regularizer' suffers from two main issues.
The first one is the weighting of the regularization which is equal to the discount factor 7. This may be problematic as y controls the agent's 's effective horizon (Sutton & Barto 2018) and is part of the problem definition. Therefore, TNs are inflexible in the sense that the weight of the regularizer cannot be controlled independently from the agent's effective horizon, and can only be controlled through the target update period T. We show experimentally in Section 4.2 that the update period T can be ineffective controlling the strength of the regularization.
The second and most important point that the "regularization" not a proper regularization term. It may be interpreted as the gradient of quadratic when Φ D p up is symmetric and positive, which is not the case in general. The underlying issue is that if Φ DP T has negative cigenvalue, the "regularizer" will actually push away from w instead of bringing them closer and thus will render the whole optimization unstable. This is an important theoretical issue as usually a basic requirement for a regularizer is that it cannot cause the original method to diverge. It is verified in Section 4.1 that TN can create additional divergence zoncs that were not present in TD(0).
Both aforementioned issues can be resolved simply. For the first point, we replace with a separate hyperparameter x 2 0. As for the second point, changing & DP to D ensures that the second term is a proper quadratic regularizer that cannot destabilize the optimization (Tihonov 1963 Tikhonov 1943
Lyapunov 1992).
3.2 Introducing Functional Regularization
Following the reasoning developed in the previous subsection, would like to design loss & whose semi-gradient is
Vol(w) = D Pp kw) + x DQ(( w))
(6)
The term x T ( w) is the gradient of the quadratic form
N: す [ (w- () D σ) = 11Q Q
as Qu = &w. We see that the natural resulting regularizer is a Functional Regularizer in norm I|-I|p, the usual norm when studying the convergence of TD in RI. (Bertsekas & Tsitsiklis, 1996). This leads to the Functionally regularized Mcan Square Bellman Error which penalizes the norm between the current Q-value estimate Qm (s, a) and lagging version of it (m,s, a), giving
FFR (w) = 11 + PP= Qw + -
1
K "1QQu - Q
where [7 denotes the stop-gradient operator. We observe that the up-to-date parameter w used in the Squared Bellman Error and there is the additional decoupled FR loss to stabilize w by regularizing the Q the regularization parameter and we recover TD learning for
function.
Critically, unlike Equation the target Q-value estimates supplied by the up-to-date Q-network, with the explicit regularization now scparately serving to stabilize training. As with TN, we can update the lagging network periodically to control the stability of the Q-value estimates. Overall, (FR is arguably similar in complexity to gTN requiring only an additional function evaluation for Q (5:41, a(+1) and an additional hyper-parameter, k. We report an exccution time analysis of replacing TN with FR in Figure t While the gradient update is now slightly longer on average, a training scenario a large portion of the time is spent inference and simulation which make the difference in gradient update time negligible.
4

---

Published in Transactions on Machine Learning Research (09/2023)
3.3 Stability of Target Network regularization and Functional Regularization
Now that we have highlighted polential issues that could arise when using TNs and proposed an alternative objective function, we will show that indeed TN can unstabilize Temporal Difference learning while FR, under some conditions, does not.
Theorem 3.1 (Target Networks can unstabilize TD). Using TD(0) with Torget Networks ( (Equation 4 can diverge while simply using TD(0) would converge.
A simple example of this situation is exhibited in Section 4.1 Note that in Appendix A.2 we precisely char- acterize the domain of convergence of TD()) with TN for large T and show that it changes the convergence domain of TD(O), leading to the theorem above.
On the other hand, for FR, we have
Proposition 3.2 (Convergence of Temporal Difference with Functional Regularization). If 4, D are chosen so that (D(0) converges, then for n. small enough, for T large enough, TD(0) with Functional Regularization is guaranteed to converge.
Proof. The proof can be found in Appendix A.3
The results above highlight the intuition the last subsections: while TNs can slow down learning they also interfere with TD learning, changing its convergence domain. On the other hand, FR with appropriately chosen parameters does not impact convergence while still being able to regularize the algorithm.
Finally, note here that results above do not mean that using TD()) with FR is always more stable than using it with TN. When TD()) is already unstable, depending the interplay between the features, the off-policy distribution, and the transition matrix, it would be possible in some cases for TNs to make the iteration stable. However, this is unlikely and shows that it is impossible when the features are low dimensional (see Corollary A.4.
Connection to GTD methods: Similarly to TN and FR updates Equation 5 and Equation 66 gradient TD methods (Antos et al., 2008) such as TDC and GTD2 (Sutton et al. 2009) make use of second set of weights. This second set of weights is used to add correction term to the TD update so that it approximates gradient update, thus not suffering from the divergence induced by the deadly triad. The correction term of gradient TD methods has strong similarities with the TN correction 7Φ DPQ(w @) of Equation However the additional set of weights used in gradient TD methods is the result. of least squares optimization process and approximates a projection of the TD error for the current state. Therefore, the thcoretical convergence these methods rely on a two-timescalc argument where the second set of weights must converge close to their optimal values. In practice, it seems that these algorithms work best. when the learning rate for the second set of weights is very small (Ghiassian al.) 2020), thus making these gradient TD methods behave like TD()). Furthermore, the update for the second set of weight needs importance sampling in the off-policy case and adapting the algorithm to deep learning is not straightforward (Silver) 2013). Thus these methods, while enjoying some theoretical guarantees, are not practical for most modern settings and are quite different from FR. FR and TN, by working directly in the function space, avoid complications related to being used with deep neural networks. However neither FR nor TN enjoy strong convergence guarantees in presence of the deadly triad as we have shown above.
3.4 From value estimation to control
While in the previous subsoctions we have discussed the value estimation case, we can also use FR for control, for instance with Q-learning. As the update in Q-learning can be seen as a TD update where the next Q- value is sampled from the greedy policy, FR can Atraightforwaril be adapted for control. We illustrate the practical difference between using FR or TN for control in Algorithm
In Algorithm we provide code for a simple Q-learning agent using either Target Networks or Functional Reg- ularization. Note that the change is minimal and adds little complexity. For our experiments in Section 4.2.2
5

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm Deep Q-Network (DQN) Algorithm with TN FR
Initialize replay memory D with capacily D
Initialize action-value function Q with random weights 0 Initialize target weights 4-0
4: for episode = to M do
5: Initialize state 8 ~ lr
6:
for = 1 to .V do
With probability select random action a, otherwise a + argmax, argmax,Qoss.a) Execute action in environment, observe reward and next state s' Store transition rs,,,) in D
10:
Sample random mini-batch of transitions (s), ag, (3:8)) from D
11:
// Set the target
12:
TN: Dj - r3 + max Q7(5 d)) FR: Vy + + max Qo(s) a') If 8' is terminal then yj f-1
13:
14:
15:
// Define the loss
16:
TN: ((0) = (v/ a Qo(s),
FR: ((0) = 1 ([v, az)) + (Qo()), ),(ss,,,))
17:
18:
Perform a gradient descent step on E(0) with respect to
19:
Every T stops set x-
20:
// Updale slate
21:
If s' is terminal then μ4 else s
22:
end for
23:
end for
Section 4.3 and Section 4.4
DQN variation (Mnih et al. 2013) of this algorithm which include sam-
pling mini-batches from replay buffer, using the Adam optimizer Kingma & Ba 2014 and using e-decay,
4 Experiments
In this section, we investigate empirically the differences between FR, TN and TD(0). First, we aim to understand the convergent. and divergent behaviors of these algorithms. We use novel visualization to give intuition and validate our theoretical results. Finally, investigate if the use of the additional regularization weight. и combined with the network update period T can lead to better performance compared to vanilla Deep Q-learning (DQL) and to merely tuning the network update period T for TN for varying degrees of off-policiness and discount factor on the Four Room cnvironment (Sutton 1988 and the Atari benchmark [Bellemare et al. 2013).
4.1 Simple 2-state MDP Off Policy Evaluation
4.1.1 Experimental Set-Up
In this subsection, we visualize the behavior of TN and FR on a simple MDP. This illustrates the theory developed in Section 3 on the convergence properties of both algorithms. Baird (1995); Tsitsiklis & Van Roy (1966) and Kolter (2011) introduced simple reward-less MDPs with few states on which TD(0) can exhibit divergence. Similarly, we define 2-state MDP in Figure la which is conceptually close to [Titsikkis & Van Roy (1996). Each state transitions to the other with probability 0.95 or transitions to itself with probability 0.05. We use 1-d features for these two states: d(su) and d(s1). We denote the off-sampling distribution of 80 by d(so) and by (80) the on-policy sampling distribution of 8o which is equal to 1/2 in this specific case. We use 7 = 0.995.
Instead of just exhibiting single instance D, for which we have divergence, we showcase the convergent divergent behavior of TD(0), TD(0) with TN and TD(0) with FR for all combinations of D and +

---

Published in Transactions on Machine Learning Research (09/2023)
0.05
0.05
0.95
80
81
0.95
(a) 2-state MDP
(b) TD(O)
(c) TN
(d) FR
Figure 1: Convergence rate for different off-policy and feature combination in a 2-state MDP. We show here the spectral radius for TD(O) (b) only, (c) with Target Network, and (d) with Functional Regularization
the simple MDP (a). For any point in the disk, its radius represents the off-policy The color represents the value
distribution of 80 while the angle is the same as the vector [0(so), of the spectral radius for each method while the contour is the frontier p separating the convergence domain (shades of blue) from the divergent domain (shades of red). The black circle has a radius т (so), the stationary distribution of P for which TD is guaranteed to converge as would be on-policy. The fact that Target Networks might cause policy evaluation to diverge for some features where simple TD converges is a surprising and important limitation of Target Networks given their ubiquity.
simultaneously.
The Euclidean norm of Φ; |10l|2 does not impact the convergence of any of the aforementioned algorithms, as can be renormalized if we scale the learning rate 17 by its squared norm. Therefore, only the angle of the vector 4- = (Ø(so), ((s)) is needed to describe all reprosentations. In our context, there are only two states, and thus the off-policy distribution is entirely determined by d(so). Thus, one can represent the set of all representations and all off-policy distributions (D, +) disk of radius 1 I In polar coordinates, the radius of a point on the disk represents d(so) while its angle the one of the vector 4 i.e., so(() arctan ((50)
d(s1)
4.1.2 Results
We draw the above-mentioned disks for the algorithms: TD(0), TD(O) with TN, and TD(0) with FR (Fig- ure 1b. 1c and ldp. On these disks, the colors represent the spectral radius p of the iteration matrix for each algorithm. Recall that p < 1 (blue) implies convergence, while 1 (red) implies divergence to to facilitate visualization. Additionally, d(so) = (so) (in black), which denotes the stationary distribution. As predicted by theory, TD(0) converges on this circle for all representations. We use the regularization coefficient k = 1.5 and target update period T 10, 000 for TD(0) with TN and TD(O) with FR.
+oo. We plot the contour lines
draw the circle of radius
As seen in Figure I FR. (Figure Ic] and TN (Figure [c] show less extreme values (lighter blue) compared to TD(0) (Figure 1b This finding implics that both convergence and divergence are slowed down, which is expected for regularization methods.
We recover the classical divergence example for TD(0) Tiitsikkis & Van Roy 1996 Van Hasselt et al. 2018 if ((s)) > 6(so) > 0 and the behavior distribution d(3o) ) is close to 1, the point (d(so), (+)) in polar coordinates lies in the large upper red region Figure 1b thus TD(0) will diverge.
As proved in Proposition 3.2. the convergence domain of TD(0) with FR is similar the one of TD()). Interestingly, and as shown in Theorem 3.1 TD(O) with TN may diverge in regions where TD(0) with FR does not. For example, in Figure Ic we see four smaller (symmetric) additional divergence regions (red) in the lower right and upper left regions of the disk for d(so) close to 1 and two symmetric ones near the center of the disk. This can happen when $ DP which appears in the implicit regularization of TN has negative eigenvalues.
Impact of " "Off- Policyooli:,:ss" As previously reported by Kolter (2011) and Scherrer (010 we also observe that when the data collection is on-policy (black circle of radius f)) for every method and every
Note that technically only need half-disk illustrate the convergence behavior, but plot the whole disk for aesthetic

---

Published in Transactions on Machine Learning Research (09/202))
(a) 7 = 0.8
(b) 7 = 0.95
(c) 0.995
(d) = 0.9995
Figure 2: Influence of the discount factor ? on (0) divergence regions. As we increase the area of the divergence regions (in red) increases. Specifically, as y approaches 1, the divergence regions almost ex- tend the on- on-policy case, meaning that some behavior policies close the on-policy distribution that might have been stable for lower 7 become unstable for higher 7. A similar behavior also observed for TN and FR.
discount factor y the method does not diverge. However as we move away from the on policy stationary distribution (increasing or decreasing the radius) the methods have divergence zones.
Impact of the discount factor 7: It is known that 7 plays a strong role in the divergence of TD methods [Kolter] 2011 Scherrer 2010 In Figure we provide an exact illustration of this relationship between the discount factor Y and the convergence or divergence behavior of Temporal Difference methods in our simple MDP. Intuitively, as the divergence issues stem from the fact that yP is non-symmetric, we can expect + to be stable as our problem would simplify into a weighted least squares problem. On the other hand, the divergence issues could be made worse for higher 7,. This intuition is validated on Figure 2 where we can see that the divergence regions (in red) increase with y for TD(0). We provide similar figures for FR and TN in Appendix C.2 Interestingly, as on-policy TD(0) is always stable, we observe that for smaller y
afford to be more off-policy than for higher y where any small deviation from the on-policy distribution could lead to divergent behavior, depending on the features d.
These depictions of the convergence or divergence behaviors of TD(0) with FR and TD(0) with TN hint at the fact that TN-based algorithms may be more unstable than FR-based algorithms for strongly off-policy distributions. This hypothesis is studied empirically the next subsoction.
4.2
Four Rooms Results
4.2.1
Experimental Set-Up
In this section, we study the behavior of vanilla. DQT., TN, and FR in the Four Room environment (Sutton let al. 1999). The environment is comprised of four rooms divided by walls with a single gap in each wall allowing an agent to move between rooms. The environment size is 11 x 11. The agent's position is given by one-hot encoding, and its available actions are up, down, left, and right. The agent starts in the bottom right position and must reach the goal in the top left position to obtain a reward of 1. At each step that the agent does not reach the goal, it receives small penalty of 0.01. The episode terminates after 121 (=11²) steps, thus an agent that never reach the goal would have a return of -1.21.
We investigate how varying the discount factor y and exploration level c can lead to divergence both their Q-value error and results. Specifically, the data is collected under behavior policy that takes random action with probability and takes arg max, Q (s, a) with probability 1 1- As tends towards 1, the data collection becomes more off-policy and makes the Q-value estimation more difficult. Similarly, as shown above, increasing the discount factor increases the difficulty of the bootstrapping and can lead to divergence.
We collect 10000 environment transitions and perform the same number of gradient steps. At the end of train- ing, we report the average per-opisode regret for 100 episodes collected under Ceval - greody policy with Ceval = 0.1 it is common for RL agents in deterministic environments such as the Atari suite. Furthermore, better understand the behavior of the algorithms studied, we measure 1 =l - 11, where Q" was obtained
8

---

Published in Transactions on Machine Learning Research (09/2023)
Exploration cw0.5
Exploration c-0.75
Exploration g=0.95
Discount y= 99
Retur (r)
0.25
-0.50
0.75
1.00
0.75
0.50
Discount y=0.9999
Retum [t]
-0.50
0.75
-1.00
-1.25
0.1 0.25 0.5 1 2.5 5
10
25
50
100
0.1 0:25 0.5 1
2.5
10 25
100
0.1 0.25 0.5 2.5
100
Network Update Period 100)
Network Update Perlod T 100)
Network Update Period (x 100)
DQL
TN
FR 0.1
FR P
FRx
(a) Final return for different hyper-paramette (Higher better) We report the average performance over 100 episodes for seeds on the Four Room environment. We observe that FR outperforms TN for combination of update period T and regularization weight &. across every scenario studied. Expleration
Exploration
10P
Discount y=0,3
07
10-1
10
10a
10a
Discouunt y= 9599
10a
10a
10
0.1 0.25 0.5
2.5
50 100
0.25 2.5 5
10
25 50
100
0.1 0.25 0.5
2.5
100
Network Update Period (x 100)
Netwerk Update Peried T 100)
Network Update Period 100)
(b) Final loss for different hyper-parameters (Lower better) We report the difference between the Q-value obtained using function approximation Qs and the optimal Q-value Q" estimated using a tabular method. We can that the decreases the performance increases (Figure 3a) We also observe that FR 0.1 obtains lower
Q errors than TN for every scenario studied.
Figure 3: Four Room Ablation Study

---

Published in Transactions on Machine Learning Research (09/2023)
Median (T)
IQM (1)
Mean (T)
Optimality Gap (4)
TN T=5000
FR 1 T=5000
T=100
FR x= T=1000
0.4
0.5
0.6
0.7
0.54
0.60
0.66
4
8
10
0.48
0.52
0.56
Figure 4: Atari Suite Human Normalized Score. Different metrics for the Human Normalized Score with their 90% confidence interval on the whole Atari suite.
using tabular Q-learning, Qu is a DNN approximation (details in Table D,, and n the number of state- action pairs. We repeat the evaluation for 40 seeds. We investigate the performance of vanilla DQL, TN and FR under 6 different combinations of discount factor y E (0.99, 0.9999) and exploration {0.5, 0.75, 0.95).
4.2.2 Results
The returns and the Q-value errors for the Four Room environment can be observed in Figure 3a and Figure 3b
respectively.
Increasing exploration e, Increasing exploration from 0.5 to 0.75 does not result in loss performance or worst Q-value approximation for the FR algorithms (in shades of blue). However, TN's (red) performance and its Q value approximation degrade. Increasing exploration past 0.75 to 0.95 decreases FR's performance 0.95 slightly increases TN performances, but
and Q-value approsimation. Interestingly, for exploration decreases its Q-value approximation accuracy. The performance of vanilla DQL (yellow) stays fairly constant as we increase exploration but for discount = 0.99 its Q-value error incrcases, while it diverges for every exploration level for discount = 0.9999.
Increasing the discount factor 7/. For smaller exploration e, increasing the discount factor y from 0.99 to 0.9999 results in much worst performance and Q-value divergence for small network update periods T. While several different combinations of network update period and regularization resulted in best performance for discount factor y 0.99, large network update periods and small regularization weights result in better performance and smaller Q-error for y 0.9999. The performance and Q-value approximation of TN and vanilla DQL degrades greatly. For exploration = 0.95, increasing the discount factor also canses the Q-value to diverge for smaller network update periods T, but interestingly does not result a large decrease in performance for larger network update period T for both TN and FR, while the performance of vanilla DQL collapses.
Overall performance. In Figure 3a for discount rate ? 0.99, there
number of regularization weights and network update periods T hyper-parameters for which FR reaches optimal near optimal performance, suggesting that our method is not particularly sensitive to the choice of the hyper-parameter K. However, for exploration e < 0.75 and for discount rate = 0.9999, we observe that optimal performance and Q-value approximation can only be reached through a combination of regularization weight e smaller than and large network update period benefits of tuning the regularization weight к and network update period T for FR compared simply tuning the network period update for TN. Furthermore, we note that in every settings studied, FR outperforms both vanilla DQL and TN. Finally, for every method we observe an inverted "[" shape that highlight the regularization trade-off where too little regularization results in divergence, while too much regularization results in slow learning.
Highlighting the
4.3 Atari Suite
4.3.1
Experimental Set-Up
In this section, we investigate if FR without additional tuning (N 1) can be used as a drop-in replacement for TN on the Arcade Learning Environment, (ALF) (Bellemare et. al. 2013). We the CleanRL library (Huang
10

---

Published in Transactions on Machine Learning Research (09/2023)
et al. 2022), and we run each algorithm for 10M steps and use 7 seeds for each game in the suite which amounts to total of approximately 60, 000 GPU hours. We keep every hyper-parameters to their default value with the exception of the network update period T that we varied.
While all learning curves for all games are available in Figure 10 we use the rliable library and the robust metrics introduced in (Agarwal et al. 2021] to analyze results over all games. In Figure 4 we report the usual mean and median scores, as well as the robust metrics interquartile mean (IQM) and optimality gap as well as their 90% confidence interval. The IQM is the mean where the top and bottom 25% of the runs are discarded. It can be understood as halfway between the mean and the median as the mean would not discard any runs and the median would discard the top and bottom 50% of the runs to only keep the median. An additional robust metric is the optimality gap which measures how far the algorithm is from reaching a score of 1, the average human score, on all games.
4.3.2 Results
Network update period T = 1, 000. We first compare FR Λ ==1 1 to TN with the default network update period T = 1, 000. We observe that the mean is significantly higher for FR к = (light blue) than for TN (light red). Furthermore, the average median and the IQM are higher for FR than TN, but their distribution are somewhat overlapping, we also note that the optimality gap for FR is noticeably better than the one achieved by TN.
Network update period T = 5, 000. We then compare FR = 1 to TN for the best network update period T = 5, 000 obtained for TN in Section 4.4 We observe that the mean and median distributions are mostly overlapping for TN (dark red) and FR = (dark blue). While we cannot claim statistical significance for our sample size, we note that the IQM is higher and optimality gap is lower for FR K = 1 than TN. Furthermore, we note that despite FR x = = 5000 does not significantly improve upon TN T = 5000 for our sample size, we observed noticeable performance improvement on the following games: atlantis, enduro, kung fu, zaxxon, crazy climbers, seaquest, frostbite, and robotank.
Overall performance. We conclude that for the network update periods tested, FR matches or out- performs TN on most games of the ALE without additional hyper-parameter tuning (setting k = 1). Further performance improvement could potentially be reached by tuning k. in concert with the target network update period as done in the Four Room environment.
4.4 Sensitivity Analysis of the Regularization Parameters on a Subset of Atari Games
4.4.1
Experimental Set-Up
In this section, we investigate if the findings from the previous sections hold for diverse and challenging environments from the Arcade Learning Environment (Bellemare et al., 2013). Specifically, we investigate the behavior of deep Q-learning under the following 6 environments from the orignal DQN paper (Mnih et al., 2013): Seaquest, Breakout, Space Invaders, Enduro, Qbert and Beam Rider For each environment, we decay the probability of a random action from to e and the discount factor y use to train the Q-value. We report the results for different y and e since they can both increase instability and result in divergence. Unless specified otherwise, we use the default syper-paramcters from the CleanRL library (Huang et al. 2022).
We study the average return of the Geval-greedy policy after 10M environment steps, where Geval 0.05. Furthermore, we report the mean Q-value at the end to study divergence. In the next section, we report the results averaged over the 6 games for 5 seeds each for total of 30,000 GPU hours. To combine the results of various games, we standardized the scores of each game. This involved comparing the scores obtained from playing each game with a random policy to the highest scores achieved by any of the policies analyzed. Non-standardized game results are provided in the appendix.
Excluding Pong since every algorithm achieves near perfect score.
11

---

Published in Transactions on Machine Learning Research (09/2023)
Exploration 2=0.01
Exploration c=0.1
Exploration gw0.25
Return (t)
Recerni peri 39
0.4
Retum 613
0.3
0.5
5
10
50
100 250 500
0.5
5
10
25
50
100
250
500
0.5
25 100 250 500
Natwork Update Period ix 1000)
Network Update Period (xx 1000)
Netwark Update Perlod 1000)
DQL
TN
FR 0.5
x 2.0
x.5
(a) Final return for different hyper-parameters (Higher better) We observe that increasing the exploration and the discount rate y result in performance decrease for every algorithms. However, we note that mix of larger network update period T and regularization weight x mitigates the performance decrease. Exploration
Exploration
Exploration e=0.1
1022
10fe
Mean Q-value
Discount y=u 99
10a
104
109
Discount y=0. 3999
Mean Q-value
104
10a
10a
0.5
5
10
25
50
100 250 500
0.5
1
10
25
50
100
250
500
0.5
1 5
10 25
50
100 250 500
Network Update Period T (x 1000)
Network Update Period T 1000)
Network Update Period T 1000)
FR
FR
(b) Final predicted mean Q value (The green dotted line denotes divergence) Mean Q values larger than the green line are considered diverging. We report the mean Q-values over the environments studied. observe that larger discount rates and more exploratory policies increase the (soft-) divergence problem. We also observe that increasing network update period or the regularization weight mitigate the divergence problem.
Figure Atari Ablation Study
12

---

Published in Transactions on Machine Learning Research (09/2023)
4.4.2 Results
The returns and the mean Q-values prediction for the normalized Atari environments can be observed in Figure 5a and Figure 5b respectively.
Increasing exploration 6. In Figure 5a for discount rate = 0.99, we observe that for e = 0.01 a large regularization weight k = 2.5 and small network update period T results in the best performance. Increasing exploration e to 0.1, the performance of FR is fairly similar for k E {0.5, 1.0} and TN, but slightly worst for R = 2.5. For c = 0.25, the performance of every method studied drop, and x achieves the highest performance. Furthermore for all methods, we observe an inverted "U" shape appearing as we increase T highlighting the regularization trade-off. Interestingly for smaller exploration c, the algorithms studied do not diverge, but for exploration = 0.25 we observe that for smaller network update period T, vanilla DQL, TN and FR. now diverge. For discount rate = 0.9999, we also observe that increasing the exploration past 0.1 decreases overall performance; and that the shape of the return curves exhibit a similar inverted "U" shape. As we increase 6, for each regularization weight K, we observe that the best performing network period update T 100000 does not change. We further observe that regularization weight x and network period update T' are both effective at reducing Q-value divergence. For example, for e = 0.25 and T 5000, FR E (0.5, 1.} diverge, while FR = 2.5 does not. Overall, incrcasing exploration decrcases performance and can result in soft-divergence of the Q-values.
Increasing the discount factor y. We observe that for a discount rate of = 0.99, both TN and FR cxhibits consistent performance across a broad range of regularization values a and network update periods T. Moreover, we observe that with a discount factor of y = 0.99 and exploration < 0.25, the Q-values do not diverge. However, for < <= 0.25, a combination of low regularization weight x and update period T leads to divergence. We again note that for fixed network update period e.g., T 1000, increasing the regularization weight prevent divergence. As the discount rate is increase to y 0.9999, we observe consistent drop in performance. We also observe, that this higher discount rate makes the model more sensitive to regularization hyper-parameters yielding a more pronounced inverted "U" shape that clearly illustrates the regularization trade-off. Furthermore, we observe that increasing Y causes the Q-value to diverge for DQL, FR, and TN when using smaller network update periods T. We observe that increasing the update period T and regularization g can prevent this divergence. Overall, incrcasing the discount factor 7 can degrade performance and lead to divergence for smaller T. For larger T, we observe small degradation in performance and no divergence.
Overall performance. We find that best overall performance is attained when the regularization weight k = 2.5 and the network period update, T, is small. This emphasizes the significance of the additional regularization weight, K, for achieving good results with FR. Furthermore, while their performance profile is similar, we observe that FR with x = 1 matches or slightly outperforms TN in all scenarios examined, indicating that FR without tuning e can effectively serve as a substitute for TN in the Atari suite and to match its performance or result in performance improvement.
5 Related work
Multiple previous works have investigated how to improve value estimation through various constraints and regularizations. Shao et al. (2020) proposed adding an additional backward Squared Bellman Error loss to the next Q-value to stabilize training and remove the need for a target network on some control problems. Farahmand et al. (2009) is perhaps the closest work to our own, they regularize the reproducing kernel Hilbert space (RKHS) norm of the Qavalue estimates, i.e., penalizing x| xiIQQ.ll 1.... However, penalizing the magnitude of the Q-values would prevent the algorithm from converging to Q* if x does not tend towards 0. Penalizing the output of a DNN draws connections with popular regularization methods in the field of policy optimization which were inspired by trust-region algorithms (Schulman et al. 2015a Abdolmaleki et al. 2018b a in which the policy is KL regularized towards its past values. In the same spirit, for value- based methods, our proposed Functional Regularization regularizes the Q function estimates towards their past values. Alternatively, Kim et al. (2019) showed that by using the mellow max operator as an alternative the max operator used in bootstrapping, was possible to stabilize training and train without target network some Atari games.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Other works have sought to stabilize the Q-value estimates by constraining parameter updates, e.g., through regularization (Farebrother et al. 2018), conjugate gradient methods (Schulman et al. 2015b), pre-conditioning the gradient updates (Knight & Lerner 2018 Achiam et al, 2019), or using Kalman filtering (Shashua & Mannor 2019 2020p. However, weight regularization might be ineffective in DNNs. For neural networks, the network outputs depend on the weights in a complex way and the exact value of the weights may not matter much. What ultimately matters is the network outputs (Benjamin et al. 2018 Khan 2019), ; and is better to directly regularize those. For instance, while Polyak's averaging [Lillicrap a common weight regularization technique, has found success in control problems (Haarnoja Fujimoto et al., 2018), periodically updating the parameters is usually preferred for complex
et al.
et al.
2015
et al.
2018
DNN
architectures (Mnih et al. 2013 Hausknecht & Stone 2015 Hessel et al. 2018 Kapturowski et al.
2018
Parisolio et al. 2020).
Recently (Zhang et al. 2021 studied how Target Networks can help stabilize the TD algorithm, which is soemingly at odds with our Theorem 3.1 However, in their paper, they study variant of DQN with TN which involves projection steps while our analysis focuses the version used in practice.
6 Limitations
1. FR is not guaranteed (thcoretically and empirically) to always better than TN. Indeed, for value estimation, TN can, in some cases, stabilize a divergence TD iteration while FR. cannot.
2. Compared to TN, FR adds new hyperparameter and needs one additional forward propagation. However we find easy to tune in practice, the default value k = being strong baseline.
3. Our theoretical analysis limited to the batch value estimation setting with linear function approxi- mation. This means that we do not analyze the stochastic case where states and actions sampled.
Additional research is required to study the interaction of FR with more sophisticated algorithms such as n-stop return and double Q-learning.
7 Conclusion
In this paper, we cast a light on the implicit regularization performed by using Target Networks in RL. We have shown that despite being used as a regularizer Target Networks can cause instabilities while also being inflexible as the regularization depends on the discount factor 7. To overcome these issues, we introduced Functional Regularization which direetly regularizes the value network towards network parameterized by lagging parameters.
We performed extensive ablation studies on TN and FR to understand their behavior on wide variety of settings. Overall decoupling the regularization parameter x results in a more flexible behavior and can lead to better performance. Importantly, we also observe that using the default λ = 1, FR matches or outperforms TN on our Atari experiments. This indicates that FR could be used as a drop-in replacement for TN in deep RL.
Broader Impact Statement
This work is mostly theoretical in nature and we do not foresce direct negative socictal impacts.
Author Contributions
AP had the initial idea, implemented the experiments on Atari and 4 Rooms, and derived a first version of the link between TN and FR (Eq 5). VT derived the theory and implemented the 2-state MDP experiment. RP helped setting up the cluster to conduct the experiments. JM helped with the derivations in previous version of the paper. GMM, CP and MEK advised the project.
14

---

Published in Transactions on Machine Learning Research (09/2023)
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller Relative entropy regularized policy iteration. or preprint arXiv:1812. 02256, 2018a.
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.0000 06920, 2018b.
Joshua Achiam, Ethan Kuight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXic:1903.08.. 08894, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29322 29320,, 2021.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and single sample path. Machine Learning, 71:89 129, 2008.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30 37. Elsevier, 1995.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 2299, 2013.
Richard Bellman. Dynamic programming. Science, 153(3731):34 37, 1966.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint ar0:v:1805. 08289, 2018.
Dimitri Bertsekas and John Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
Amir Massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized fitted q-iteration for planning in contimuous-space markovian decision problems. In 2009 American Control Conference, pp. 725-730. IEEE, 2009.
Jesse Farebrother, Marlos C Machado, and Michael Bowling, Generalization and regularization in dqn. arXiv preprinl Xio:1188000000 00123, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Internalional Conference on Machine Learning, pp. 1587-1596. PMTR, 2018.
Sina Ghiassian, Andrew Patterson, Shivam Garg, Dhawal Gupla, Adam White, and Martha White. Gra- dient temporal-differee diffeeeeee learning with regularized corrections. In International Conference Machine Learning, pp. 3524 3534. PMLR, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018
Matthew Hausknecht and Peter Stone. Deep recurrent q-lcarning for partially observable mdps. arXiv preprinl ar xix::11507.0652 2015.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabuey, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality singlo-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 2022. URL http://jmmr: httppir...gppprs//////// f html
15

---

Published in Transactions on Machine Learning Research (09/2023)
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning represenlations, 2018 8.
Mohammad Emtiyaz E Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into gaussian processes. In Advances in neural information processing systems, pp. 3094-3104, 2019.
Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the Twenly Fighth International Joint Conference on Artificial Intelligence, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412. 6980, 2014.
ar preprint
Ethan Knight and Osher Lerner. Natural gradient deep q-learning. orXiv preprint arXim: 2018.
J Kolter. The fixed points of off-policy td. Advances in Neural Information Processing Systems, 24:2169-218 2011.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.082 02997, 2015.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3):531-534 534, 1992.
Volodymyy Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXie preprint arXiv:1312.5009 2013.
Emilio Parisotto, Francis Song, Jack Rac, Razvan Pascamu, Caglar Gulcchre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaulman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforce- ment learning. In International Conference on Machine Learning, pp. 7487-7498. PMLR, 2020.
Doina Precup. Eligibility traces for off-policg policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pp. 417 424, 2001.
Martin L Puterman. Markow decision processes: discrele stochastic dynamic programming, John Wiley &
Sons, 2014.
Bruno Scherrer. Should one compute the temporal difference fix point or minimize the bellman residual? the unified oblique projection view. arXiv preprint arX1v:1011.4962, 2010.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint Niw::1506.02400 2015b.
Lin Shao, Yifan You, Mengyuan Yan, Qingyun Sun, and Jeannette Bohg. Grac: Self-guided and self- regularized actor-critic. arXiv preprint or arXiv:2009.08:: 08973,, 2020.
Shirli Di-Castro Shashua and Shie Mannor. Trust region value optimization using kalman filtering. arXio preprint arXtw:1901.007660 2019.
Shirli Di-Castro Shashua and Shie Mannor. Kalman meets bellman: Improving policy evaluation through value tracking. Xixiv preprint 071120000071711 2020.
16

---

Published in Transactions on Machine Learning Research (09/2023)
David Silver. Gradient temporal difference networks. In European Workshop Reinforcement Learning. pp. 117-130. PMI.R, 2013.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3(1):9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211 1999.
Richard S Sutton, Csaba Szepesvári, and Hamid Reza Maei. convergent o (n) algorithm for off-policy temporal-difference learning with linear function approximation. Advances in neural information processing 2008.
systems, (1(21):1609-1
Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-ddiffrencce learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning, pp.
993-1000, 2009.
Andrei Nikolajevits Tihonov. Solution incorrectly formulated problems and the regularization method. Soviel Math., 4:1035 1038, 1963.
Andrey Nikolayevich Tikhonov. On the stability of inverse problems. In Dokl. Akad. Nouk SSSR, volume 39, pp. 195 198, 1943.
John N Tsitsiklis and Benjamin Van Roy. Featuro-based methods for large scale dynamic programming. Machine Learning, 22(1):59-95) 94, 1996.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint Xrx: 2.06488, 2018.
Shangtong Zhang. Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with target network. In International Conference on Machine Learning, pp. 12621-12631. PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (09/2023)
A Proofs for Linear Function Approximation case
A.1 Supporting lemma
Lemma A.1 (Difference of inverses). For A and B two non-singular matrices
A B = (B A)B-1
Proof. We multiply each side by A on the left and B on the right and show they are equal. Left hand term:
A(A - B B= A
And for the right hand term:
(B- - B B- A
By equating both terms and multiplying by A-1 on the left and B-1 on the right we have the equality.
Proposition A.2 (Convergence of TD-TN). For Tl $ $ D the projection malrie onlo the span o, if (IIl, PT) < then for T large enough, Target Network Value Iteration is guaranteed to converye.
A.2 Proof for the domain of convergence of TD(0) with TN
The solution for the TD itcration is
w² (TDD(T >P" 1 DR
A.2.1 Regularized iteration
Let us call t7 the frozen weight. We consider in this subsection only the "inner loop iteration', i.e the ones for fixed frozen weight vector. The update for TD with larget network is
wh+1 W/ n D(R + y pp pw;)
We call (17) the fixed point of that iteration rule which satisfies
w w0) = (Φ D T D(R + y (
(9)
Thus our updates satisfy
W++1 ww) W; + 1/ D(R p 4w.)) w² w²(1i)
(T 1//b T DE ( + 1 T D(R + p. p w (m)
(1 1740 D4 D n D4w" ( w" w()) using
(I n DT -0 (T))
So
Wi - w (w) = (1 - 17 DD)'((0 nn (w))
(10)
As T D&p is symmetric real with positive cigenvalues, n X with d its largest eigenvalue, wy guaranteed to converge linearly to w² ww))
18

---

Published in Transactions on Machine Learning Research (09/2023)
A.2.2 Distance between regularized and original optima
Now we look at the vector w" ( w*
w" (w) w" (Φ P (+D - DD
(($) - -
DR + (6) D4)
- -
PP
--1(#" D2) "P" ( -
-))))) - 1 DR+ >2+000 DΦ) -'' DPPT, using Lemma A.1
=->(#' - DP tw + - DPb0, By definition of w²
=>( D$) b DP (
We summarize it here, this is the bias between the regularized and true optima
7 D DP (( w")
(11)
A.2.3 Distance between inner loop iterate and optimum
Tk (T) w" (T( w" w" + w (w) w'
(I n/ D②) K (w w" w'(w))))))))))))))(() + 7 ( 1 DP
(I n/ DΦ) K (1 w² - (w) - w/) ( 00000000000000000000000000 DP - ww)
[(I 1740 D K (I ( D DP + 7 D - DP 4 w²)
It is possible to rewrite it in value space by multiplying by & the left
(x(Q)- - Q- = =b T/ T 1
- Q") 1 160
w')
④(I 170
n/)40
D&(1 7(47 D DP u (Q Q2)
( 17th DD) -1) -1)) D(I ( -
+ + p p=(@ Q*)
4(I 1/ - -
T " D(I -1I. -Q) + P Q*)
[Ø(] 174 D k DΦ)-1 nI) D(I yII P*) + 111 (" -Q))
(ē
The interesting bit here when analyzing how this behaves for large K is that (1 (1 D a matrix that can become arbitrarily small when K is large.
K-1 is essentially
Too QQ' P Q*)
- Q-
(12)
This is guaranteed converge if TT a contraction, which is the case for instance when the stationary distribution of P.
Also Q ow" is the fixed point of the projected Bellman iteration, have
T~ (Q(z)) = Hy (R (R+
This is the projected value ileration projected policy evaluation algorithm Bertsekas & Tsitsiklis
1996).
19

---

Published in Transactions on Machine Learning Research (09/2023)
A.2.4 Continuity of norms and spectral radius
Now if we suppose that p(IIg.P)) < 1 or written differently p(rIIg S a < 1, as is a continuous function of its input entries, for K large enough, we can guarantee that
VE O, SK E N/p(0(11 - r
DØ) -11) - P(I- P) yllx < cate
In particular for a which would ensure the convergence of the algorithm.
The lower bound K can be made more precise using the Bauer-Fike theorem ((A+B) ((A)) We can write
Pk s (P(TTT + <=[(T 1/t DE
- (T DT) DP*4)|
S (T(l) + CTI -77 D
where C K - - - - DP* for k the condition number of the eigenvector matrix of II
So we can take
log 1- T Pr)
K
log (1--1 D&II
which ensures the spectral radius of TD-TN is <.
A.3 Proof for TD-FR, Proposition 3.2
Let us consider the case where parametrize with linear function approximation, Q Dw, $ E ||S)- w E IRP.
A.3.1 Part 1: Fixed point of the regularized loss
Let us look at when the gradient is O:
$ T D(R+ 7 kww + TDDDDD (7) =0
Thus TT
K- - STDR+ K DTb.. We call
wx wx(w) (m) =
(T
DRR DR RTT DPw)
the solution of the regularized problem. matrix. The TD(0) update is therefore
We denote A. TDDD( + >P*) the regularized iteration
W++1
(T - 1-mAA)) + 1)( DR + nDb DTw
= (1 ( -1- Aw + w.
Thercfore
We+1 wx(w) A)0x)) w((w))
(13)
20

---

Published in Transactions on Machine Learning Research (09/2023)
A.3.2 Part 2:
Difference between regularized optima and true optima
Let us look at the quantity (xx( - w² where w² = A &TDR. DR, the solution of the unregularized problem. First, we will use the fact that A-1 1-1-B-1- A ( A)B to gel
Wx (x( w² A= DR + x D A 1 DR
-KA 40 A - DR - =KA, ④ D
KA D w")
Thus we have
wx (x( (T)
w² KA D w*)
(14)
A.3.3 All together
Th (we) w² The (w;) (w:)
wa(wi) + ww(w)) w²
= (I - nAR) T (wee - (x(we)) + KA, - Dm u - -w") using 13 and 14
(T AA,) "we w² (wx( w²)) )} + k AA 1T DE (w) w²)
(T Ax,, [Tw ==" -.--KA -KA, Φ5 D [t0e- D
w )) + AA T DV w²), Using 14 again
= [1 - -)A))) (T(( -AA- DΦ) + KA T D& (we - -
A.3.4 Studying I - A
For this, let us study the eigenvalues of Ax, and make sure they have a positive real part. Then by choosing 17 Re( ( the matrix will be stable.
For this, we will need, an assumption, i.c that TD(0) converges for that combination of $, D, P and y. Assumption A.3. Sp(TT DΦ - D)
C+
Let us call 6x minAeSp(A..) Re(A)
0t Ax = T D$ - DP D we have co > 0 and by continuity of the spectrum, 38 that Va 8, 6x 0. Thus we just showed that for small enough the matrix A, was also stable.
As o
A.3.5 Studying xA=2 D
Here we just need to show that for k small enough this matrix will be stable, ie have a spectral radius bounded by 1. It is sufficient to notice that A_ $5 D& has a finite limit when к + O, which is equal to Ao T D& which is indeed bounded as Ag = & DΦ x DP has non-zero eigenvalues as they are C C+ as per the Assumption above.
Thus, for lime-so -A a DΦ 0 which has a spectral radius 0. Then again, by continuity of the spectrum thus spectral radius, for small enough, -A-1 D& has spectral radius < 1.
Corollary A.4 (Stability of TD-FR and TD-TN). When p 1, for T large enough, the convergence domain of TD-TN is included in the one of FR-TN TN.
A.4 Proof for Corollary A.4
We first need to show that Sp( DΦ - - DD D)* C C+ implies that p(( - <
Let us consider a eigenvector of (oT D DP associated with the eigenvalue X. Thus
t DP Φv X DTv
21

---

Published in Transactions on Machine Learning Research (09/2023)
From this multiply by and substract and add o DΦv:
(D- DΦ - - D - DΦv
Multiplying by v
v D ( X DΦv
D&v
As & T DΦ is positive definite (non-singular by assumption), v "T DΦv > O. However we can't conclude anything in the general case with non-symmetric matrices. When 1, these matrix products become real scalars as all the coefficients are real. So for x² = DΦ
( - T DP= = -1
(D D implies that \7) > 0, thus X < 1 the maximum and only eigenvalue, thus the spectral radius of (Φ DΦ) - DP + is smaller than 7.
and DrN = {Φ, - }} the convergence domains of TD and TN (T +00). We just proved for p that DIN C DID We know from Proposition A.2 and 3.2 that for k small enough, FR(T) and TN(T) (FR and TN using period update of T) and DFR(T): DTN(T) their convergence domains that lim DFR(T) DTD and limr-++e DTN(T) DIN. Thus by taking the limit we have the inclusion we needed.
We denote by DTD = {Φ, DISp(1 + T DX - DD C C+}
<
22

---

Published in Transactions on Machine Learning Research (09/2023)
B
Time comparison
Execution Time Comparison
Time (second) for 1000 gradient updates
1.8
1.6
1.4
1.2
1.0
0.8
TN
FR
Algorithm
Figure 6: Time comparison of the gradient updates between FR and TN. We note that in training scenario large portion of the time is spent on inference and simulation which make the difference in gradient update time negligible.
C
Additional Results and Hyper-parameters
C.1
Four Rooms
Table Four Rooms
Hyperparameter
Value
learning rate
le-4
optimizer discount. factor
adam
Kingma & Ba 2014
0.99
DNN layers
[128, 128, 4]
dimension
x 11
23

---

Published in Transactions on Machine Learning Research (09/2023)
C.2
2-state MDP
We see Appendix that increasing the discount lactor leads to larger divergence regions. However; the circle of radius (30) which corresponds to the on-policy algorithm stays in the convergent domain as predicted by thcory.
1,000
viss)
m(sp]
(a) Spectral radius for FR
(b) TN
(c) FR
Figure 7: 2-state MDP. 7 0.9995
n(sp)
m[sa)
-05992
0.5992
(a) Spectral radius for FR
(b) TN
(c) FR
Figure 2-state MDP. 7 = 0.995
1.0000
D.0003
A[Se)
7(5p)
((sg]
003995
0.5994
Spectral radius for FR
(b) TN
(c) FR
Figure 9: 2-state MDP. 7 0.95
24

---

Published in Transactions on Machine Learning Research (09/2023)
C.3 Atari Suite
25

---

Published in Transactions on Machine Learning Research (09/2023)
Figure 10: Atari suite learning curves.
26

---

Published in Transactions on Machine Learning Research (09/2023)
C.4 Atari Sensitivity Analysis
BreakouthooFrameskipvv
Exploration 0..01
Exploration g=0.1
Exploration
300
250
300
200
200
200
150
100
100
100
50
50
DQL
TN
FR - 0.5 1.0
FR =
500
400
6686 Oiz. Junsoe
200
Retum
300
150
200
200
100
100
50
5
25 50 100 500
0.5
25 50 250 500
0.5
10
100 250
Network Update Period 1000)
Network Update Period (x 1000)
Network Update Period 1000)
Figure 11: Breakout results
27

---

Published in Transactions on Machine Learning Research (09/2023)
BeamRidrrNofraeesiipva
Exploration
Exploration ==0.1
Exploration c=0.25
12000
7000
10000
8000
6000
Daczcr Sanaaa P-0.90
5000
4000
6000
4000
4000
2000
2000
DQL
FR a 0.5 1.0
6000
5000
5000
4000
6666 mere Seouun
4000
3000
3000
3000
2000
2000
1000
1000
0.5
5
25 100 200
0.5 5 25 100 500
0.5
10 50 250 500
Network Update Period T [x 1000)
Network Update Period 1000)
Network Update Period 1000)
Figure 12: Beam Rider results
QbertNoFrameskip-v4
Exploration
Exploration 6=0.1
Exploration
14000
12000
6000
5000
10000
Discount y=0.99
10000
4000
8000
6000
6000
3000
4000
2000
2000
2000
1000
DQL
FR 0.5
x= 1.0
6000
#.5
8000
6000
5000
5000
6666'0 Or and Amooee
6000
4000
3000
4000
2000
2000
2000
1000
1000
0 0.5
5 10 25 50 100250 500
0.5 5 10 25 50 100250500
0 0.5
10 25 50 100 250500
Network Update Period (x 1000)
Netwerk Update Period 1000)
Network Update Perfod ( 1000)
Figure 13: Qbert results
28

---

Published in Transactions on Machine Learning Research (09/2023)
EnduroNoframeskipv4
explooation
1600
1000
1400
1200
1200
Reccont y- 0.99
1000
Return
800
600
800
600
400
400
400
200
200
200
DQL
FR a 0.5 1.0
1750
1400
=
1500
1200
1250
Discount y- 9999
1000
800
600
400
200
250
200
10 25
100 500
0.5 10 100 250
0.5
5
25. 50 250
Netwark Update Period (x 1000)
Network Update Peried 1000)
Network Update Peried 1000)
Figure 14: Enduro results
SeaquestNoFramessippv
Exploration
Exploration
Exploration
12000
7000
10000
6000
bixcount y-0
6000
0000
4000
6000
3000
2000
2000
1000
DQL
FR 0.5
1.0 2.5
7000
8000
8000
5000
6000
6000
3000
4000
2000
0.5
5
10 25
100 250
0.5
5 25 50 100250500
0.5
10
50 100 250 500
Network Update Period T (x 1000)
Network Update Period 1000)
Network Update Period 1000)
Figure 15: Seaquest results
29

---

Published in Transactions on Machine Learning Research (09/202))
SpacelnaaeessoFrrmsskiiwww
Exploration
Exploration
Exploration
1750
1750
1400
1500
1500
1200
Secoumt D.9999
1250
1250
1000
Reter
1000
1000
500
DQL
250
250
TN
200
FR = 0.5
2000
1600
FR 1.0
1750
1400
1400
FR x.5
1500
1200
6666 A Junazuia
1000
800
000
600
500
400
250
200
25
100 200
0.5
10
250 500
0.5
5
50
250
Netwark Update Period (x 1000)
Network Update Peried 1000)
Network Update Period
Figure 16: Spacelnvaders results
30


Reviews:
[{"review_id": "EXgTg5HikQ", "reviewer": "Reviewer_GKoB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The theoretical results appear sound and depend on sensible assumptions. \n2) The paper is exceptionally clear. The motivation is sound, interesting to a broad audience, and timely. \n3) The empirical results in Fig 1 are the best that I've seen motivating the increased potential instability from using target networks, while also clearly highlighting the loss of sample efficiency. Also these are quite beautiful plots!\n4) The sensitivity curves in Fig 3 clearly highlight that the proposed algorithm dominates target networks for a very wide range of configurations of the proposed regularizer. Results are clearly statistically significant with 40 samples per condition.\n5) The proposed modification is simple to implement and easy to understand. The only added cost is an additional forward pass through the NN during learning, which is negligible compared to the cost of backprop. \n\n\nWeaknesses:\n\n1) There is an unfortunate limitation that theoretical results are exclusive to the linear setting, while target networks are almost exclusively used in nonlinear settings. This deviation nullifies one of the major motivations of the proposed algorithm.\n2) The paper would benefit from a richer discussion of placement in the literature. Specifically, there is no mention of gradient TD methods, and these share nontrivial similarities to the proposed approach.\n3) There seems to be a subtle contradiction in motivation and the proposed algorithm. Much of the motivation is around stable learning and the proposed modification to target networks is to avoid instability due to the matrix $\\Phi DP \\Phi$. However, the proposed algorithm is only provably stable under the same conditions as TD, where we've already placed strong conditions on the highly related matrix $\\Phi D (I - P) \\Phi$.\n4) The additional hyperparameters are implied to be a strength---i.e. increasing configurability of the method. I tend to view additional problem-dependent configuration parameters as a weakness, especially when it appears there is a natural dependency on observable problem information (i.e. $\\gamma$). Given that the paper suggests $\\kappa = 1$ is generally a good choice, which is approximately $\\kappa = \\gamma$, it seems we could eliminate a configuration parameter by making _fewer_ deviations from target networks.\n5) The paper claims superiority on Atari, but at best the experiments provide evidence of equivalency. Because the proposed method has an additional hyperparameter, and because no single setting of that hyperparameter performs consistently well (looking in appendix B.3) it seems any marginal superiority per-game is due to additional degrees of freedom and loose confidence intervals. These claims should be toned down, or the evidence ramped up (e.g. by tightening confidence intervals which is likely outside of budget, or reducing configuration degrees of freedom).\n\n\nMinor nitpicks:\n- Eqn 3 really isn't the gradient of any loss function. Writing it as such on the left side of the equals sign isn't quite correct, and really isn't necessary to what is being communicated.\n- At the top of page 3, it is stated that linear functions provide a poor approximation to $Q$ in complex environments. This isn't really true---consider that a neural network can be decomposed into a feature-generating function up to the penultimate layer and a linear function of those features in the last layer. If someone handed me that feature-generating function ahead of time, I could learn a linear approximation of equivalent quality as the neural network. Consider also Machado et al. 2018 that showed Atari can be easily solved with linear function approximation of the right features. Really DNNs are useful when high-quality features are not readily available and must be learned, and has little to do with environment complexity.\n- I'm not sure Fig 2 provides any novel insight, and the related paragraph in the middle of page 7 might be overclaiming its novelty / contribution. It's been quite well-established that $\\gamma$ plays a strong role in the divergence of TD (e.g. Kolter 2011, Scherrer 2010, White 2017)\n\n-----\n\nFurther details:\n\nWeakness 1 - The deviation between linear and nonlinear settings does significantly limit the insights we can draw. The dependency on linear function approximation is also nontrivial and it is unclear whether any of the insights hold beyond the linear case. One of the major contributions of the proposed algorithm is to modify the matrix $\\Phi DP \\Phi$ to become $\\Phi D \\Phi$ in order to have a convex loss. In the linear case, this is totally sensible. In the nonlinear case, target networks change both the mapping from features to value function (i.e. $w$) _and_ the feature matrix (i.e. $\\Phi$). \n\nIf we look at the learning system at a given timestep, freezing the neural network and target network weights, then we can produce a \"feature matrix\" by feeding each state into the respective neural networks and storing the outputs of the penultimate layer. The neural network would produce some matrix $\\Phi$ which is $|S| \\times d$ and the target network would produce some $\\Phi_{tn}$ of same dimensions. Now our regularizer term under target networks has matrix $\\Phi DP \\Phi_{tn}$. Removing the stochastic matrix $P$ is not enough to ensure convexity---i.e. $\\Phi D \\Phi_{tn}$ may still have negative eigenvalues.\n\nMy current perspective is that this largely inhibits us from transferring intuitions from the linear -> nonlinear case.\n\n---\n\nW2 - Literature placement. \n\nThe second paragraph of the introduction seems to suggest that the study of stable value function learning started in 2013 with the DQN paper and target networks. Really this has been a focus of study since the early 90s with a rich literature of proposed strategies.\n\nOne notable line of work is the gradient TD methods starting from  Antos et al. 2008 and Sutton et al. 2009. This seems particularly relevant because the regularizer in eqn 5 of this submission looks _quite_ similar to the TDC method from Sutton et al. 2009. Both are a correction of the standard semi-gradient TD update rule, both down-weighted by gamma, both dependent on the value function at the next state (i.e. $P \\Phi w$), both using some lagged secondary weight vector. A few recent papers have proposed gradient TD methods as a strategy to remove the need for target networks in nonlinear learning.\n\nGradient TD methods also provide a solution to two of the open challenges in this work. GTD methods are provably convergent (even in the nonlinear setting) under broad conditions and even when TD is not convergent. Also TDC, specifically, is adaptive in the degree that it depends on its correction term. For functional regularization, $\\kappa$ must be selected ahead of time through tuning. TDC uses an estimate of the TD-error to control the degree of regularization. When the value function is nearly accurate, we no longer need as much \"correction\" so the algorithm reverts to TD(0). When the value function is highly inaccurate, TDC more heavily leverages the regularization.\n\nI certainly don't mean to suggest GTD methods subsume the proposed method here, as the perspectives are quite different and the resultant algorithm has some minor differences. But I do believe this submission is incomplete without a discussion on the relationship between these.\n\n---\n\nW3 - Stability.\n\nThis paper does a great job of showing that target networks can sometimes introduce novel unstable regions to the optimization process. However, it also has been shown that target networks can introduce novel _stable_ regions (e.g. off-policy TD(0) with target networks on Baird's counterexample converges). From my understanding, the proposed functional regularization does not change the stability region from standard TD(0) while continuing to pay the sample efficiency cost of target networks---and perhaps a worse sample efficiency cost according to Fig 1. \n\nThese theoretical and empirical findings in the linear case do an exceptional job of convincing me that target networks are not a valid solution for stabilizing TD learning. However, they do not convince me that functional regularization is a suitable replacement, in all cases the standard TD(0) seems strictly superior. The empirical findings in the nonlinear case are more motivating, but also much harder to understand and limited to a very \"nice\" environment: FourRooms. The empirical findings on Atari appear insignificant and are very likely due to the proposed method having additional degrees of freedom.\n\nUltimately, I strongly suspect something interesting is happening with functional regularization here and that it provides something beyond TD(0), but I'm not sure the linear FA results get to the heart of what that is. It doesn't appear to be stability, as suggested.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a perspective of viewing target networks as a form of regularization for value function learning with bootstrapping. Through this perspective, the paper shows that target networks can introduce new regions of instability for TD learning methods---counter to conventional wisdom that target networks assist with stability. The paper highlights this instability is due to nonconvexity of the regularization problem. A simple modification to target networks allows for a (sequence of) convex loss(es) which the paper shows adds no novel instability regions---though also does not improve stability. These findings are shown both theoretically and empirically in the off-policy linear function approximation case in expectation. The paper concludes with an empirical study with nonlinear function approximation on a small set of Atari games and a wide range of hyperparameter settings for the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Discuss the relationship between functional regularization and gradient TD --- at the very least discussing that both share the same goals, but preferably also the significant overlap between TDC and FR.\n2) Concretely discuss how FR relates to and improves stability. The results (both theoretical and empirical) in the linear FA section don't seem to support this claim.\n3) Consider a strategy to reduce hyperparameter sensitivity of $\\kappa$---the atari results required a sweep over a wide range of this hyperparameter and the hyperparameter seems strongly problem-dependent. Is it possible that defining a new hyperparameter $\\eta$ such that $\\kappa = \\eta \\gamma$ is much less sensitive?\n4) Align the Atari claims with the evidence---either tone down the claims to say that FR is at least as good as TN or ramp up the evidence to suggest FR is truly superior. As an aside, the paper does not _need_ superiority claims to be impactful so toning down the claims seems the easiest path to me! ", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4chfggnK4z", "reviewer": "Reviewer_uvyZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] This paper analyzes the Target Network's implicit regularization from a theoretical point of view. The analysis is enlightening and reveals the flaws of the Target Network method based on theoretical analysis. The authors utilize the results of theoretical analysis to propose optimization goals and design a new algorithm.\n[+] The introduced Functional Regularization method exhibits flexibility, allowing for improved performance by decoupling the regularization parameter κ.\n[+] The presentation is clear and easy to follow. The experiment results are well discussed and the figures, such as Figure 1, are well organized and informative.\n[-] The logic of the paper is: TN method can be interpreted as a kind of implicit regularization, but this implicit regularization has its own problems, so the authors propose a more direct regularization method. My doubts lie in: 1. The authors have not been able to theoretically explain why such regularization would be helpful for TD learning, so the logical argument that their method benefits TD learning through better regularization is questionable. 2. Perhaps the authors could observe the regularization effect in experiments, especially in general benchmark experiments through some ablation studies, to match the article's logic, rather than just directly observing the final experimental results. Because even if the experimental results improve, the logic in the previous text (FR improves TD learning through better regularization) cannot be verified.\n[-] While the theoretical part of the paper is limited to a very simple setting, which is linear function approximation with finite state and action spaces, and no exploration effect is considered, the effectiveness of the proposed method should be evaluated carefully using experimental methods. However, only two simple environments show that it outperforms the TN method, while in Atari games, it is not significantly better than the baseline method. Also, the Atari experiments do not seem to have converged for all algorithms.\n[?] In Four Rooms and Atari games experiments, all returns are very wild since they all drop at the end, even worse than the initial return. Can you explain why?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the implicit regularization role of Target Networks in deep Reinforcement Learning and points out that Target Networks may lead to instabilities and inflexibility. To address these issues, the authors introduce a Functional Regularization method that directly regularizes the value network towards a network parameterized by lagging parameters. By conducting extensive ablation studies on Target Networks and Functional Regularization across various settings, the paper demonstrates that decoupling the regularization parameter κ results in more flexible behavior and improved performance. The paper conducts several experiments to verify the effectiveness of the proposed Functional Regularization method and compares it with the traditional Target Network method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses.", "broader_impact_concerns": "None at this moment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OyDIU74RjW", "reviewer": "Reviewer_pCv4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is very well written. \n\n+ The plots are clear and visually pleasing. \n\n+ The paper has potentially high impact and is of interest to the community. I agree with the authors that FR is both simple, and also makes more sense than the currently used TN. I see no reason it shouldn't replace TNs in the common packages (say with kappa=1).\n\n- The limitations mentioned by the authors themselves. I think the biggest issue for me is the extra parameter kappa, as it feels like some theoretical connection between kappa, gamma and T should exist. \n\n- Several works were mentioned in the related work section, but I feel like the comparison to the closer 1 or 2 works should've been included somehow in the paper (in the experiments, or the theory). Otherwise the work feels a bit secluded.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyse TD(0) with a target network and show that it can in some cases cause divergence. \n\nThey propose an alternative solution that solves this issue using function regularization, and perform several experiments to show its performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All of the following would strengthen the work, and are not critical for securing my recommendation.\n\n1. If possible, please add to the supplementary the convergence plots of each Atari game (rewards of roll-out episodes) for the best set of parameters. It usually gives better perspective on which algorithm is better.\n\n2. If you manage to find a theoretical connection between kappa, gamma and T it would largely contribute to the paper.\n\n3. I would mention some of the related works along the paper in relevant locations. \n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tBj4n9xcYb", "reviewer": "Reviewer_nz5F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n**Novelty**\n- novel theoretical results\n- new analysis on classic environments \n- new results for Atari showing the FR can perform as well as TN, and in some cases stronger. \n\n**Quality**\n- Overall the presentation is good, the analysis on the toy examples are especially interesting.\n- figures and results are presented and explained well, highlight in alg 1 is great. \n- empirical study spans a wide range of FR hyperparameter and environment settings\n\n**Clarity**\n- the paper is written in a clear way, there are many interesting results, but is overall easy to follow\n\n**Significance**\n- the theoretical results plus the classic environment analysis can make some significant contribution \n- empirical results on Atari are not very strong, but also interesting \n\nWeaknesses:\n- Additional technical details: Regarding computation overhead, how is FR compared to using TN? I guess it is not too big but would like to know the exact number. (sth like a computation time table can be good)\n- some typos but can be easily fixed\n- the results on Atari are over a limited number of environments\n- on Atari, seems to me there are some performance gain, however, this is when looking at a number of different FR settings, and some of them do better, while others might do worse in cases. (I guess FR brings in more flexibility with the extra hyperparameter)\n- Atari experiments are compared to DQN which is an older baseline and not with SOTA methods. (though the main goal of the paper is not to improve over SOTA)\n\nAn overall solid paper with some nice contributions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors show that target network (TN) in DRL serves as an implicit regularizer that can have benefits but also has its own issues. The authors then propose an explicit functional regularization (FR) with theoretical support for its convergence. The authors compared TN and FR on classic toy example environments, and provided some interesting analysis. Further empirical results are provided for tasks in Atari, showing that FR can do better in terms of performance and prevent divergence. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggestions: \n\n- please more explicitly report computation overhead and compare to TN. \n- more experiments over more tasks (maybe just pick the best FR setting and apply to other environments) can be interesting, though what are in the paper now already make some interesting contribution. \n\nTypos: please go through the paper and fix typos, including: \n- end of page 2 \"We denote the the off-policy sampling\"\n- page 6 \"disk or radius 1\"\n- page 7 \"Figure 2 where can see\"\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests modification to a widely-used algorithm, supported by empirical and intuitive justifications that are likely to capture community interest. The raised concerns by the reviewers have been addressed in the rebuttal, and the paper contributes insights toward enhancing stable reinforcement learning. The revisions adequately address the initial concerns expressed in the reviews.

I encourage the authors to address any minor issues before submitting their camera ready version.
Decision: Accept (poster)