Deep Q-Network with Proximal Iteration
Authors: Kavosh Asadi
Rasool Fakoor
Omer Gottesman
Michael Littman
Alex Smola
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
deep learning

Abstract: We employ Proximal Iteration for value-function optimization in reinforcement learning. Proximal Iteration is a computationally efficient technique that enables us to bias the optimization procedure towards more desirable solutions. As a concrete application of Proximal Iteration in deep reinforcement learning, we endow the objective function of the Deep Q-Network (DQN) agent with a proximal term to ensure that the online-network component of DQN remains in the vicinity of the target network. The resultant agent, which we call DQN with Proximal Iteration, or DQNPro, exhibits significant improvements over the original DQN on the Atari benchmark. Our results accentuate the power of employing sound optimization techniques for deep reinforcement learning.

Under review as a conference paper at ICLR 2022
DEEP Q-NETWORK WITH PROXIMAL ITERATION
Anonymous authors Paper under double-bling review
ABSTRACT
We employ Proximal Iteration for value- function optimization in reinforcement learning. Proximal Iteration is a computationally efficient technique that enables to bias the optimization procedure towards more desirable solutions. As con- crete application of Proximal Iteration in deep reinforcement learning, we endow the objective function of the Deep Q-Network (DQN) agent with a proximal term to ensure that the online- netwoe component of DQN remains in the vicinity of the target network. The resultant agent, which we call DQN with Proximal Itera- tion, or DQNPro, exhibits significant improvements over the original DQN the Atari benchmark. Our results accentuate the power employing sound optimiza- tion techniques for deep reinforcement learning.
INTRODUCTION
A core competency of a reinforcement learning agent is the ability to learn in environments with large state spaces, which is key to solving important applications such robotics (Kober et al. 2013). dialog systems (Williams et al.] (017). and large-scale games (Tesauro) 1994; Silver et al, 2017 Recent breakthroughs in deep reinforcement learning have demonstrated that existing algo- rithms such as Q-learning (Watkins and Dayan 1992] Sutton and Barto (2018) can surpass human- level performance on large-scale problems when they are equipped with deep neural networks for function approximation (Mnih et al. 2015).
Two key components of gradient- based deep reinforcement learning agent are its objective func- tion, and an optimization procedure. 'Tee optimization procedure takes estimates of the gradient of the objective with respect to network parameters, and updates the network parameters accordingly. In DQN (Mnih et al. 2015), for example, the objective function is the empirical expectation of the one-step temporal difference (TD) error (Sutton 1988) on a buffered set of environmental interac- tions (Ln 1992). and advanced variants of stochastic gradient descent are employed to update the network to best minimize this objective function.
A fundamental difficulty in minimizing this objective function stems from operating in what is known as the deadly triad (Tsitsiklis and van Roy 1997] /Sttton and Barto 2018] van Hasselt et al.) 2018): Updates are performed (1) off-policy and in conjunction with (2) non-linear function approx- imation and (3) bootstrapping. In the context of reinforcement learning, bootstrapping refers to the dependence of the target of our gradient updates on the parameters of the neural network, which is itself continuously updated during training. Employing bootstrapping in reinforcement learn- techniques, or even Monte-Carlo reinforcement learning (Sutton and Barto 2018), where the target of our gradient updates does not directly depend on the parameters the neural network.
ing stands in contrast typical supervised-learnin
Mnih et al. (2015) proposed a simple and intriguing approach to hedging against issues that arise when operating in the deadly triad, specifically to use a target network for bootstrapping in value- function optimization. In this case, the target network is updated periodically, and tracks the online network with some delay. While this modification constituted a major step towards combating mis- behavior in Q-learning (Lee and He 2019 Kim et al. 2019: Zhang et al. 2021), extreme volatility in the optimization procedure is still prevalent. To ease optimization, we can synchronize the target and the online network less frequently. but doing could be thought of as delaying policy improvement, therefore making the ultimate performance sensitive to the period of synchronization selected (Kim) et al. 2019).

---

Under review as a conference paper at ICLR 2022
Our contribution is to employ Proximal Iteration to improve the stability of the optimization proce- dure in DQN. In particular, by observing a synergy between DQN and Proximal Iteration, we endow the objective function in DQN with proximal term that ensures the parameters of the online- network component of DQN remain the vicinity of the parameters of the target network. This novel combination adds additional computational cost to the original DQN agent. We present comprehensive experiments on Atari benchmarks (Bellemare et al. 2013) demonstrating that our new algorithm, which we refer to Deep Q-Network with Proximal Iteration or DQNPro, yields significant improvements over its vanilla DQN counterpart, thus revealing the benefits of combining deep RL algorithms with Proximal Iteration.
BACKGROUND ON REINFORCEMENT LEARNING
Reinforcement learning (RL) is the study of the interaction between an environment and an artificial agent that learns to maximize reward through experience. The Markov decision process (Puter- man 1994), or MDP, used to mathematically define the RI problem. An MDP specified by the tuple (S, A, R, P, 7), where S is the set of states and A is the set of actions. The functions R x A R and P S x A + [0, 1] denote the reward and transition dynamics of the MDP. Finally, by discounting future rewards, 7 formalizes the intuitive notion that short-term re- wards are more valuable than those received later.
The goal in the RL problem to learn a policy, a mapping from states to a probability distribution over actions, π S + P(A), that obtains high sums of future discounted rewards. An important concept in RI. the state-action value function (Q-function). Formally, it denotes the expected discounted sum of future rewards when taking particular action in state and following policy thereafter:
(=ss.a)=EE[A + yB442 + 2 ... [S, =, s. a.
Define Q' as the optimal value of a state-action pair: Q" a) = max, Q" a). The Bellman optimality operator T: R Al - RiS|iAA| is defined as follows:
[T()) a) = a)
Σ EESS
P(s, a, max a(s',a).
Q"(s, a) is the unique fixed-point of T, meaning Q" := T(Q"). The Bellman operator is at the heart of many planning and RL algorithms such as Value Iteration (Bellman 1957). which proceeds as follows given an arbitrarily initialized Qo:
Q++1 + T(Qx).
This iterative process is guaranteed to converge to the unique fixed point of T, namely Q", because is y-contraction in the infinity norm:
(T(Q) I( <11 -
In large-scale settings, it is tractable learn separate number for each state-action pair. common way to address this curse of dimensionality is to learn representation of the Q function in the form of parameterized function approximator, such as deep neural network: Q(s, a; w) s Q*(s, a), where w here denotes the parameters of the function approximator, No- tice that the Bellman operator may no longer be contraction in the presence of general function approximation.
3 THEORY
Define the Bregman divergence generated by the convex function F as follows:
Dr(2, v) F(z) F(v) VF(y)
(a y)
where we assume that F is smooth in the sense that the gradient of F non-expansive:
FF - VF(1)) || vll
2

---

Under review as a conference paper at ICLR 2022
We define the Bergman Bergman-smoothed smoothed Bellman operator, TD follows:
TD(e) arg min 1 プ [T()) e¹11
Dr('',u).
1
2
2c
Define = Tb(u). we have:
8 == T + (VF())-VF(0) VF(T)).
2c
We can then show:
(ITr(e)) (b2))l = ||0 u2ll
1
[[((e)) + -VF()))VF(() T(u)) (VF(02) -
2c
2c
1
s [[(ee)) - (22))1
(F(()) VF(22)]1 + - [VF(6;) VF
20
2c
1
s [[((1)) - T(2))ll
[VF(2)) - VF(D2)] +
|0 022
2c
2c
This implies:
2c - 1
1
[Tb(vi) Tn
<
[T((i) T(v2)l1 +
[VF(i)) vF(02)
2c
2c
1
[[]() - T(22)l +
-
|0| U22l
26
207 > - 22ll +
[1] - U2
2c-
2c
207 1
||0| - v2lb
2c
allowing us conclude that:
< 7+ 2c
[[T((1)) - (22)))
2c |10 =22l
Therefore, the operator Tn is +2 2c-1 contraction long 1-7
4 PROXIMAL OPERATOR
Consider the following optimization problem:
minimize, ((::w).
where h is the objective function (e.g., standard temporal difference (TD) error in reinforcement learning (5), w denotes the learnable parameters (e.g.. the parameters of Q-network), and x the input data (e.g., (s, a, s')). If is differentiable, then stochastic gradient descent (SGD), which proceeds as follows:
+ # ag V (=) k = 1,2,3....
(2)
could be used to solve the optimization problem ①I. Here, ag denotes the step size at iteration k, and we is an arbitrary starting point
While SGD is well-behaved asymptotically and under standard assumptions, it can perform reck- lessly in the interim, leading into a situation where the iterates get exponentially far from the solu- tion (Moulines and Bach 2011: Ryu and Boyd 2014). This behavior can be particularly disruptive in the context of RL where sample efficiency is an important concern.
One simple and effective approach to stabilizing SGD is to augment the objective with quadratic regularizer to ensure that the solution at each iteration stays in the vicinity of the previous ones:
1 ||w wx-1l2
Wi = arg min (w) +
(3)
10
2c
'To simplify the notation and whenever it is clear from the context, we drop from the notation, i.e., h(w) A h(x; and Vh(wa-1) Vh(x; wk-1))
3

---

Under review as a conference paper at ICLR 2022
The quadratic term is basically proximal term that ensures the next iterate wy will not stray too far from the previous iterate Wk-1- Having completed iteration k, we replace the previous iterate Wk-1 by its new value we and then proceed to iteration lt + 1.
To understand the behavior of this iterative process, called Proximal Iteration (Parikh and Boyd 2014 Ryu and Boyd 2014), define the proximal operator:
1
prox,, (wx) arg min h(w) + 1| Wk
2c
Proximal Iteration could be written as:
prox (wx-1) k= 1,2,3...
From the Banach fixed- sred-ppiinteee theorem, we know that iteratively upplying prox, eventually converges to unique fixed point provided that the operator is contraction. Bauschke et al. (0122) have shown that, if the function h p-strongly convex, then the operator prox,, indeed contraction:
| (w) - prox, (w))l - + | - w/ll
and therefore iteratively applying the operator would guarantee convergence.
What is the fixed-poin of this operator? It is possible to show that it is the minimum of the function h, i.e., arg min,, h(w), the minimum of problem which we originally sought to solve (Bauschke) and Combettes) 2011
Therefore, the intuitive way of thinking about the Proximal Iteration algorithm, which is sometimes referred to as disappearing Tikhonov regularization (Parikh and Boyd (014)) is as principled way to bias the optimization path towards the previous iterate using the quadratic-norm penalty without altering the final answer. In other words, as Proximal Iteration makes progress, the previous iterate w gets closer to w" = arg min h(w) in light of the contraction property, therefore the contribution of the quadratic term vanishes zero thus not biasing the asymptotic solution. The following remark sheds more light on the relationship between Proximal Iteration and trust-region algorithms:
Remark (Relationship between trust region and Proximal Iteration). Define the trust-region
problem as follows:
minimize, (w)
s.1. -ooe-wk-lly Sp
Parikh and Boyd (2014) show that solving the above trust-region formulation is equivalent to step of Proximal Iteration under appropriate choices of p (above) and (in prox). Therefore, iteratively applying the two formulations leads to the same solution. (See their Section 3.4 for more details.)
The second remark highlights the strength of Proximal Iteration relative to SGD:
Remark 2 (The non-asymptotic advantage of Proximal Iteration over SGD). While the asymp- totic performance of SGD and Proximal Iteration are analogous (Ryu and Boyd 2014), they may exhibit qualitatively different behavior in non-asymptotic sense. More concretely, denote the step- size of SGD and Proximal Iteration at iteration le by Cr k and w² = arg min h(w). For SGD, have (Bauschke and Combettes 2017):
expl exp(Ca
E [w w 111] 5
RC
k
The exponential dependence C can indeed manifest itself as Ryu and Boyd (2014) demonstrate. In contrast, for Proximal Iteration we have (Ryu and Boyd (0014)
E [| w² w1] 11 w0 Σa. 0.
While the iterates of SGD can get exponentially far from the optimal answer (Moulines and Bach 2011 Ryu and Boyd 2014), Proximal Iteration is fairly well-behaved in the interim. This property is particularly valuable the context of reinforcement learning where -fficeeeff and not just the asymptotic performance, of utmost importance.
4

---

Under review as a conference paper at ICLR 2022
In the context of parameterized Q-function in RL, commonly used loss functions not convex; however, several works have introduced convex loss via linear approximation. Lee and Hc (2019) have shown that in linear policy evaluation the loss function is convex in the weights. Serrano et al. (22T)) introduced a convex loss function for control with linear approximation. These works demonstrate that many useful convex minimization objectives exist for RI..
In the case of deep RL, the loss function is not convex, and few convergence results exist for most state-of-the-art algorithms. Nevertheless, follow standard practice and show that Proximal It- eration achieves significant empirical improvements when applied to standard deep RL algorithms. Furthermore, recent work provides convergence results for Proximal Iteration in specific non-convex cases, such as when h(w) can be decomposed into sum of a smooth non-convex function and non- smooth convex function (Fukushima and Mine, 1981| J Reddi al. 2016 Li and Li 2018). These recent results open the door for potential future developments toward proving convergence for Prox- imal Iterations in RI., which would further strengthen its theoretical grounding.
5 DEEP Q-NETWORK WITH PROXIMAL ITERATION
In this section, we introduce a new deep RL algorithm by applying Proximal Iteration to DQN. Recall that the original DQN agent employs two parameterized value functions: online network (s,, w) that is updated at each step, and target network ((s, a; 0) that is used for bootstrapping and synchronized with the online network periodically. More formally. let tuples (s, a, r, s') denote the buffered environmental interactions of the RI. agent. Define the following objective function akin DQN:
h(w)= == (1/2)E [Ir y max ((s',d':0) - Q(s,a;
(5)
a'
Given a fixed target-network weight 0, in the original DQN, our desire is to find a setting of weights that minimizes h. We can do so by applying SGD, but as argued above, it can lead to reckless updates of w. To hedge against this possibility, we can bias the online weights towards the previous iterate. The target network a natural choice for this purpose, motivating the following objective:
1 2c
arg min hww)
||0
01 prox,, (8).
To solve this minimization problem, take multiple descent steps using the stochastic gradi- ents of the objective. Thus, starting from the initial point w 0, we perform multiple w updates (specified by the period hyper-parameter) as follows:
for = 1... period:
+ w a (Vh(w) + (w 0))
(6)
0 - w.
In the last step, akin to the original DQN, we synchronize the two networks by performing + w. before moving the following iteration. This iterative optimization proceeds until convergence is obtained.
Observe that the
update 6 can equivalently be written as: (1 - (a/c)) + (a/c) 0 aVh(w).
Notice the intuitively appealing form of the update: We first compute a convex combination of 0 and w, based on the hyper-parameters a and c, then add the gradient term to arrive at the next iterate of w. If w and 0 are close, the convex combination is close to w itself and so this DQNPro update would in effect perform an update similar to that of the original DQN. However, when the online weight strays too far from the previous nurget-network iterate, taking the convex combination ensures that the online network gravitates towards the target network by default. other words, the gradient signal from minimizing the squared TD error 5 needs to be strong enough to compensate
5

---

Under review as a conference paper at ICLR 2022
Algorithm Deep Q-Network with Proximal Iteration (DQNPro)
1: Initialize 0, N, period, replay buffer D, a, and c
2: 8 f- env.reset),, w + 0, numUpdates +- 0
3: repeat
4: a a~ e-gerddyyy((( (Q(s,-;
5: s',r env.step(s, a)
6: add (s,a, (,,,,,,8'), to D
7:
if s' is terminal then env.reset()
8:
9:
end
10:
for in {I...., N} do
perform N updates
11:
sample a mini-batch B = {(8, r,,,,,))))))))) from D
Vh(w) V 2|
1
Σ
(r 7 max ('s','': Q(s,;
(B
w - (1 (a (a/e))
(a//c) 1 a aV vw))
numUpdates numUpdates +
12:
if numUpdates % period = ( then
update the target network
0 < w end if
14:
15:
end for
16:
until convergence
for the default gravitation towards 9. Notice also that the update rule includes the original DQN update as a special case when e + oc.
The pseudo-code for DQNPro is presented in Algorithm I The difference between DQN (Mnih al. (215)) and our DQNPro minimal, and highlighted in red. Note that our proposed method adds almost no additional computational cost to the original DQN. While the particular form of DQNPro presented in Algorithm [ uses SGD as an optimizer, we note that the general idea be- hind our method can be applied with other forms of first-order gradient-based optimizers such as ADAM (Kinmma and Ba 2015), ADAGRAD (Duchi et al., 2011), etc.
EXPERIMENTS
In this section, we first evaluate the DQNPro agent relative to the vanilla DQN counterpart on a set of Atari-2600 benchmarks (Bellemare al. 2013), and show that endowing the agent with the proximal term can lead to significant improvements in overall performance. We next investigate the utility of our proposed proximal term through ablation analyses and experiments. See also Appendix A for a complete description of our experimental pipeline, as well as Appendix B and Appendix for more results and analyses.
6.1 SETUP
We used 40 games from the Atari-2600 benchmark suite (Bellemare et al., 2013) to conduct our experimental evaluations. Following Machado et al. (2018) Castro et al. 2018) , our experiments used sticky actions to inject stochasticity into the otherwise deterministic Atari-2600 emulator
In our experiments, we endow DQN (Miih et al. 2013) and DDQN (van Hasselt et al. 2016) with
the proximal term and compare with the vanilla counterparts. Our training and evaluation protocols and the hyper-parament settings closely follow those of the Dopamine") (Castro et al. 2018 base- line, which is consistent with the existing literature. To report performance results, we measure the
A When sticky actions are enabled, the emulator will ignore agent's current action choice and execute agent's 's previous action with small prohability (Machado et al. 2018). https: / thub. com/ goog] e/ e/dopami ne

---

Under review as a conference paper at ICLR 2022
500%
500%
100%
100%
10%
10%
0%
0%
10%
-10%
100%
110%%
500%
500%
DemoroRaac
Ansuw
Camiral
Knuli
Yerscovengs
Robottank
Allen
Batteezine
Acantts
Cracyclinke
Tutanihhm
Obert
Riveraid
Bonkt-ert
Venture
froatate
Bettork
Kungtumaster
Monachad chan
Phopporcamanni
Ammder
Gopher
Pos jan
videapenbal
Axcoms
Boxing
Airrad
jamoztrond
Breakout
Stargurnay
Seaquest
Zasion
Enduro
Timeptoo
Spacealvedos
Kanyproo
Namethisg Numechissgama
Grevita
Worandofoor
Figure/ 1: Final performance gain for DQNPro over vanilla DQN. Results are averaged over four random seeds, and the y-axis scaled logarithmically.
undiscounted sum of rewards obtained by the learned policy during evaluation. We further report the learning curve for all experiments averaged across 4 random seeds in Appendix [C] We reiterate that we used the same network architecture as that of Mnih al. (2015) for all experiments in this paper and we also used the exact same hyper-parameters for all agents to ensure a sound and fair comparison. More details are provided in Appendix A
We emphasize that our DQNPro only adds a single hyper-parameter, namely c, to the original DQN. 0.2 throughout all experiments with DQNPro and in all games.
We set the hyper-parameter We did a minimal random search on four games, namely Asterix, Breakout, Qbert, and Seaquest, to tune this value. We also observed no need to tune e on a per-game basis, as 0.2 worked well consistently for all games and the two algorithms demonstrating that obtaining good performance does not hinge on per-game exhaustive search over c. We also observed that any value of c 2 0.1 resulted in improvements over vanilla DQN.
6.2 RESULTS
The most important question to answer is whether endowing the DQN agent with the proximal term can yield significant improvements over the original DQN. To answer this question in the affirmative, our first result is a large-scale comparison between DQN and DQNPro.
Figure T shows comparison between DQN and DQNPro in terms of the final performance. In particular, for each game, we compute the performance difference between the final policy learned by the DQN agent and the uniformly-random policy, i.c., (DQN random), as well (DQNPro random). Given these two values, we compute the relative improvement of the more performant agent over the weaker one in percentile. Bars shown in red indicate the games in which we observed better final performance for DQNPro relative to DQN, and bars in blue indicate the converse case. The height of a bar denotes the magnitude of this improvement for the corresponding benchmark; notice that the y-axis is scaled logarithmically.
clear that DQNPro dramatically improves upon DQN, and this improvement manifests itself in all but 6 games tested. This improvement is particularly rewarding in light of the fact that the implementation of the DQNPro agent is just minimally different than that of the DQN, and that DQNPro adds almost additional computational burden the original DQN agent. We also show learning curves on 4 Atari games (Asterix, Breakout, Gravitar, and SpaceInvaders) in Figure 2 which demonstrate that DQNPro typically yields better interim performance well. We defer to Figure S4 in the Appendix for full learning curves on all games tested.
Can combine DQNPro with some of the existing algorithmic improvements of DQN? One in- teresting idea in this space is to bootstrap based on Q(s, arg max, Q(8, a; w): 8) as opposed to max (s, a; 8) during value-funnce function optimization (van Hasselt et al. 2016). Referred to as Double DQN, DDQN, doing so hedge against over-estimation issues that arise in Oelarnngg in light of the convexity of the max operator (Thrun and Schwartz 1993). Recall that DQNPro encourages the online network to stay in the vicinity of the target network. As such, it would be interesting to
7

---

Under review as a conference paper at ICLR 2022
Asterix
Breakout
Gravitar
Spaceinvaders
6000
Auaraee Raturn
Averoos Return
8000
600
4000
100
Aweezoge
600
Awerage
1020
200
2 Training Herations
Trainin Barabons
Training Iteraioos
107
Training tterations
Figure 2: Learning curves for DQN (blue) and DQNPro (red) on four Atari games. X-axis indicates the number of steps from the environment used in training and Y-axis shows undiscounted return (sum of rewards). See also Figure [4 learning curves for all experimental environments.
1.0
A comparison between DQNPro and various deep RL base- lines. To compare the agents across all games, we first normalize the per- formance of all agents we can av- erage over all games meaningfully. To this end, for each combination of al- gorithm, game, and number of train- ing iterations, mance relative to the best policy learned by Rainbow (Hessel al.. 2018). We then average out this quantity across all games.
Figure 3:
0.8
0.6
0.4
Rainbow
compute the perfor-
DQNPro (Ours)
0.2
C51
DQN (Our code) DQN (Dopamine)
0.0
0
0.5
1.0
1.5
25
Training Iterationn
1e7
see if we can still fruitfully combine DDQN and DQNPro since successful combination hinges on the presence of some discrepancy between the two networks.
In Figure S2, we answer this question affirmatively as well by comparing the final performance of DDQN and DDQNPro on the same set of games tested in the previous experiment, where we used the same values for all hyper-parameters including c = 0.2. Notice again, that the value of e 0.2 across all games and for both DQNPro and DDQNPro, demonstrating that this parameter is easy to tune. (See Figure S3 for all learning curves.) Numerous other interesting improvements for DQN exist
RL literature (Hessel et al., 2018). Our goal was to show such combinations are promising, and not to exhaustively test all of them, so we leave further exploration of other combinations for future work.
Finally, it is valuable to evaluate the performance of DQNPro relative to the more recent base- lines. In Figure 3 we provide a comparison between DQNPro, our own DQN implementation, and the Dopamine Castro et al. 2018) implementation for DQN, C51 (Bellemare et al. 2017). and Rainbow (Hessel et al. 2018). . First, observe that our DQN implementation obtains performance analogous to that of the DQN implementation from Dopamine, thus confirming the soundness of our results. More importantly, our DQNPro algorithm is able to achieve more than 50 percent of the improvement of Rainbow to C51. We find this result appealing, because DQNPro is remarkably simpler to understand and to implement relative baselines such as Rainbow.
the original DQN, and is also performing equally well relative
6.3 ABLATION EXPERIMENTS AND FURTHER ANALYSES
Effect of the proximal term. Notice that the purpose of our endowing the agent with the proximal term was to keep the online network in the vicinity of the target network, so it would be natural to ask if this property can be observed in practice. In Figure 4 answer this question affinmatively by plotting the magnitude the update to the target network during synchronization. Notice that periodically synchronize online and target networks, the proximity of the online and target
8

---

Under review as a conference paper at ICLR 2022
Beamnider
Wizardofwor
Gopher
Boxing
Norm of Update
of Uperate
Norm of Update
15
DGNPre
10
10
0.03
0.25
3.00
0.25
0.75
1.0
0.25
0.75
1.03
0.00
0.25
1.00
lleration
Figure 4: The effect of the proximal term on the magnitude of updates to the target network. These results clearly demonstrate that our method has more success than DQN in keeping the online network close to the target network.
Beamrider
Wizardfworr
Gopher
Boxing
100
13000
Averouge Raturn
Averrgg Retter
2000
2000
3000
2000
20
DON pertos 5833
2000
DGN pertoc-- 9333
-20
periou
Training Herraionn
167
Nerratooss
Beamnider
Wiaardowoor
Gopher
Boxing
100
5000
1000
Average Retuar
#2000
Average Roeturn
0030
5000
Awweage
2000
Training Haraaions
107
Training Barations
167
Traiiing Iarations
Training Herations
Figure 5: A comparison between DQNPro and DQN with periodic (Top) and Polyak (Bottom) updates for target network.
network should manifest itself in a low distance between two consecutive target networks. Indeed, the results demonstrate the success of the proximal term in terms of obtaining the desired proximity of online and target networks.
Sensitivity to target network update strategy. While using the proximal term leads to significant improvements when combined with DQN/DDQN, one may still wonder if the advantage of DQNPro over DQN is merely stemming from a poorly-chosen period hyper-parameter in the original DON, as opposed to truly more stable optimization in DQNPro. To refute this hypothesis, we ran DQN with various settings of the period syper-parameter {2000, 4000, 8000, 12000). This set included the default value of the hyper-paramett (8000) from the original paper (Mnih et al. 2015). but also covered a wider set of settings.
Additionally, we tried an alternative update strategy for the target network, referred to as Polyak averaging, which was popularized the context of continuous-action RL (Lillicrap et al., 2013). and which proceeds by updating the target network as follows: 0 <- Tw + (1 (1-7)0 For this update strategy, too, we tried different settings of the т hyper-parameter, namely {0.05, 0.005, 0.0005).
which includes the value 0.005 used in numerous papers (Lillicrap et al. 2015 Fujimoto et al. 2018 Asadi et al. 2021).
Figure presents comparison between DQNPro and DQN with periodic and Polyak target up- dates for various
settings of period and т. clear that DQNPro is consistently outperforming the two alternatives regardless of the specific values of period and т, thus clearly demonstrating that the improvement stemming from a more stable optimization procedure lead- ing better interplay between the two networks.

---

Under review as a conference paper at ICLR 2022
7 RELATED WORK
The introduction of the proximal operator could be traced back to the seminal work of Moreau 1962. 1965), Martinet (970) and Rockafellar (196)), and the use of the proximal operators and related algorithms has since expanded into many areas of science such as signal processing (Combelles and
Pesquet 2009). statistics and machine learning (Beck and Teboulle 2009 Polson et al. 2015 Reddi
et al. 2015), and convex optimization (Parikh and Boyd 2014] Bertsekas 2011b a)
In the context RL. Mahadevan al.) (2014) introduced theory of proximal RL for deriving convergent off-policy temporal difference algorithms with linear function approximation. One in- triguing characteristic their family of algorithms is that they perform updates in primal-dual space, property that was recently leveraged in finite sample complexity analysis (Liu et al.] (2020) for the proximal counterparts of the gradient temporal-diffrence algorithm (Sutton et al.) 2008). Prox- imal operators also have appeared in recent work in the deep RL literature. For instance, in the context of meta-learning, Fakoor et al. (020b) used proximal operators during the adaption phase to keep the parameters close to the meta parameters. Similarly, [Maggipinto et al. (2020) im-
proved TD3 (Fujimoto et al. 2018) by employing a stochastic meximal-poing interpretation and bootstrapping action value estimates for continuous control.
also worthwhile to note that the effect of the proximal term in our work to ensure that the
online network remains in the vicinity of the target network (see also Remark [T). which is remi- niscent of the use of trust regions in policy gradient-based based methods (Schulman et al. 2015 2017] Wang et al. 2019 Fakoor al., 2020a| Tomar et al. 202T). However, three factors differentiate our work: we define the proximal term using the value function, not the policy, we enforce the proximal term in the parameter space, as opposed to the function space, and we use the target network as the previous iterate in our proximal definition.
Finally, existing work RL showed the benefits of leveraging the structure of the RL problem dur- ing optimization. For example, |recupp and Sutton (997)) argued that the space of value functions could better be understood as a manifold with an entropic metric, resulting in the exponentiated TD algorithm. Mahadevan and Liu (2012) used mirror descent penalize weights with large p-norms, a biasing scheme that unfortunately changes the fixed-point of the resultant Bellman equation. Sim- ilarly, van Seijen et al. (219) advocated for performing Q-updates in logarithmic space to combat issues that arise in problems where the action-gar (Farahmand) (01T)) can vary drastically on a state- by-state basis; however, extra work is necessary in their approach ensure that negative values can be represented.
8 CONCLUSION AND FUTURE WORK
We showed a clear advantage in using Proximal Iteration in the context of value-function optimiza- tion in deep reinforcement learning. Proximal Iteration ensures that the online network remains in the vicinity of the target network, and shows robustness with respect to noise and stochasticity in reinforcement learning. Several improvements to proximal methods exist, such as the acceleration algorithm (Nesterov] 1983] Li and Lin 2015), as well as using other proximal terms (Combettes and |Pesquet 2009). which we leave for future work. Our results demonstrate the rewarding nature of developing principled optimization techniques for deep reinforcement learning.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
K. Asadi, N. Parikh, R. E. Parr, G. D. Konidaris, and M. L. Littman. Deep radial-basis value functions for continuous control. In AAAI Conference on Artificial Intelligence, 2021.
H. H. Bauschke and L.. Combettes. Convex analysis and monotone operator theory in Hilbert spaces. 2011.
H. H. Bauschke, M. Moffat, and Wang. Firmly nonexpansive mappings and maximally mono- tone operators: correspondence and duality. Set- Valued and Variational Analysis, 2012.
A. Beck and M. Teboulle. A fast iterative sarinkage-thresholdde algorithm for linear inverse prob- lems. SIAM Journal Imaging Sciences, 2009.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, 2017.
R. E. Bellman. Dynamic Programming. 1957.
D. P. Bertsekas. Incremental gradient, subgradient, and proximal methods for convex optimization:
A survey. Optimization for Machine Learning, 201 la.
D. P. Bertsekas. Incremental proximal methods for large scale convex optimization. Mathematical Programming, 2011b.
P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare. Dopamine: A Research Frame- work for Deep Reinforcement Learning. 2018
L.. Combettes and J.-C. Pesquet. Proximal Splitting Methods Signal Processing. Fixed-point algorithms for inverse problems in science and engineering, 2009.
Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 201
R. Fakoor, P. Chaudhari, and Conference on Uncertainty in Artificial Intelligence, 2020a.
Smola. P3O: Policy-on policy-off policy optimization. In
R. Fakoor, P. Chaudhari, S. Soatto, and A. Smola. Meta-Q-leamnig. In International Conference on Learning Representations, 2020b.
A. Farahmand. Action-gap phenomenon in reinforcement learning. Advances in Neural Information Processing Systems, 2011.
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic meth- ods. In International Conference on Machine Learning. 2018.
M. Fukushima and H. Mine. A generalized proximal point algorithm for certain non-convex mini- mization problems. International Journal of Systems Science, 1981.
M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski. W. Dabney, D. Horgan, B. Piot,
M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
S. J Reddi, S. Sra, B. Poczos, and A. J. Smola. Proximal stochastic methods for nonsmooth noncon- vex finite-sum optimization. Advances in Neural Information Processing Systems, 2016.
S. Kim, K. Asadi, M. Littman, and G. Konidaris. Deepmellow: removing the need for a target network in deep leaarning. International Joint Conference on Artificial Intelligence, 2019.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
11

---

Under review as a conference paper at ICLR 2022
J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. International Journal of Robotics Research, 2013.
D. Lee and N. He. farget-based Machine Learning, 2019.
learning, In International Conference on
H. Li and Z. Lin. Accelerated proximal gradient methods for nonconvex programming. Advances in neural information processing systems, 2015.
Li and J. Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimiza- tion. In Advances in Neural Information Processing Systems, 2018.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2015.
L.-J. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching- Machine learning, 1992.
B. Liu, J. Liu, M. Ghavamzadeh, S. Mahadevan, and M. Petrik. Finite-sample analysis of proximal gradient TD algorithms. In Conference on Uncertainty in Artificial Intelligence, 2020.
M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling, Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
M. Maggipinto,
Susto, and Chaudhari. Proximal deterministic policy gradient. In International Conference on Intelligent Robots and Systems, 2020.
S. Mahadevan and B. Liu. Sparse q-learning with mirror descent. In Conference Uncertainty in Artificial Intelligence, 2012.
S. Mahadevan, B. Liu, P. Thomas, W. Dabney, S. Giguere, N. Jacek, I. Gemp, and J. Liu. Proximal Reinforcement Learning: New Theory of Sequential Decision Making in Primal-Duu Spaces. arXiv, 2014.
B. Martinet. Regularisation, d'inéquations variationelles par approximations succesives. Francaise d' d'informatique et de Recherche operationelle, 1970.
Revue
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 2015.
J. Moreau. Fonctions convexes duales points proximaux dans un espace hilbertien. Comptes rendus hebdomadaires des séances de I' Académie des sciences, 1962.
J. J. Moreau. Proximité et dualité dans espace hilbertien. Bulletin de la Société Mathématique de France, 1965.
E. Moulines and F. Bach. Non-asympiotic analysis of stochastic approximation algorithms for ma- chine learning. In Advances in Neural Information Processing Systems, 2011.
Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence (1/k*) 2). In Doklady USSR, 1983.
N. Parikh and S. Boyd. proximal algorithms. Foundations and Trends in optimization, 2014.
N. G. Polson, G. Scott, and B. T. Willard, Proximal algorithms in statistics and machine learning. Statistical Science, 2015.
D. Precup and R. S. Sutton. Exponentiated gradient methods for reinforcement learning. In International Conference on Machine Learning, 1997.
M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. 1994.
12

---

Under review as a conference paper at ICLR 2022
S. Reddi, B. Poczos, and A. Smola. Doubly robust covariate shift correction. In AAAI Conference on Artificial Intelligence, 2015.
R. T. Rockafellar. Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization, 1976.
E. Ryu and Boyd. Stochastic proximal iteration: non-asymptotic improvement upon stochas- tic gradient descent. 2014.
J. Schulman, S. Levine. P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv, 2017.
Serrano, S. Curi, A. Krause, and G. Neu. Logistic Q-learning. In International Conference on Artificial Intelligence and Statistics, 2021.
D. Silver, J. Schrittwieser, K. Simonyan, Antonogiou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, Bolton, al. Mastering the game go without human knowledge. Nature, 2017.
R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 1988.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
R. S. Sutton, C. Szepesvári, and H. R. Maei. A convergent O(n) temporal-differenee algorithm for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems, 2008.
G. Tesauro. TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 1994.
S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, 1993.
M. Tomar, L. Shani, Efroni, and M. Ghavamzadeh. Mirror descent policy optimization, 2021.
N. Tsitsiklis and B. van Roy. An analysis temporal-diiference learning with function approxi- mation. IEEE transactions on automatic control, 1997.
H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
H. van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil. Deep reinforcement learning and the deadly triad. arXiv, 2018.
H. van Seijen, Fatemi, and A. Tavakoli. Using a logarithmic mapping to enable lower discount factors in reinforcement learning. Advances in Neural Information Processing Systems, 2019.
Y. Wang, H. He, X. Tan, and Y. Gan. Trust region- proximal policy optimization. In Advances in Neural Information Processing Systems, 2019.
C. J. Watkins and P. Dayan. Q-learning. Machine learning, 1992.
J. D. Williams, K. A. Atui, and G. Zweig. Hybrid code networks: practical and efficient end-to- end dialog control with supervised and reinforcement learning. Association for Computational Linguistics, 2017.
S. Zhang, H. Yao, and S. Whiteson. Breaking the deadly triad with target network. arXiv, 2021.
13

---

Under review as a conference paper at ICLR 2022
Appendix: Deep Q-Network with Proximal Iteration
A
IMPLEMENTATION DETAILS
computing infrastructure, and libraries used for the experiments in this paper for all games tested. Our training and evaluation protocols and the hyper- parameter settings closely follow those of the Dopamine (Castro et al. 2018) baseline. To report performance results, measured the undiscounted sum of rewards obtained by the learned policy during evaluation.
Table ST and Table S2 show
DQN and DDQN hyper-parameters (shared)
Replay Buffer size
200000
Target update period
8000
Max steps per episode
27000
Evaluation frequency
10000
Batch Size
64
Update period
4
Number of frame skip
4
Number of episodes to evaluate
2
Update horizon
1
6 e-greedy (training time)
0.01
e-greedy (evaluation time)
0.001
c-greedy decay period
250000
Burn-in period/ Min replay size
20000
Learning rate
10-4
Discount factor (7)
0.99
Total number of iterations
3 107
Sticky actions
True
Optimizer
Adam (Kingma and Ba 2015
Network Architecture
Nature DQN network (Mnih et al. 2015) 1,33,
Random Seeds
Our DQNPro and DDQNPro hyper- parameee
0.2
Table S1: Hyper-parameters used for all methods for all games of Atari-2600 benchmarks. All results reported in our paper are averages over repeated runs initialized with each of the random seeds listed above and run for the listed number of episodes.
Computing Infrastructure
Machine Type
AWS EC2 p2. 16xlarge
GPU Family
Tesla K80
CPU Family
Intel Xeon 2.30GHz
CUDA Version
11.0
NVIDIA-Driver
450.80.02
Library Version
Python
3.8.5
Numpy
1.20.1
Gym
0.18.0
Pytorch
1.8.0
Table S2: Computing infrastructure and software libraries used in all experiments in this pa-
per.
14

---

Under review as a conference paper at ICLR 2022
Anterts
Obert
Breakout
Senqueet
Arridaa
Patt
Traliing beration.
Tokking Racilion
Suakiig becaties
Resdura
Tukking Racalion
13000
Kangaroo
Gourrider
Oopiner
Bosing
Zaxeon
Ave Saaaaga Ftdure
Traningltrrdaon teradins
Ttakking taralions
Training beraiions
Tokkiing hersiion
Training terations
Prostbele
Stargunner
Timespiitt
Venture
Wizardohwer
1000
Traking baratioa
traning
Training berades
Traiinn teraiions
57
Figure S1: Learning curves for DQN with prioritized experience replay (PER) (gary) and DQNPro (red) on 15 Atari games. X-axis indicates the number of steps from the environment used in training and Y-axis shows average undiscounted return.
500%
500%
100%
100%
10%
0%
0%
10%
10%
100%
100%
500%
Jamerbuna
Attres
Camtral
Canyelmber
Demonatte
Endir
Tucaminam
Boxing
Temeptuu
Inveraid
Assaut
Pooran
Battizoons
Froutable
Airad
Sorgumie or
Amidee
Grantt
Boanedd
Nomet thoaae
Astees
Venture
Wassdattoon
chopprccomma
500%
Kang
Figure S2: Performance gain for DDQNPro over the original DDQN in term of the final per-
formance. Results averaged over four random seeds.The y-axis is scaled logarithmically.
B ADDITIONAL EXPERIMENTS
Prioritized Experience Replay (PER). In the standard DQN (Mnih et al.) 2015), data are uni- formly sampled from the reply buffer regardless of their importance. To further improve the perfor- mance of DQN, Schaul al. (006)) proposed effective way, called prioritized experience replay, to prioritize samples that are more conducive to sample-efficiene learning. To further evaluate and situate DQNPro, we compare DQNPro using standard replay buffer against DQN with prioritized experience replay (called PER) on 15 Atari games. As Figure [ST shows our method outperforms PER on the majority of these 15 games. These results provide another data point that our method is more effective approach for improving the performance of DQN while adding negligible compu- tation cost relative to PER.
C LEARNING CURVES
In this section, we provide all learning curves for the experiments described in Section 6.2 of the main text. Specifically, Figure S4 shows learning curves for DQN and DQNPro and Figure S2 and Figure show results comparing DDQNPro against DDQN for 40 Atari-2600 games.
15

---

Under review as a conference paper at ICLR 2022
Ansterty
Gherr
Breakoust
Beoquesi
Auk
6000
0003
1900
2000
10000
2000
Traniing berador
Trakning Penatooa
Training harstiona
Training tarsiore
Tranng
Kangaroo
Courrider
Oopiner
Booing
Zaseon
B
MODE
2000
Traniing taradins
Trakking bacaaions
Traiking beraioon
Tokking hersiion
Teaiiig berations
Prostbele
Stargunner
Timespiiet
Venture
Wizardohher
2000
Training Ieraaiona
Training beraloos
Trainin terations
Alrradi
Allon
Adsouft
Aluerts
Burkhwes
C000
150000
10000
Training Ieraaion
Trakning karaloea
Training Ioraaion
Training baration
Training teraioons
Butthoooe
Beroork
Curvinll
Chopperoeeeaad
Diagydieeber
120000
7000
5000
Training Iernioo
"raininn karaliona
Training Ioradonn
Training taraiies
Training twai
Derronattack
Yarcrovonge
Wideopinballl
Tutarkiaum
dpocekraaders
4000
Traiiing baraions
Trakning baratora
Training Iemioon
Traning tsrnioo
Trannng Tranin Isriion
Rolbstank
Riverrakd
Pooyya
Narv Strreet
Mapacrean
1023
5000
2000
Training kandorra
Trakning tarafora
Training tomesior
Training terai
Training Barnnngerrioon Demnioo
Hunglumaader
Kouil
Granfior
Endero
100
Dook
Traniing Inrdions
Theining Aanalore
Tharning handleea
Traning Hermiom
Tranegternion
Figure S3: Learning curves for DDQN (green) and DDQNPro (orange) 40 Atari games. X- axis indicates the number of steps from the environment used in training and Y-axis shows average undiscounted return.
16

---

Under review as a conference paper at ICLR 2022
Aateett
Obert
Breakout
Senqueet
Ariidar
Heture
10000
Traliing berations
Tokking Pacifion
Sukking bucaiiee
Rendere
Tukeing Racatione
1000
Kangaroo
Courrider
1000C
Oopiner
Bosing
Zaxeon
Traniing laradios
Trakking tacalions
Traiking beraioon
Tokking heraiion
Traiiin berations
2000
Prostbele
Slargunsee
Timespiiet
Venture
Wizardohwer
1900
Traiiing taration
Traniin Sasatieee
Training berad
Training terations
Alrradi
Allon
AdsoulI
Altaris
Berkinest
600
50
Training Ieraaion
Trakning karalion
Training taratonn
Training baration
Trakning baralion
Butthoooee
Borzokk
Coriinll
Choppereeeeaae
Discyclenber
100000
Training Ierriow
Training hersious
Training Ioraaoon
Training terai
Training bwai
Derronattack
Yancrowonge
Videooiibball
Tutarkiaam
Spacekraaders
X000
Averag Seam P.
1000
Traiiing turaions
Trakning haratora
Training Iemioon
Traning thraioo
Training Traningttraion Dsraioo
Rolbstank
Riverrak
Proayan
Mapacrean
1003
5000
2003
2000
Training kandoora
Trakning terafora
Training tomesior
Training terai
Training Traningtrrioo Demiioo
Hunglunaader
Kouil
Grantior
Endero
2008
2000
Das
Training Inrdions
Theining tanaioes
Tharnig handieea
Thanegtersion
Tranegternion
Figure S4: Learning curves for DQN (blue) and DQNPro (red) 40 Atari games. X-axis indi- cates the number of steps from the environment used in training and Y-axis shows average undis- counted return
17

---

Under review as a conference paper at ICLR 2022
Asterix
Broakout
Gravitar
Spatelmaders
:00
25300
8000
Auarogg Neturt
2.90 220
bestug Ratar
15:00
Bestog
100
1230
DONPrO Iparen
DON
Training
Training
Traniing torations
Traiiing
Figure S5: A comparison between DON, DQNPro in parameter space, and DQN in function space.
would be natural to ask if adding the proximal term in the function space can yield similar improve- ments. To test this hypothesis, we augment the loss function in DQN by performing the following update given a batch:
Vh(w) V
Σ (8.0,
1 (r+y Q (()::::)) + (0(s.a:
2|
This will ensure that we still update based on the original loss, but keep the two networks close in the function space. Results shown in Figure compares this idea with the original DQNPro performed in the parameter space. We observe that adding the proximal term in the function space is not nearly as effective as the parameter space. We conjecture that in this case we can only maintain closeness samples drawn from the batch. In other words, the function are still allowed to differ significantly in all areas of the state-action space but those used for training. This ultimately results in inferior performance, Additional work may be needed perform proximal gradient steps in the function space, such as through using natural gradients.
18

---

Under review as a conference paper at ICLR 2022
ADDITIONAL REFERENCES FOR THE APPENDIX
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Mnih, K. Kavukcuoglu, D. Silver, Rusu, J. Veness, M. Bellemare, Graves, Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. International Conference Learning Representations, 2016.
19


Reviews:
[{"review_id": "_IRReKilhqb", "reviewer": "Reviewer_rd5C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points:**\n\nThe main strength of this method is in its relative simplicity relative to its empirical performance improvements. Incorporating an L2 weight regularization penalty into existing frameworks is relatively straightforward, as the authors emphasize in Algorithm 1 by modifying a single line. As shown in Figures 1 and S2, this simple change results in a fairly consistent and significant improvement in task performance.\n\nIn addition to the conceptual simplicity of the proposed method, its computational cost is also negligible, involving only a squared L2 norm between sets of weights.\n\nWhile the proposed method introduces an additional hyperparameter, $c$, the authors note that it is relatively easy to tune, setting it to 0.2 for all experiments based on a hyperparameter sweep on four environments. Thus, it seems likely that this method will be fairly straightforward to apply to other environments and algorithmic setups.\n\nThe empirical analysis provides performance comparisons with relevant baselines over a substantial portion of the Atari 2600 benchmark, a standard set of environments for pixel-based deep RL with discrete actions. The authors directly compare with both DQN and DDQN, while also providing a comparison with the DQN implementation from the Dopamine codebase. One can conclude from this analysis that the proposed method generally improves performance in these settings. The authors also provide additional analyses, investigating the magnitude of updates, as well as sweeping over various update rates and techniques for DQN.\n\n**Weak Points:**\n\nThe primary weak point of this paper is an overall lack of clarity regarding why the proposed method helps. I view this as equally as important, or more, as the empirical evaluation itself. That is, proposing a simple change to deep Q-learning methods may help improve current methods, but understanding why this change helps would improve or unlock a range of future methods, illuminating the root cause of the issue. Currently, the paper alludes to the ‘deadly triad,’ mentioning, but not explaining, “extreme volatility in the optimization procedure.” The authors go on to propose proximal iteration as a method for combatting such volatility, however, as the authors acknowledge, these intuitions are not necessarily applicable in the case of non-linear function approximators. Indeed, the deadly triad manifests in issues with estimating Q-values, e.g., optimism bias. Yet, with non-linear function approximators, proximal iteration in the space of weights provides no guarantees on the stability of Q-value outputs. To properly situate their method, the authors would need to explain a) what is unique about the “extreme volatility” that arises in the deadly triad, b) why it is appropriate to apply proximal iteration in the weight-space rather than the output space, e.g., Piche et al., 2021.\n\n\nThe proposed method is somewhat lacking in novelty, and the authors are missing a substantial number of citations of previous works that have investigated constraining parameters or outputs in value networks. Examples include natural gradient deep Q-learning (Knight & Lerner, 2018), PreQN (Achiam et al., 2019), KOVA (Di-Castro Shashua & Mannor, 2020), GAE (Schulman et al., 2015), etc. While the proposed method does not exactly match the setups in these previous works, the motivation and overall idea are highly similar. At the very least, a more in-depth discussion of related work is required. Further, given the similarity with these previous works, it would be helpful to compare with some other stabilization schemes.\n\nThe authors investigate their method entirely in the Atari domain. Accordingly, it’s unclear whether the hypothesized instabilities in Q-learning are more pronounced in these environments and whether the proposed method is generally applicable outside of this domain. While I do not see this as grounds for rejection, experiments in other domains would help to bolster the authors’ claims.\n\nLikewise, because the authors only evaluate on Atari, their method is entirely evaluated on discrete control tasks, whereas Q-networks are also utilized in continuous control tasks. To truly demonstrate the generality of the problem and proposed solution, it would be helpful if the paper included at least some experiments with continuous control experiments, e.g, with actor-critic algorithms. The proposed method could also be applied for the purposes of state-value estimation.\n\n\n- Piche et al., 2021. Beyond target networks: improving deep q-learning with functional regularization\n- Knight & Lerner, 2018. Natural gradient deep q-learning\n- Achiam et al., 2019. Towards characterizing divergence in deep q-learning\n- Di-Castro Shashua & Mannor, 2020. Kalman meets Bellman: improving policy evaluation through value tracking\n- Schulman et al., 2015. High-dimensional continuous control using generalized advantage estimation.\n\n\n**Additional Comments:**\n\nSome of the background in sections 2 and 3 feels overly pedagogical and/or not applicable to the setting used in the paper. For instance, it’s not clear what the discussion of the contraction mapping really adds to the paper. Similarly, much of the paper is devoted to discussing proximal methods in the context of strongly convex objectives, e.g., with linear functions, which is not the setting used in the paper. This space would be better utilized by characterizing the instabilities with Q-learning and explaining why proximal iteration in weight space is appropriate for tackling these instabilities.\n\nEquation 6: I believe the last line shouldn’t be indented. Also, this feels somewhat redundant, given that it’s a simplified version of Algorithm 1.\n\nResults: I would present the DDQN results in the main paper, perhaps right alongside the DQN results. It’s not clear how the 40 environments were selected from the total set of environments. The learning curves in Figure 2 are cherry-picked. It’s unclear what the update magnitudes tell us about the algorithms. These results need to be tied more specifically to the issues with deep Q-learning.\n", "summary_of_the_paper": "This paper applies proximal iteration, via L2 regularization from the weights of the target network, to deep Q-learning. Although the change in the algorithm is minimal, the authors demonstrate performance improvements across Atari environments as compared with baselines.", "summary_of_the_review": "My reasons are outlined in the weak points section above. While this paper provides a set of promising empirical results, the relative lack of explanation and analysis around why proximal iteration helps in deep Q-learning significantly detracts from the impact of this paper. I would like to see these aspects of the paper improved, perhaps drawing on previous works that have tackled this problem. Relatedly, given that other works have attempted to improve deep Q-learning via regularization, these works need to be cited and, ideally, compared against. Finally, it would help to expand the scope of the results section to include other environments or setups in which deep Q-learning is used, e.g., actor-critic algorithms, demonstrating the generality of the proposed method. In its current form, this paper will leave readers a) confused regarding what problems the proposed method is tackling, b) unaware of previous works in this area, and c) unsure about the method’s generality. For these reasons, I cannot recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "U-ND4TyoI9T", "reviewer": "Reviewer_xGvm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\nThe method proposed is simple and effective, although proximal iteration is common in RL algorithms, this is the first trial to apply it to the parameters of the critic networks. The authors theoretically analyze the advantage of proximal iteration. The experimental results are solid and the ablation study is carefully conducted. The overall writing and organization are excellent.\n\n$\\textbf{Weaknesses}$\nThe technical contribution is limited given what the authors show in the experiment part, both DQN and DDQN are old models, so I think the authors should try to address at least one of the following aspects.\n\n(1) Apply the proposed technique to more advanced deep reinforcement learning models, like DDPG and SAC on continuous control problems.\n\n(2) Analyze the source of improvement of other DQN variants, in terms of the magnitude of the updates to the target network similar to Figure 4, I think this will give a new perspective into the understanding of the critic network.", "summary_of_the_paper": "This paper innovatively adds the proximal iteration to the update of the critic network, without the additional computational cost compared with the original SGD optimization method, it empirically demonstrates the advantage of this technique.", "summary_of_the_review": "This paper is innovative and well written, the experiment parts are very solid and the authors give a clear analysis. But I'm concerned about its technical contribution and conservatively rate it as around the borderline. I hope the authors could address or explain their thoughts about my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NJOkuw-WkT_", "reviewer": "Reviewer_UNLa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors point out the various challenges in stabilizing deep reinforcement learning. They argue that applying the idea of proximal iteration to DQN can improve the stability of the learning and the performance of the algorithm. In practice, this implies a small change in the weight update term of the DQN algorithm, which biases the weight changes towards the target network. \n\nThe authors compare the resulting algorithm with the original DQN both the version from Google Dopamine and the authors' own reimplementation of it. The approach improves on the original DQN, but it does not matching the current state of the art. \n\nAs a note: Formula 6 and algorithm 1 are different - formula 6 does not capture the fact that the \\theta is updated only very rarely.", "summary_of_the_paper": "The paper proposes a modification of the DQN algorithm by adding a proximal term that biases the learning to retain the Q algorithm in the proximity of the target network. The authors show that the proposed algorithm perform better compared to the original DQN algorithm on several Atari games. ", "summary_of_the_review": "The paper describes a relatively simple modification of the DQN algorithm using a proximal term. The approach improves the performance of DQN, but it does not improve on the current state of the art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "LS-m1fRmr8p", "reviewer": "Reviewer_ULKo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nThis submission is well written and easy to read. The authors conduct extensive experiments to demonstrate the proposed regularization term in proximal iteration boost the performance of different DQN-type algorithms on a lot of Atari games.\n\n**Weakness:**\n1. My biggest concern is about the novelty of this submission. Although the regularization term in Equation 4 is effective for DQN-type algorithms, the authors did not give a convincing explanation to explain why proximal iteration works in this setting. In addition, the theoretical analysis in the main text (Remark 1 and 2) are all existing results, and the authors do not introduce new theoretical results to the community.\n2. The authors propose to bias the online weights towards the target network, but no detailed explanation is given for this choice. A more natural option is to bias the online weights towards the old weights in the previous iteration, the authors should at least compare the performance of these two options.\n3. In Equation 6, $\\theta$ is updated every iteration, but in Algorithm 1, $\\theta$ is updated only when numUpdates % period = 0.\n4. The author just verified through experiments that DQNPro can be combined with DDQN. Since there are some contradictions between the functions of DQNPro and DDQN, is there a deeper explanation for the performance improvement?\n5. Minor issue: It is better to redraw some figures in the experiment section using more professional software, Figures 2-5 look blurry.\n", "summary_of_the_paper": "This submission exploits *Proximal Iteration* to optimize DQN-type RL algorithms which ensures the online-network remains in the vicinity of the target network to stabilize the training process. This simple *trick* improves the performance of DQN and DDQN on a set of Atari-2600 environments.", "summary_of_the_review": "The authors use a simple trick to improve the performance of DQN-type algorithms on some Atari environments. Although the proposed method has achieved good results, the novelty is somewhat limited, and the reasons for the performance improvement are not explained clearly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper applies proximal iteration to Q-learning, which significantly improves the performance of DQN. Reviewers agreed the paper is not ready for publication, for a couple reasons. DQN is quite far from current state-of-the-art. Improvements therefore need to be well-founded to be of broad interest. If the algorithm that is being improved is not competitive, there should be more general lessons that can be extracted from how and why the improvement works. Unfortunately, the reviewers felt that there was insufficient understanding of why proximal iteration helps.
Decision: Reject