Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization
Authors: Ke Sun
Yafei Wang
Yi Liu
Yingnan Zhao
Bo Pan
SHANGLING JUI
Bei Jiang
Linglong Kong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Anderson Acceleration
Reinforcement Learning

Abstract: Anderson mixing has been heuristically applied to reinforcement learning (RL) algorithms for accelerating convergence and improving the sampling efficiency of deep RL. Despite its heuristic improvement of convergence, a rigorous mathematical justification for the benefits of Anderson mixing in RL has not yet been put forward. In this paper, we provide deeper insights into a class of acceleration schemes built on Anderson mixing that improve the convergence of deep RL algorithms. Our main results establish a connection between Anderson mixing and quasi-Newton methods and prove that Anderson mixing increases the convergence radius of policy iteration schemes by an extra contraction factor. The key focus of the analysis roots in the fixed-point iteration nature of RL. We further propose a stabilization strategy by introducing a stable regularization term in Anderson mixing and a differentiable, non-expansive MellowMax operator that can allow both faster convergence and more stable behavior. Extensive experiments demonstrate that our proposed method enhances the convergence, stability, and performance of RL algorithms.

Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization
Ke Sun"1, Yafei Wang"), Yi Liu', Yingnan Zhao? Bo Pan', Shangling Jui",
Bei Jiang', Linglong Kong
University of Alberta, Edmonton, Canada
Harbin Institute of Technology. Harbin, China
HHuawii Technologies Ltd.
(ksun6, (ksun6,yafe12) yaw.ia, yliu16,,11111 yingnan,n pan1 beil, [koon)) jui shangling@huaweel.
Abstract
Anderson mixing has been heuristically applied reinforcement learning (RL. algorithms for accelerating convergence and improving the sampling efficiency deep RL. Despite its heuristic improvement convergence, a rigorous mathe- matical justification for the benefits of Anderson mixing in RL has not yet been put forward. In this paper, we provide deeper insights into a class of acceleration schemes built Anderson mixing that improve the convergence of deep RL al- gorithms. Our main results establish a connection between Anderson mixing and quasi-Newton methods and prove that Anderson mixing increases the convergence radius of policy iteration schemes by an extra contraction factor. The key focus the analysis roots in the fixed-point iteration nature of RL.. We further propose stabilization strategy by introducing a stable regularization term in Anderson mix- ing and differentiable, non-expansive MellowMax operator that can allow both faster convergence and more stable behavior. Extensive ex experiments demonstrate that our proposed method enhances the convergence, stability, and performance of RL algorithms.
Introduction
In reinforcement learning (RL) 28 an agent seeks an optimal policy in sequential decision- making process. Deep RL has recently achieved significant improvements a variety challenging tasks, including game playing [20 26 [8] and robust navigation [19]. A flurry of state-of-the- art algorithms have been proposed, including Deep Q-Learning (DQN) 20 and variants such as Double-DQN 13], Dueling-DQN [31], Deep Deterministic Policy Gradient (DDPG) [16), Soft [2] and distributional RL algorithms [7 33), all of which have successfully solved end-to-end decision-making making problems such as playing Atari games. However, the slow convergence and sample inefficiency of RL algorithms still hinders the progress of RL research, particularly in high-dimensional state spaces where deep neural network are used as function approximators, making learning in real physical worlds impractical.
Actor-Critic
To address these issues, various acceleration strategies have been proposed, including the classical Causs-Seidel Value Iteration 23 and Jacobi Value Iteration [24). Another popular branch of tech- niques accelerates RL by leveraging historical data. Interpolation methods such as Average-DQN 2
"Equll contributions in alphabetical order Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

have been widely used in first-order optimization problems [6] and have been proven to converge faster than vanilla gradient methods. As an effective multi-step interpolation method, Anderson mix- ing [30 2 also known as Anderson acceleration, has attracted great attention from RI. researchers. The insight underpinning of Anderson acceleration is that RI. [28] is intrinsically linked to fixed-point iterations: the optimal value function is the fixed point of the Bellman optimality operator. These fixed-points are computed recursively by repeatedly applying an operator of interest [[]. Anderson mixing is a general method to accelerate fixed-point iterations [30] and has been successfully applied to fields, such as the computational chemistry [22] or electronic structure computation M. In particu- lar, Anderson acceleration leverages the previous estimates in order to find better estimate in fixed-poin iteration. To compute the mixing coefficients in Anderson iteration, it searches for a point with minimal residual within the subspace spanned by these estimates. It is thus natural to explore the efficacy of Anderson acceleration in RI. settings.
Several works 15 25 have attempted to apply Anderson acceleration to reinforcement learning. Anderson mixing was first applied to value iteration in [I] L.5 and resulted in significant conver- gence improvements. Regularized Anderson acceleration 225 was recently proposed to further accelerate convergence and enhance the final performance state-of-the-art RL algorithms in various experiments. However, previous applications of Anderson acceleration were typically heuristic: con- sequently, these empirical improvements in convergence have so far lacked a rigorous mathematical justification.
In this paper, we provide deeper insights into Anderson acceleration in reinforcement learning by establishing its connection with quasi-Newton methods for policy iteration and improved conver- gence guarantees under the assumptions that the Bellman operator differential and non-expansive. MellowMax operator is adopted to replace the max operator in policy iteration to simultaneously guarantee faster convergence of value function and reduce the estimated gradient variance to yield sta- bilization. In addition, we analyze the stability properties of Anderson acceleration in policy iteration and propose a stable regularization to further enhance the stability. These key two factors, i.e., the stable regularization and the theoretically-inspier Mellow operator, are the basis for our Stable Anderson Acceleration (Stable AA) method. Finally, our experimental results on various Atari games demonstrate that our Stable AA method enjoys faster convergence and achieves better performance relative to existing Anderson acceleration baselines. Our work provides a unified analytic framework that illuminates Anderson acceleration for reinforcement learning algorithms from the perspectives of acceleration, convergence, and stabilization.
2 Acceleration and Convergence Analysis of Anderson Acceleration on RL
We first present the notion of Anderson acceleration in the reinforcement learning and then provide deeper insights into the acceleration if affords by establishing a connection with quasi-Newton methods. Finally, theoretical convergence analysis is provided to demonstrate that Anderson acceleration can increase the convergence radius of policy iteration by an extra contraction factor.
Background Consider a Markov decision process (MDP) specified by the tuple (S.A. where S is a set of the states and A is a set of actions. The functions R. S x A R and p : S A x - [0, 1] are the reward function, with R2 = R(s, a), and transition dynamics function, respectively for the MDP. The discount rate is denoted by 7 E [0, 1) and determines the relative importance of immediate rewards relative to rewards received in the future. The Q- value function evaluates the expected return starting from given state-action pair (s, a). that is, Q= a) = E م> 7² Re+1 s0 = s, ag = a]. A policy π (aas) is a distribution mapping the state space S to the action space A.
2.1 Anderson Acceleration in Policy Iteration
We focus on the tabular case to enable the theoretical analysis of Anderson acceleration in value (policy) iteration, which can be naturally applied to function approximation. Both the value itera- tion (V (V-notation) and the policy iteration (Q (Q-nooaaion) can have Anderson acceleration applied to them to improve convergence. However, theoretical analysis has shown that value iteration enjoys a y-linear convergence rate, i.e., |]V(() V* & ((-1 V* where V(O) is the value function in the iteration step t and V* is the optimal value function, while policy iteration converges

---

faster. This is due to the fact that policy iteration more fully evaluates the current policy than does value iteration. Additionally, policy iteration more fundamental and scales more readily to deep reinforcement learning. For this reason, we analyze our method in the policy iteration setting under Q-notation as value iteration special case of policy iteration. Thus, our analysis also applies under value iteration.
We first focus on the control setting where the optimal value of state-action pair (* a) is defined recursively as function of the optimal value of the other state-action pair:
(* = R(s,a( + Ev(o' 18,0) max Q' (d',a').
(I)
E'ES
Combining policy evaluation and policy improvement, the resulting policy iteration algorithm is equivalent to solving for the fixed point of the Bellman optimality operator T RISSA - IR Sx.A]
with TQ(s, a) R(s,a) a) + 7 Ex'Es ( (s'(s,a) max, Q (s', d').
As a general technique to speed up fixed-point iteration [30]. Anderson acceleration has been successfully yet heuristically applied to reinforcement learning algorithms [15 Specifically, Anderson acceleration linearly combines the m previous estimates to yield a better iteration target in the fixed point iteration. Geometrically, Anderson acceleration applies the operator to a point that has a minimal residual within the subspace spanned by these estimates. In policy iteration, Anderson acceleration maintains memory of the previous Q function values and updates the iterate linear combination of these values with dynamic weights (A) in the kth iteration step. Specifically,
Q(k+t)(s, = (1 3x)
(e) Q () (8, + 8K
(k) TQ (st
where 0 S 3x S is the damping parameter. All of the coefficients a; in the coefficient vector (i) are computed following
m
(k) argmin agRm+1
Σ a4 (TQ(k-mt)) -
-0i0t)))Q(mm-ta Q(k-m+t)) argmin oERm+1
A T a s.t. Σ a -a=1, 1=0
where A/2" [ek-m.... Cal (15xAIx(m++) and e TQ Q(*) E RSSx is the Bellman residuals matrix. By the Mranh--uhn-TTuee conditions, the analytic solution of optimal coefficient vector a is
(k) (A%.4) 1
(4)
-
1 (AI))))
where 1 denotes the vector with all components equal to one.
2.2
Connection Between Damped Anderson Acceleration and Quasi-Newton Methods
We know that the optimization problem is closely linked with solving fixed-point iteration problem by directly solving its first-order condition. Inspired by [ we show that Anderson acceleration in policy iteration attempts to perform special form of quasi-Newton iteration from its optimization problem behind.
To illuminate this connection, we firstly show that the original constrained optimization to obtain the optimal (k) in Eq. 33 can be equivalent to the unconstrained one
(k) argmin |lez - H&T
(5)
where we let |S| x |A], and then Hy [es 1k-1, Eh Ch-m+1 Ck-m] 8
(k)
(k) Ti
(k) (k) E R with T. (k) Σm- (k)
Let dy = Q(k) Q(=-1),
aj
A = [5c, ".. 5 +1 Rexm, We show that the updating rule of Q(s, a) Anderson acceleration can be expressed as quasi-Newton form in Proposition]
3

---

Proposition 1. By conducting the damped Anderson acceleration (Eq. 2 and 3 on the policy iteration, the updating of Q (x+1)) can be reformulated as
Q(++1) := =Q Ggek
where G2 = (Ax + H&) (HT H - can serve as an approximation of inverse Jacobian matrix 6x = TQQ() - ((k), and I is an identity matrix.
Proposition I points out that Anderson acceleration on policy iteration additionally leverages more information about the fixed-point residual 6 TQ( - (()) to update the Q function. Particularly, the first part (Ax + (8. (H, (H? - H in Gj contains partial structure matrix information about the real inverse of Jacobian matrix, which has the huge potential to speed up the convergence of the fixed-point iteration. More importantly, the results established in [30 and [ can been seen special cases of Proposition [ with 3x = 1. If we directly get rid of the first part in G3 and set 34 = 1, the updating rule exactly degenerates the Q-value function iteration without Anderson acceleration.
2.3 Convergence Rate Analysis of Anderson Acceleration RL
The success of the Anderson acceleration to reduce the residual coupled in the algorithm iteration (8) (TQU) -
at each stage. Let eg = Σ 1=0 0
Q())). The stage-k gain 02 can be defined by ||eğ =
Os |exll// As Ogg (k) 1, (k) 0.j i.e., = 0, is an admissible solution to the optimization problem in Eq. 3 it immediately follows that 0 < 0g S 1. The key to rigorously show that Anderson acceleration can speed up the convergence of policy iteration by taking linear combination of history steps is connecting the gain 0x to the differences of consecutive iterates Q(A) and residual terms ek- As discussed in the following part, the improvement of the convergence rate of the policy iteration by using the acceleration technique is characterized by 0g. We first consider the following assumption about the operator T used to guarantee the first and second order derivatives of T are bounded, as in
Assumption 1. Assume the Bellman operator T acting on state-action value function Q has a fixed point Q", and there are positive constants e₁ and 02 such that
T E (RISXAI).
2. The first derivative T is bounded by c1-
3. The second derivative T is bounded by c2-
Theorem 1. Under Assumption 1 if the coefficients oi; (k) remain bounded and away from zero, the
following bound holds for the fixed point residual from Eq. Q and (3) with depth m
|| S 0% {((1 - 3x-1) + 3-) /ex--llach + C2 (1ldello + +116, 11
+ C2
@ + -102_ +2
Σ 1=1
66 1000000
116
Note that the first term of RHS in Eq. 7 characterizes an increased convergence radius by an extra contraction factor 0g. Therefore, if the error terms [|0g= are small enough, and the operator T is differentiable and has bounded first and second derivatives, the faster convergence result characterized by 0g can be derived for the Q-value function iteration with Anderson acceleration.
Unfortunately, the commonly used max operator T does not satisfy Assumption 1 as it is not a differentiable operator. Moreover, the "hard" max operator in T always commits to the maximum action-value function according to current estimation for updating the value estimator, lacking the ability to consider other potential action-values. A natural alternative is the Boltzmann Softmax operator, but this operator is prone to misbehave
(3] is not non-expansive operator. Mellow Max operator (3) which can help strike balance between exploration and exploitation, is considered in this paper. More importantly, the more meaningful convergence result of Anderson acceleration in policy iteration under Assumption Bellman operator under MellowMax operator. The results are given in Theorem 2
2 can be established due to the contraction properties of the
Assumption 2. Assume the Bellman operator T acting state-action value function Q is a 7 contraction operator, ie., I[QQ TQ'Il oo S <1 - Q' for each state-action function pair Q and
Q'.

---

Theorem 2. If both Assumption I Mandl hold, the coefficients og zero. The following bound holds for the residual ej with depth m
(K)
remain bounded and away from
| s Os 1k-1) 3h 1 1ellle--llla + 0 l 1ex=)
From Theorem 2 we find that there is theoretical advantage consider Anderson acceleration for policy iteration with depth m due to the gain 0g even with the higher-order accumulating terms. Fortunately, the Bellman operator with MellowMax operator is 7-contraction operator, satisfying Assumption
2 Specifically, the resulting Bellman operator Tmm under the Mellow.axx operator is
defined as
Tam (ss, a) = R(s, a) +
p('s (s' Ep(s'
a, mmw (Q ()),
(9)
S'ES
where mm, is the MellowMax operator and mm (s'..) log( [i] E. expluQ(s', d')l)/w. Rigorously, we show that MellowMax can simultancously satisfy Assumption [and Zi Appendix B As such, the faster convergence result presented in Theorem 2 can be derived. In other words, the faster convergence of policy iteration under MellowMax operator can be established by applying Anderson acceleration. Based this theoretical principle, where a detailed discussion is also provided.
apply MelloxMax operator in the
Bellman operator to design our method in Section 3
Finally, the Q-value estimate ((A) can be obtained by iteratively applying the MellowMax operator by starting from some initial value Q()):
Q(e+1) - (1 - B&)
(K) Q ( + BE
( Q(k-mtt),, V(s,a) E (S. (10)
t=
1=
3 Stabilization Analysis and Our Method
In this section, stable regularization is firstly introduced and its stability analysis is provided as well. We then briefly analyze the role that MellowMax operator plays when conducting Anderson acceleration on deep reinforcement learning. These two factors eventually inspire our algorithm, which we call Stable Anderson Acceleration (Stable AA).
3.1 Stable Regularization
Inspired by recent stable results of Anderson acceleration 19] we introduce the stable regularization term on the aforementioned unconstrained optimization problem Eq. (5) obtain mixing coefficients 7h_ Particularly, we add f2 regularization of 7 scaled by the Frobenius norm of A₁ and H₁, to improve the stability. This yields the new optimization problem
argmin |lex 7ER"
H&r||² + (IAx ||H& |||
(1I)
where 17 is positive tunning parameter representing the scale of regularization. The solution is ex. We introduce this stable regularization under the unconstrained variables 7 which facilitates the optimization. Intuitively, if the algorithm converges, we have limg-+o ||A limg-+oe || 11 0. Therefore, the coefficient on the regularization term vanishes as the algorithm converges, degenerating to Anderson acceleration method without stable regularization plays more important role the early phase training, which we demonstrate Section [ Based on the solved stable regularization rx and the relationship between n and (&), we derive the updating of ((++)) in the policy iteration as follows
7 (H. H2 + 1| | |
the stable regularization. In this sense, the stability owing
((++1) = =Q(() GRex,
(12)
where G₁ - (Ax BeF (H He v + ||Hx||21) HE Moreover, the following Theorem characterizes the stability ensured by regularization in Eq Please refer to the proof in Appendix C
5

---

Theorem 3. The matrix G3 satisfy ||Gxll2 - 12/n Bxl, ||G GAlle < 1.
Theorem derives the upper bound of |G which is determined by 7j and 3x. Intuitively, larger strength of regularization 7 and proper magnitude of 3x can yield more stability. In addition, ||& G is strictly less than 1, revealing a smaller violation in Q(k) iteration compared with non-regularized form.
To quantify the effect of regularization the coefficient (A), we provide some analytical results regarding the obtained mixing coefficients (*) in Proposition The proof is provided in Appendix C
(k) Proposition 2. Let Ohoo and Oreg be the mixing coefficient vectors obtained by vanilla unconstrained and our stable regularized Anderson acceleration, respectively. Define the transformation matrix as A, satisfying (k)) A 7h with ((k) (1, (1 (detailed structure of A is in the Appendix Cy. ||A||2 ||
(k)
Let cond2(A) be the conditional number of A, i.e., condz(A) =
11.. Then we have the
following inequalities
(k) | 112 I 4(1 +
1ea11 112
|| (R)
200 11 (cond2(A))
a
(R)
2m + 1
(13)
From the first inequality, observe that the norm of the derived coefficients Oreg is controlled by the regularization parameter An overly large tends to reduce the bound for the norm of Oreg
(k)
(k)
implying stable solution of the mixing coefficients orgg. Besides, we can conclude from the second (R)
inequality that there is an inevitable gap between
(R) and Orjeg
3.2 Stability Effects of MellowMax
The adopted MellowMax operator bridges the Anderson acceleration and reinforcement learning algorithms and it has two-sided stability effects. Firstly, based on the convergence analysis in Section 2 MellowMax operator satisfies the differential and non-expansive properties, which allows the faster convergence of Anderson acceleration in policy iteration. In contrast, the commonly used violate one the theoretical assumptions respectively, and thus the (faster) convergence of Anderson acceleration under them may not be guaranteed. This likely to yield instability while the training of algorithms.
max and Boltzmann Softmax operator 4 28
Secondly, it is well-known that the "hard" max updating scheme in the popular off-policy methods, such as Q-learning [32] may lead to misbehavior due the overestimation issue in the noisy environment 22
13 21 By contrast, it has been demonstrated that MellowMax and Softmax operators capable of reducing the overestimation biases, therefore reducing the gradient noises to stabilize the optimization of neural networks 33 (7). The stable gradient estimation based on MellowMax operator leads the enhancement of final performance for algorithms.
3.3 Algorithm: Stable AA
The introduced stable regularization approach combined with the MellowMax operator finally form our Stable AA method. In our algorithm, we focus on exploring the impact of Stable AA Dueling- DQN [31]. In particular, under the procedure of off-policy learning in DQN, we firstly formulate the general damped Anderson acceleration form with the function approximator Qo follows
Qu (8t, oz) = 3r Σ agQus (8t, az) +
+11 (1 B.
Tt +
Σ 1=1
On max Qui (8e+1, Gx+1) o
(14)
where 0; are parameters of target network before the i-th update step. d; can be computed either by vanilla Anderson acceleration [II), or Regularized Anderson acceleration (25) Then the obtained Qo (se, az) serves the target in the updating of Q-networks. In our Stable AA method, we firstly solve the optimization problem in Eq.
(TT) to compute n². Next we obtain (*()) by making use of the quantitative relationship between 7 and (k), More importantly, we substitute max with MellowMax operator mm... The resulting target value function Qe in our Stable AA algorithm is

---

Algorithm Stable AA Dueling-DQN Algorithm
1: Initialize Q value network Qo and target networks with parameters 0g (i= 1, 1,... m). Set the total training steps K and updating step M.
2: while k - do
3: Observe the initial state So;
4: fort to T do
5:
Select 0g arg max, Qo(se, a) with probability 1- and a random action with probability
6:
Perform the action ag, obtain rt and 81+1- Store the transition (st, dz, Tty $1+1) in the replay buffer.
7:
Sample the batch of transitions (s, (5,0,7 r,, from the replay buffer.
8:
1* Step 1: compute ((k) /
9:
Compute AS and H3, and then solve the optimization problem with stable regularization in Eq. (TT) to obtain -².
Obtain the optimal coefficient vectors ((k) via ((k) .. where the transformation matrix A is defined in Proposition 2
11:
1* Step 2: compute the target Qe by Anderson Mixing *
12:
Compute the value after the MellowMax operator for each target network Qe i.e., mmw (Qo, ())
Evaluate the target value function Oo (s,, at) via Eq. (15) under the Melloxmax operator.
13:
14:
1* Step 3: update the Q value networky *
Update the Q value network by minimizing the loss Eq. with the target yt from
Step 2.
16:
Update target networks every M steps, i.e., 0, 0z+1 (i 1,... m) and Om O.
Set k k +
18:
end for
19:
end while
reformulated as
2o (st. ax) = Be Σ 2o a1) + (1- Be Be
+
āg mmm(Qur (s141:
(15)
where (Q(s,) is the MellowMax operator. Finally, the updating is performed by minimizing the least squared errors of Bellman equation between the current Q value estimate Q0(8e, at) and the target value function yt obtained from Eq. (15
((0) = E OED (w Qo
Qo (sc,
(16)
where the distribution of previously sampled transitions.
In summary, the key of Stable AA method in policy iteration lies in two factors: the stable reg- ularization in Eq. (11) in computing coefficient (k)), and the MellowMax operator enabling the faster convergence updating ((k), both which improve the convergence and sample efficiency. Moreover, we provide detailed description of Stable AA Dueling-
algorithm in Algorithm Similar to the strategy in 25 the incorporation of Stable AA into policy gradient based algorithms, including actor critic (28 and twin delayed DDPG (TD3) (10) can be directly implemented in their critics part. It can be viewed as a straightforward extension, and we leave this exploration as future works.
4 Experiment
Our theoretical results about Anderson acceleration mainly apply the case of tabular value function representation, but our derived Stable AA algorithms can be naturally applied into the function approximation setting. The goal of our experiments to show that our Stable AA method can still be useful in practice by improving the performance of Dueling-DQN algorithms. Our experimental results demonstrate that such an improvement is attributed the joint benefits of the proposed stable regularization and the MelloMax operator.

---

Experimental Settings We perform our Stable AA Dueling-DQN algorithm on a variety of Atari games, and mainly focus on reporting four representatiave games, i.e., SpaceInvaders, Enduro, Breakout, and CrazyClimber. Results of other games are similar, which we provide in Appendix We compare our approach with Dueling-DQN [31 and Regularized Anderson Acceleration (RAA) In addition, we also provide an ablative analysis about our Stable AA algorithm to illuminate the separate and joint impacts stable regularization and MellowMax operator. In the following experiments, we report results statistics by running three independent random seeds. We set Br in Eq. [13 as 0.1 for convenience.
Implementation of Mellow Max Operator The Mellow operator mam. satisfies the desirable differential and non-expansive properties, enabling the faster convergence in policy iteration with Anderson acceleration. Nevertheless, we need perform an additional root-finding algorithm [3] to compute the optimal w in each state in order to maintain these properties and help the MellowMax operator identify a probability distribution. Unfortunately, this root-finding algorithm is compu- lationally expensive to be applied. Following the strategy in [27]. we tune the inverse parameter w from [1, 5, 10) and then report the best score.
4.1 Performance of Our Stable AA
We select DuelingDQN and DuelingDQN-RAA as baselines for the evaluation on the four Atari games. These two algorithms and our approach DuelingDQN-Stable AA are trained under the same random seeds and evaluated every 10,000 environment steps, where each evaluation reports the average returns with standard deviations. Our implementation is adapted from RAA 225 After the grid search, we set w in MelloxMax of Stable AA as 5.0 and in stable regularization as 0.1 across Atari games.
SpaceInvaders
Enduro
800
DellePOK
500
Average Return
700
DellepPON
Average Return
DeskegOOK, SubiiAApii
640
400
400
300
300
200
200
100
0.5
0.3
1.0
1.2
1.4
0..5
0.8
1.0
1.2
14
Time Steps (1c7)
Time Steps (1c7)
Breakout
CrazyClimber
Delleggg
Dovingggg
Average Returnn
240
DallngOO
Average Return
60000
Dulliggh
DalkegOO SuhiAApead
Duilnghh StalaAAienn
80000
1.50
100
40000
20000
Time Steps (1c7)
Time Steps (1c7)
Figure 1: Learning curves of DuclingDQN, DuelingDON-RAA and our approach DuclingDON- Stable AA on SpaceInvaders, Enduro, Breakout, and CrazyClimber games over seeds. Shaded region corresponds to the standard deviation.
From Figure M we that our DuclingDQN-Stable (red line) significantly outperforms Regular- ized AA (blue line) and baseline (black line) across all four games. Overall, Dueling -RAA enables to accelerate DuelingDQN to improve the sample efficiency and enhance the final performance, but our approach can lead to further benefits. Remarkably, our DuelingDQN-Stablee StableeAA (red line) is superior to RAA to large margin, especially on Breakout where our approach achieves around 250 average return while RAA only achieves 150 return. In summary, we conclude that the joint impact of both stable regularization and theoretically prinncppee MellowMax further accelerate the convergence and improve the sample efficiency of the popular off off-colicy DuelingDQN algorithm.

---

4.2
Ablation Analysis
We further examine the separate impact of the proposed stable regularization (shown Eq. (TT) and the theoretically-prinipph
MelloxMax operator via the rigorous ablation study. Starting from Due, we firstly add stable regularization with different scales 1/ while comparing with our resulting Stable AA method. Meanwhile, we separately replace Max operator in DuelingDQN with MellowMax operators with various inverse parameters w to explore their impacts.
SpaceInvaders: Stable Regularization
Enduro: Stable Regularization
Average Return
700
PadiraPQN
-Suhidkapt
Averago Return
*sabikggh11
690
600
400
-SukiiRept i
+Sub23 agh i-Mrleedoosa
200
200
0.6 Time Steps (Ie7)
Time Steps (1
SpaceInvaders: Mellow Max
Enduro: MellowMax
800
Average Return
PadieaPNN
-Milinohlaci
Average Return
500
Daslagoji
-Mclswhhees8
-Miisohlac118
600
+Sddbwillattio
300
-Robidkopt i
400
400
300
0.2
0.6
0.5
0.5
3.0
144
0.2
0.4
0.6
0.8
1.0
3.2
Time Steps (1c7)
Time Steps (1c7)
Figure 2: Learning curves of DuelingDQN, +MellowMax, +Stable Regularization (+StableReg). and +Stable AegularizztionnMelllwwaaa on Spacelnvaders, Enduro games over 3 seeds.
Impact of Stable Regularization From diagrams in the first row of Figure we can observe that the benefit margin of stable regularization on Anderson acceleration differs from game to game. Concretely, naively applying stable regularization on DuelingDQN regardless of the MellowMax to guarantee the faster convergence of Anderson acceleration can still significantly accelerate the convergence in Spaceinvaders. In contrast, the stable regularization is able to boost the sample efficiency mildly on Enduro. For example, when 7j = 1.0 (green line), "+StableReg sample efficient (higher than yellow and orange lines) in the early phase of training. However, the benefit of stable regularization vanishes as the training proceeds, achieving comparable performance with DuelingDQN. Interestingly, if we further add the additional MellowMax operator (red line), the resulting Stable AA approach can accomplish the improvement of performance to a large margin.
more
Impact of MellowMax Operator As exhibited in diagrams in the last of Figure the benefit of MellomMax operator still depends on the game. Particularly, the improvement of WeelowMax operator on SpaceInvaders negligible, where the lines representing "+MelloxMax" overlap subtly with DuclingDQN. Nevertheless, our Stable AA additionally incorporates the stable regularization. achieving remarkable improvement of sample efficiency. In addition, due to the fact that hard to compute the optimal inverse temperature w: in MellowMax, we tune the parameter and report the corresponding results Figure 2 It manifests from the diagram on Enduro game that MellowMax under = 5.0, 10.0 (green and yellow lines) can substantially enhance the final performance. More importantly, under the joint benefits of both the stable regularization and the theoretically pprnncccpeep Mellow operator, our Stable AA DuelingDQN algorithm can simultaneously accelerate the convergence and improve the final performance.
Discussion and Conclusion
Apart from MellowMax, other variants of Softmax operator can also be combined with Anderson acceleration, although their theoretical principles have not been studied. For instance, the competitive

---

performance of the Boltzmann Softmax operator suggests that it is still preferable in certain domains, despite its non-contraction property. We leave the exploration towards more desirable operators as future works. Additionally, the study of our approach on wider variety of Atari games, and implement on more state-of-the art algorithms are expected in the future.
In this paper, we firstly provide deeper insights into the mechanism of Anderson acceleration on the reinforcement learning setting by connecting damped Anderson acceleration with quasi-Newton method and providing the faster convergence results. These theoretical principles about the faster convergence of Anderson acceleration inspire the leverage of MellowMax operator. Combing with stable regulation, the resulting Stable AA strategy is applied in DuelingDQN, which has been further demonstrated to significantly accelerate the convergence and enhance the final performance.
Acknowledgement
We would like to thank the anonymous reviewers for great feedback on the paper. Yingnan Zhao and Ke Sun were supported by the State Scholarship Fund from China Scholarship Coun- cil NN:2020061204000 and No:2020060100880 the Natural Sciences and Engineering Research Council Canada (NSERC). Dr. Kong was also supported by the University of Alberta/Huawei Joint Innovation Collaboration, Huawei Technologies Canada Co., Ltd., and Canada Research Chair in Statistical Learning.
Dr. Jiang and and Dr. Kong were supported by
References
[1] Hengbin An, Xiaowei Jia, and Homer F Walker. Anderson acceleration and application to the three-temperature energy equations. Journal of Computational Physics, 347:1-19, 2017.
[2] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqut Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning. pages 176-185. PMLR, 2017.
[3] Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pages 243-252. PMLR, 2017.
[4] Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert Kappen. Dynamic policy program- ming. The Journal of Machine Learning Research, 13( 3):7777333245 2012.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458, PMLR, 2017.
[6] Sébastien Bubeck. Convex optimization: Algorithms and complexity. arXiv preprint 488 2014.
[7] Claire Evans, Sara Pollock, Leo G Rebholz, and Mengying Xiao. A proof that anderson acceleration improves the convergence rate in linearly converging fixed-point methods (but not in those converging quadratically). SIAM Journal Numerical Analysis, 58(1):788-88 2020.
[8] Haw-ren Fang and Yousef Saad. Two classes of multisecant methods for nonlinear acceleration. Numerical Linear Algebra with Applications, 16(3):197-221, 2009.
[9] Anqi Fu, Junzi Zhang, and Stephen Boyd. Anderson accelerated douglas douglas-rachfort rachford splitting. SIAM Journal on Scientific Computing, 42(6): 2020.
[10] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018
[11] Matthieu Geist and Bruno Scherrer. Anderson acceleration for reinforcement learning. arXiv preprint 1eX9v01809.0955 2018.
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
10

---

[13] Hado Hasselt. Double q-learning.. Advances in Neural Information Processing Systems, 23:2613- 2621, 2010.
[14] Seungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Deepmellow: re- moving the need for a target network in deep q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence, 2019.
[15] Yujun Li, Chengzhuo Ni, Guangzeng Xie, Wenhao Yang, Shuchang Zhou, and Zhihua Zhang. Accelerated value iteration via anderson mixing. 2018.
[16] Timothy P Liilicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1rXiv:1509.029771 2015.
[17] Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning, pages 4424 4434. PMLR, 2019.
[18] Borislav Mavrin, Shangtong Zhang, Hengshuai Yao, and Linglong Kong. Exploration in the face of parametric and intrinsic uncertainties. In Proceedings of the 18th International Conference on Autonomous Agents and MulliAgent Systems, pages 2117-2119, 2019.
[19] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning navigate in complex environments. arXiv preprint arXiv: 1.03673, 2016.
[20] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 533, 2015.
[21] Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, Longbo Huang, and Tie- Yan Liu. Reinforcement learning with dynamic boltzmann softmax updates. International Joint Conference on Artificial Intelligence (IJCAT), 2020.
[22] Yue Peng, Bailin Deng, Juyong Zhang, Fanyu Geng, Wenjie Qin, and Ligang Liu. Anderson acceleration for geometry optimization and physics simulation. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
[23] Martin L Puterman. Markov decision processes. Handbooks in Operations Research and Management Science, 2:331-436 1990.
[24] Martin L Puterman and Shelby L Brumelle. The analytic theory of policy iteration. Dynamic Programming and Its Applications, pages 91-113, 1978.
(25) Wenjie Shi, Shiji Song, Hui Wu, Ya-Chu Hsu, Cheng Wu, and Gao Huang. Regularized anderson acceleration for oof-py deep reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[26] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che. Julian Schrittwieser, Ioannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)44444488 2016.
[27] Zhao Song, Ron Parr, and Lawrence Carin. Revisiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning, pages 5916- 5925. PMLR, 2019.
[28) Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[29] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double gelaarning In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.

---

[30] Homer F Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715-1735, 2011.
[31] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, pages 1995-2003, PMLR, 2016.
[32] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
[33] Shangtong Zhang, Borislav Mavrin, Linglong Kong, Bo Liu, and Hengshuai Yao. Quota: The quantile option architecture for reinforvement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5797-5804, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
2.3.
(b) Did you include complete proofs of all theoretical results? [Yes] See all the proofs in Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See Supple- mentary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited the related works, which we used their code from.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "P979by4ul5", "reviewer": "Reviewer_8vvP", "summary": "Anderson mixing is a heuristic applied to RL to improve sample-efficiency and convergence. This paper delves into the theoretical side of the Anderson mixing that improves the convergence of deep RL algorithms, showing that the Anderson mixing adds an extra contraction. This paper also shows the connection between Anderson mixing and Mellowmax operator, showing more stable performance and fast convergence. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- I think the introduction part is very well-written; background explanation about GSVI, JVI, average-DQN, and the context of using Anderson mixing was helpful. \n- Section 2.1: Anderson acceleration: linear combination of m previous estimates of Q-values, regulated by the damping parameter \\beta_k\n- I think connection between damped Anderson acceleration and quasi-Newton method is nicely presented. \n- Mellowmax operator fulfills the two assumptions made in Assumption 1, 2. I think the non-expansiveness of the operator is already given in the previous work. Can you give a citation on the Proof of Assumption 2 (Asadi et al. 2017) ?\n- Stable regularization: tau^k contains Frobenius norm delta_k and H_k, which is tuned by the parameter \\eta. \n- Stable AA algorithm = stable regularization approach combined with Mellowmax operator. (1) computes delta_k and H_k for the optimization problem to obtain \\tau_k, (2) target value obtained using Mellowmax, (3) updates target networks. \n- Experiments comparing Dueling-DQN and Dueling-DQN-RAA is provided in multiple Atari game domains. Ablation studies comparing the effect of regularization and mellowmax parameters are also given. \n\nI think this paper is a well-written paper with sound theoretical analysis. The connection between Anderson mixing and stable regularization, and their combination with a Mellowmax operator is the main key point of the paper. I think the originality of the paper is somewhat overlapping with the previous work on Anderson mixing and RL (Shi et al. 2019), but the main strength of this paper comes with the analysis. Experimental results in Atari games were solid.  \n\nQuestions\n\nQ1. There are m target networks in Stable AA Dueling-DQN algorithm. However, I couldn't find detailed information on how many target networks the authors used in practice. How does the choice of m affect the performance of the algorithm? How did you select m? How does it affect the computation and memory?\n\nQ2. It seems that the range of Mellowmax inverse temperature parameter \\omega {1,5,10} is somewhat limited. And there doesn't seem to be a big performance change depending on the choice of \\omega in SpaceInvaders and Enduro. Can we argue that the temperature sensitivity is low in this case? could you add more experiments using a very large inv temperature parameter? e.g. \\omega=100, 1000? \n\nCitation suggestions\n\n- I hope you add the following citation to the non-expansiveness proof of the appendix (although it is already cited in this paper) \n\n[1] K. Asadi and M. L. Littman. An alternative softmax operator for reinforcement learning. ICML 2017\n\n- Overestimation alleviation (section 3.2.) of Mellowmax (proof): \n\n[2] S.Kim, K. Asadi, M.L. Littman, G. D. Konidaris. DeepMellow: Removing the Need for a Target Network in Deep Q-learning. IJCAI 2019. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "IZQnqCRtPMJ", "reviewer": "Reviewer_5Yxv", "summary": "The paper provides a mathematical justification for the benefits of Anderson Accelerating in RL by connecting Anderson mixing and quasi-newton method. The paper also propose a stable Anderson Accelerating (stable AA )algorithm with a stable regularization term and a MellowMax operator and evaluate the performance with Dueling DQN on a variety of Atari games environments. Experimental results show that the proposed algorithm improves convergence, stability and final performance compared to vanilla Dueling DQN and Dueling DQN with previous RAA method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " Originality: The idea of introducing Anderson acceleration to the context of RL is not new, but the as the authors mentions, mathematical insights for the benefits needs more study. The authors bridge the gap by introducing a connection between Anderson mixing and quasi-Newton. While the originality is somewhat limited in using AA in DRL, developing a new algorithm combining AA and MellowMax operator which needs better performance is not trivial.  \n\nQuality: The paper is technically sound, and the experimental analysis is fair and supports the main thesis of the paper. The improvements  improves both the learning speed and final performance over Dueling DQN and Dueling DQN with RAA on some Atari games environments. \n\nClarity: The paper is clearly written. The background on Anderson acceleration and the proofs are clearly stated, as well as the motivation and extension to the deep RL domain.  \n\nSignificance: This work proposes a modification to existing off-policy algorithm. The results support the promise of moderate performance gains. The established connection between Anderson mixing and quasi-Newton methods bridge the gap of mathematical analysis in the field. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "cons:\n1. In addition to Dueling DQN, I'm interested in how stableAA will perform when combined with TD3. (I think in the RAA paper, the RAA algorithm has been used with TD3 to evaluate on continuous Mujoco environments.) If we can see similar performance gains on both TD3 and Dueling-DQN comparing to. previous work, the conclusion would be more promising and meaningful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "40"}, "extra_text": {}}, {"review_id": "HWeVjpG402V", "reviewer": "Reviewer_p6FQ", "summary": "This paper looks to characterise the use of Anderson acceleration in Deep Q learning.  Theoretical convergence results are given in tabular settings. A novel algorithm is developed to satisfy the assumptions made in the theory. The algorithm outperforms reasonable baselines and ablation studies.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThe paper provides strong theoretical results in characterising the convergence benefits of Anderson acceleration in tabular settings, and backs them up empirically, with good ablation studies. Assumptions are explained clearly, and the analysis is novel in the context of RL.\n\n# Weaknesses\nAs far as I can tell, the algorithmic contribution here is limited, substituting the traditionally employed max operator for the already well-understood mellowmax operator.\n\nThe result obtained is relatively niche, as use of Anderson mixing is not standard practice in RL.\n\nThe connection between Anderson mixing and quasi-Newton's method has been established, though the application to RL provided here is insightful. \n\nThe tabular setting is restrictive.\n\n# Correctness\n\nThe paper is correct throughout, and the empirical methodology appears to be sound. Three seeds is generally very few for RL settings, though the algorithm seems to perform consistently across those three seeds.\n\n# Clarity \n\nThe paper is easy to read and straightforward, though the presentation of empirical results is somewhat confusing. The difference between the RAA baseline and the ablation studies is not clear to me.\n\n# Relation to Prior Work \nThe paper is well placed in the context of acceleration methods as applied to RL, and makes use of contemproary results from the existing literature on Anderson mixing.\n\n# Reproducability\nThe work seems reasonably reproducible. Hyperparameters are reported.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a reasonable characterisation of the limitations of their work. Societal impacts are not included, though the work doesn't have obvious social implications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a nice mix of theoretical and experimental results. It analyzes Anderson mixing in RL, finding a link with quasi-newton methods and providing a convergence rate. The assumptions of the analysis hold for the MellowMax operator (introduced in published work). Fairly thorough experiments on Atari with ablations provide evidence for the usefulness of stable Anderson acceleration and the mellowmax operator.
Decision: Accept (Poster)