Overcoming The Spectral Bias of Neural Value Approximation
Authors: Ge Yang
Anurag Ajay
Pulkit Agrawal
Venue: ICLR 2022 Poster (2022)
Keywords: spectral bias
neural value approximation
Q learning
reinforcement learning
neural tangent kernels
kernel regression

Abstract: Value approximation using deep neural networks is at the heart of off-policy deep reinforcement learning, and is often the primary module that provides learning signals to the rest of the algorithm.  While multi-layer perceptron networks are universal function approximators, recent works in neural kernel regression suggest the presence of a \textit{spectral bias}, where fitting high-frequency components of the value function requires exponentially more gradient update steps than the low-frequency ones. In this work, we re-examine off-policy reinforcement learning through the lens of kernel regression and propose to overcome such bias via a composite neural tangent kernel. With just a single line-change, our approach, the Fourier feature networks (FFN) produce state-of-the-art performance on challenging continuous control domains with only a fraction of the compute. Faster convergence and better off-policy stability also make it possible to remove the target network without suffering catastrophic divergences, which further reduces TD(0)'s estimation bias on a few tasks. Code and analysis available at https://geyang.github.io/ffn.

Published as a conference paper at ICLR 2022
OVERCOMING THE SPECTRAL BIAS OF NEURAL VALUE APPROXIMATION
Ge Yang" 14, Anurag Ajay*$$ & Pulkit Agrawal 8S
NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI)* Computer Science and Artificial Intelligence Laboratory (CSAIL)*
Improbable AI Lab
Massachusetts Institute Technology
ABSTRACT
Value approximation using deep neural networks is at the heart of off-policy deep reinforcement learning, and is often the primary module that provides learning sig- nals to the rest of the algorithm. While multi-layer perceptron networks are univer- sal function approximators, recent works in neural kernel regression suggest the presence of spectral bias, where fitting high-frequency components of the value function requires exponentially more gradient update steps than the low-frequency ones. this work, we re-examine off-policy reinforcement learning through the lens of kernel regression and propose to overcome such bias via composite neu- ral tangent kernel. With just a single line-change, our approach, the Fourier feature networks (FFN) produce offe-of-theearr performance on challenging continuous control domains with only fraction of the compute. Faster convergence and bet- ter off- off-policy stability also make it possible to remove the target network without suffering catastrophic divergences, which further reduces TD(0)'s 's estimation bias on few tasks. Code and analysis available at https://geyang github. io/ffn.
INTRODUCTION
At the heart of reinforcement learning is the question of how to attribute credits or blame to specific actions that the agent took in the past. This is referred as the credit assignment problem (Minsky, 1961). Correctly assigning credits requires reasoning over remporally-extendee sequences of obser- vation and actions, so that trade-off's can be made, for example, choose a less desirable action at current step in exchange for higher reward in the future. Temporal difference methods such as TD(A) and Watkins' Q-learning (Sutton, 1988; Watkins, 1989) stitch together the immediate rewards local to each state transition, to estimates the discounted sum of rewards over longer horizons. This is an incremental method for dynamic programming (Watkins Dayan, 1992) that successively improves the value estimate at each step, which reduces the computation that would otherwise be needed to plan ahead at decision time.
A key tension in scaling TD learning to more challenging domains is that the state space (and ac- tion space in continuous control problems) can be very large. In Neurogammon (Tesauro, 1991), for example, the complexity of the state space is the scale of O(022)), which prevents one from storing all state-action pairs in a table. Such constraints and the need to generalize to unseen board arrangements prompted Tesauro (1991) to replace the look-up table with an artificial neural network to great effect, achieving master-level play on backgammon. In general, however, adopting neural networks the value approximator introduces learning instability that can sometimes lead the it- erative learning procedure into a divergent regime where errors are amplified at each step (Sutton & Barto, 2018; Bertsekas, 1995; Baird, 1995). A slew of algorithmic fixes have been developed to tackle this problem from different angles, including sampling from a replay buffer (Lin, 1992; Riedmiller, 2005a); using a delayed copy for the bootstrapping target (Mnih et al., 2013); and using ensembles (Hasselt, 2010; Lee et al., 2021).
Despite of the popularity of these techniques, the neural value approximator itself remains un- changed. Popular treatments view these networks as universal function approximators that would
Equal contribution, order determined by rolling a dice. Correspondence (geyang endey Bossi med

---

Published as a conference paper at ICLR 2022
eventually converge to arbitrary target functions given sufficient number of gradient updates and training data. In practice, however, sfae-of-the-an policy algorithms interleave optimization with sampling, which greatly limits the amount of optimization that is accessible. This puts deep reinforcement learning into the same regime as a neural network regularized through early-stopping where the model bias, the amount of optimization available and the sample size interact with each other in intricate ways (Belkin et al., 2018; Nakkiran et al., 2019; Canatar et al., 2021). Therefore to understand off-policy learning with neural value approximators we first need to understand how deep neural networks generalize, and how they converge under the dynamic process of gradient de- cent. Then find a solution, we need figure out way to control the generalization and learning bias, so that deliberate bias-variance trade-offs could be made for better convergence.
Recent efforts in deep learning theory and computer graphics offer new tools for both. Jacot et al. (2018) uncovers a spectral-bias for shallow, infinitel--wide neural networks to favor low-frequency functions during gradient descent (Kawaguchi & Huang, 2019; Bietti & Mairal, 2019; Ronen al., 2019). This result has since been extended to deeper networks finite-width of any "reasonable" architecture via the tensor program (Yang, 2019; 2020; Yang & Littwin, 2021; Yang & Salman, 2019). Our analysis the toy MDP domain (Section 3) shows that value approximator indeed un- derfits the optimal value function, which tends to be complex due to the recursive nature of unrolling the dynamics. Learning from recent successes in the graphics community (Sitzmann et al., 2020; Tancik et al., 2020), we go on to show that we can overcome the spectral-bias by constructing a composite kernel that first lifts the input into random Fourier features (Rahimi Recht, 2008; Yang et al.. 2015). The resulting neural tangent kernel is tunable, hence offering much needed control over how the network interpolates data.
Our main contributions are twofold. First, we show that the (re-)introduucion of the Fourier fea- tures (Konidaris et al., 2011) enables faster convergence on higher frequency components during value approximation, thereby improving the sample efficiency of off-policy reinforcement learn- ing by reducing the amount of computation needed to reach the same performance. Second, our improved neural tangent kernel produce localized generalization akin to Gaussian kemel. This reduces the cross-talk during gradient updates and makes the learning procedure more stable. The combined effect of these two improvements further allows to remove the target network on a few domains, which leads to additional reduction in the value estimation bias with TD(0).
BACKGROUND AND NOTATION
We consider an agent learning act in a Markov Decision Process (MDP), P, where S and A are the state and action spaces, P S x A + S is the transition function, R : S x + R is the reward and p(s) is the initial state distribution. We consider an infi- nite horizon problem with the discount factor y. The goal of the agent is to maximize the ex- pected future discounted return J E [Ei= 7" R(s), de)] by learning policy r(ass) that maps state to distribution over actions. The state-action value function (Q-function) is defined as (s, = E E Y² R(s,, (so, ao) = (s. a)]. The optimal Q* (s, a) is the fixed point of the Bellman optimality operator B*
B (², R(s, a) + > [max [s)),"")].
(I)
The fitted Q iteration family of algorithms (Emsst al., 2003; Riedmiller, 2005a) iteratively finds the (sub) optimal Qg- by recursively applying the gradient update
0' = v/ Vo Qo(s, a) - ('
Let X = Q0(s, - Bº Qo(ss, a) and apply the chain-rule. The update in Equation 2 becomes
8' = 0 2 (s.a.s
x VoQo(s)
(3)
We can approximate the updated Q in function form through its first-order Taylor expansion
(0 a) Qo(s, a) 2n Ber
a'")''
(s(,)aaa)) a') is the neural tangent kernel where the bilinear form KCss a; S', aaa) = V (oQo(s, a) VoQo
(NTK, see Jacot et al. 2018 and Section 4). This procedure be interpreted as producing
2

---

Published as a conference paper at ICLR 2022
A
Nake
Skac
Stat (6.11
State
(a) FQI + MLP@ 400 (b) Sup. MLP@400 (c) Sup. 12-layer
(d) Sup. MLP@@k
(c) FQI 400
Figure 2: Q-value approximation the toy MDP. All baselines collected at 400 epochs unless otherwise mentioned. (a) Fitted Q iteration using a 4-layer MLP with 400 hidden neurons. (b) Supervised by the ground-truth value target(c) larger network that is xx (12-layers). (d) the same 4-layer network, but optimized longer, for 2000 epochs. (c) 4-layer FFN, no target network.
sequence (Qo: Q1, Q2....) using the iterative rule Qi+1 ITB (Q+) starting with Qo(s, a) = R(s,a) a). IT represents projection to functions expressible by the class of function approximators, which reduces to the identity in the tabular case. The ith item in the sequence, Qs, is the projected optimal Q function of derived MDP with a shorter horizon H = i.
Dynamiiss
Value
A MOTIVATING EXAMPLE
Action
Action
1.3
Action 11
10.0
Action II
7.5
We motivate through toy MDP adapted from Dong et al. (2020), and show that neural fitted Q iteration significantly underfits the optimal value function. Despite of the simple-looking reward and forward dynamics, the value function is quite complex. Our spectral analysis in Section 3.1 indicates that such complexity arises from the recursive application of the Bellman operator.
5.0-
0.9
1.0
Figure 1: A Toy MDP with a simple for- ward dynamies and complex value function, adapted from Dong et al. (2020).
Toy MDP Distribution Consider class of toy Markov decision processes M. The state space is defined on the real-line as S R (0 The reward is the identity function. The action space a discrete set with two elements {0, 1}. each corresponds to a distinct forward dynamics that is randomly sampled from the space of piece-wise linear functions with & "kinks." For all of the examples below, we use a fixed number h = 10, and uniformly sample the value of each turning point in this dynamic function between 0 and 1. The result is distribution M that we can sample from
M M MPMppMM
(5)
Curious Failure of Neural Fitted Q Iteration When we apply standard neural fitted itera- tion (FQI, see Riedmiller 2005b) to this toy problem using a simple four-layer multi-layer perceptron (MLP) with 400 hidden units at each layer, we observe that the learned MLP value approximator, produced in Figure 2a, significantly underfits in comparison to the optimal value function. One hy- pothesis is that Q-learning is to blame, and in fact prior works such as Kumar et al. 2021 have argued that underfitting comes from minimizing the TD Ceempora--Differencee error, because bootstrapping resembles self-distillation (Mobahi et al., 2020), which is known to lead to under-paramettrizzaioo in the learned neural network features. To show that Q learning is not to blame, can the same MLP architecture, but this time directly regress towards the (ground truth) optimal Q function via supervised learning. Figure 2b shows that the under-fitting still persists under supervised learn- ing, and increasing the depth to 12 layers (see Figure 2c) fails to fix the problem. This shows an werrpaaameterized network alone is insufficient to reduce under-fitting. Finally, if we increase the number of training iterations from 400 epochs to 2000, the original four-layer MLP attains a good fit. This shows that solely focusing on the expressivity of the neural network, and making it big- ger (Sinha et al., 2020) can be misguided. The class of function that can be approximated depends on how many gradient updates one is allowed to make, number capped by the compute budget.
3.1 SPECTRAL SIGNATURE OF THE TOY MDP
The Bellman optimality operator imprints a non-trivial spectrum on the resulting Q: as it is applied
3

---

Published as a conference paper at ICLR 2022
recursively during fitted Q iteration. We present the evolving spectra marginalized over M in Figure As the effective horizon increases, the value function accrues more mass in the higher-frequency part of the spectrum, which corresponds to less correlation between the values at near-by states, In a second experiment, we fix the hori- zon 200 while increasing the discount factor from 0.1 all the way up to 0.99. We observe similar whitening of the spectrum longer effective recursion depths. In other words, the complexity of the value function comes from the repeated application of the Bellman optimality operator in process that is not dissimilar to an "infin- ity mirror." The spectrum the Bellman operator gets folded into the resulting Q function upon each iterative step. Although our analysis focuses on the state space, the same effect can be intuitively extrapolated to the joint state-action space.
(a) Value Spectrum vs Horizon
1-0000
--000
(b) Value Spectrum vs
Figure 3: The spectrum of the optimal value function noticeably whitens over (a) longer horizon and (b) larger y.
3.2 KERNEL VIEW ON OFF-POLICY DIVERGENCE
Following the formulation of convergence in Dayan (1992); Tsitsiklis (1994); Jaakkola et (1994):
Consider complete metric space S with the norm || ||. An automorphism f on S is a contraction if Va, b S, (lf((a) f(b)] S 7/la bll. Here y E [o, 1) is called the contraction modulus. When 1, f is nonexpansion.
Definition:
Banach fixed-point theorem. Let S be non-empty with contraction mapping f. Then f admits unique fixed-point x" S s.t. f(x") x². Furthermore, Vrg E S,,x" 5 is the limit of the sequence given by Ti+1 = (x;).. a.k.a x² lim Zi.
10000
Without lost of generality, we can discretize the and action space S and A. The NTK becomes the gram matrix K [II Transition data are sampled from a distribution p(s, a). Theorem. (Achiam et al., 2019) Let indices i, j refer to state-action pairs. Suppose that K., and satisfy the conditions:
Vi, 2n/nnii <1,
Vi, (1+1) 7)
EiKiipp S (1-7)KKup)
j
Then, Equation 4 induces contraction on Q in the sup norm, with fixed-point Q* and the TD loss optimization converges with enough optimization steps.
For relatively large 7 (for instance, 7 E (0.99, 0.999)), the above theorem implies that small off- diagonal terms in the NTK matrix are sufficient conditions for convergence.
4 OPECTRAI--BIAS AND NEURAL KERNEL REGRESSION
Consider simple regression problem where we want to learn function f(E; E R. is sample from the dataset. To understand how the output of the network changes w.r.t small perturbations to the parameters, can Taylor expand around
f(E: 0 + 80) f(E:0) xa (V)f(E:0), 80).
(8)
During stochastic gradient descent using training sample & with loss function L, the parameter update is given by the product between the loss derivative L' f(E) and the neural tangent kernel I.C. (Jacot et al., 2018)
50 = - (F(L))K( where ICEE (Vof(E: Vof(E:0)) )
In this distribution, we modify the reward function be sin sin(2 simplify the reward spectrum.
4

---

Published as a conference paper at ICLR 2022
In the infinite-limit with over over-parannnn veer-parameterieed networks, the function remains close to initialization during training (Chizat et al., 2019). The learning dynamics in this "lazy" regime, under an L2 regression loss behaves minimum norm leuat- solution
St -
f*);
(10)
where ft is the function under training at time t and fo is the neural network at initialization. Replac- ing with the Gramian matrix K. between all pairs of the training data, we can re-write Equation in its spectral form K = OAOZ where each entry in the diagonal matrix A is the eigenvalue A >> 0
for the basis function O, in the orthogonal matrix O. Using the identity e OeAO", we can decompose the learning dynamics in Equation into
( - f) -
(11)
The key observation is that the convergence the component O₁, nds, depends exponentially on its eigenvalue X; a.k the absolute error
-fo-f"))
(12)
Multiple works (Rahaman et al., 2019; Shah et al., 2020; Yang & Salman, 2019; Huh et al., 2021) have shown that the NTK spectrum (A,) of a regular ReLU network decays rapidly at higher fre- quency. In particular, Bietti & Mairal (2019) provided bound of (( for the kth spherical harmonics'. Such results have been extended to width networks arbitrary architecture and depth via the tensor programs (Yang, 2019; 2020).
The Gramian matrix form of the NTK also offers an intuitive way to in- spect state-aliasing during gradient descent. This is because the off- diagonal entries corresponds to the similarity between gradient vectors for different state-action pairs. The kernel of an multi-layer perceptron is not stationary when the input is not restricted a hypersphere as in Lee et al. (2017). We can, however, com- pute of popular network architec- tures over (, shown Figure 4. The spectral-bias of the NTK of both ReLU networks and hyperbolic tan-
-04
(a) MLP + ReLU
(b) MLP + Tanh
(c) FFN (Ours)
Figure 4: NTK comparison between (a) MLP with ReLU activation, (b) MI.P with tanh activation, and (c) Fourier fea- ture networks (Ours). The MLP NTK in (a.b) both contain large off-diagonal elements. The addressing by the gradient vectors is not specific each datapoint.
gent networks produce large off-diagonal elements, increasing the chance of divergence.
OVERCOMING THE SPECTRAI--BIAS OF NEURAL VALUE APPROXIMATION
To correct the spectral- ppestral-bias
bias of feed-forward neural network, we construct composite kernel where a random map : Rdd + RDD first "lifis" the input into a randomized harmonics basis. This ex- introduced by Rahimi & Recht (2007) and it allowed the authors to
plicit kernel lifting trick fit complex datasets using a linear machine. The mixing brings high-frequency input signals down lower and more acceptable band for the ReLU network. Data also appears more sparse in the spectral basis, further simplifies learning. To emulate arbitrary shift-invariant kernel K*, Rahimi & Recht (2007) offered procedure that sample directly from the nominal distri- bution given by 'S spectrum I ((C*)
k(x, v) (o(E), (())) ~ =(E) 2(E) where z(E) = Σ 2 and w; ((C)).
(13)
We choose to initialize FFN by sampling the weights from an isotropic multivariate Gaussian with a tunable cutoff frequency b. We modify the normalization scheme from Rahimi & Recht (2007) and divide b with the input dimension d, s.t. similar bandwidth values could be applied across wide
the input restricted to sphere. This bound is loose, and for tighter bound refer Cao et al. (2019).
5

---

Published as a conference paper at ICLR 2022
Algorithm Learned Fourier Features (LFF)
b-16
bed
8
bed
0.5
class AFF(In.Linea:"
-20
def (self, in, out, b.scale);
20
0
20
uuperO...innt...iiin out)
(a) FFN kernel Spectrum (b) NTK Cross-section
init. (se. std=b.scale/in) init. fwwm bias, -1.0, 1.0)
Figure 5: FFN provide direct control over gen- eralization. (a) The kernel spectra. Peak the center is due to windowing effect. Higher band-limit leads to flatter spectra. (b) The cross- section of the kernel at different band-limit.
def forward(self, x):
np.pi super(). (o)
return torch.sin(x)
variety of reinforcement learning problems with drastically different state and action space dimen- sions. We slightly abuse the notion by using by as the bias parameters, and & (without subscript) for the bandwidth
(x)()), sin Σ wiy + bj) where way ~ (0(0, and by ~((TT,,)) (14)
To attain better performance, Rahimi & Recht (2008) learns a weighted sum of these random kitchen sink features, whereas Yang et (2015) adapts the sampled spectral mixture itself through gradient descent. Using the modern deep learning tool chain, we can view the entire network including the random Fourier features as learnable kernel. For small neural networks with limited expressivity, we found that enabling gradient updates on the RFF parameters is important for performance. We refer to this adaptive variant as the learned Fourier features (LFF), and the shallow MLP with an adaptive Fourier feature expansion as the Fourier feature networks (FFN).
Kernel Spectrum, Convergence, and Interpolation The composite kernel of the FFN has larger eigenvalues on the high-frrequecyy harmonics (see Figure 5), which improves the speed at which the network converges. We can tune this kernel spectrum by changing the band-limit parameter b. With larger bandwidth, the interpolation becomes more local. On the toy domain used to motivate this work FFN achieves a perfect fit with just 400 optimization epochs whereas the MLP baseline requires at least two thousand gradient steps (see Figure 2e). The gain even more prominent on more challenging environments in the results section (Quadruped run, see Figure 9). Optimization overhead is key bottleneck in off-policy learning, therefore faster convergence also translates into better sample efficiency, and faster wall-clock time.
Improving Off-policy Stability "cross-talks" between the gradients of near-by states a key factor in off-policy divergence. Such "cross- talk" manifest as similarity between gradient vectors, the measure of which is captured by the Gram ma- trix of the NTK. The Fourier feature network kernel is both localized (Figure 4c) and tunable (Figure 5), offering direct control over the bias- variance trade-off. The improved learning stability allows us to remove the target network on a number of domains while retaining substantial performance. A curious find when the target network is used, the Icamned value approximation still contains large constant offset from the optimal value function (yellow curve in Figure 6). Such offset disappears after we remove the target value network (blue curve).
Neural Fitted Q Iteration
Bias & Variance
7-
Ground Truth
FFN Target)
Approx. Eiror
6-
FEN
2.0-
MIP
1.55
2.14
1.0
0.13
0.0
02
0.4
0.6
MI.F
FFN
PEN
State [0, 1)
Target
TTaget No Target
(a) Learned Q Function
(b) Approximation Error
Figure 6: (a) Comparison of the learned Q function (for ac- tion II). (b) Approximation eπrоr averaged over 10 random seeds. FFN with a target network still contains an offset, whereas removing the target network eliminates such bias.
Visualizing Mountain Car The Mountain Car environment (Moore, 1990; Sutton, 2000) has a simple, two-dimensional state space that can be directly visualized to show the learned value func-

---

Published as a conference paper at ICLR 2022
(a) layer MLP
(b) 12- 12-layer MLP
(c) MLP + tanh
(d) 4-layer FFN
(c) Tabular
Figure Visualizing the learned value function for action 0 in Mountain Car using (a) a four- layer ReLU network, (b) eight-layer network, (c) four-layer network with hyperbolic-tangent activation, which is commonly used in deep reinforcement learning. (d) that of a four-layer FFN. (e) shows the ground-truth optimal value function produced by running tabular value iteration. The two axis corresponds to velocity (horizontal) and position (vertical) of the car.
tion. We compare the value estimate from three baselines In Figure 7: the ground-truth value es- timate acquired using tabular value iteration; one obtained from four-layer ReLU network using fitted Q iteration; and one from the same network, but using a dimensional Fourier features on the input. All networks use 400 latent neurons and are optimized for 2000 epochs.
SCALING UP To COMPLEX CONTROL TASKS
We scale the use of FFN nigh-dimensional continuous control tasks from the DeepMind control suite (Tassa et al., 2020) using soft actor-critic (SAC, Haarnoja et al. 2018) as the base algorithm. Our implementation built off the pytorch codehase from Yarats Kostrikov (2020) which the current state-of-the oatte-arr- state space DMC. Both the actor and the critic have three latent layers in this base implementation. We use FFN for both the actor and the critic by replacing the first layer in the MLP with learned Fourier features (LFF) without increasing the total number of parameters.
We provide extensive empirical analysis on eight common DMC domains and additional results with DDPG in Appendix A.9. Due to space constraints present the result on four representative domains Figure 8. FFN consistently improves over the MLP baseline complex domains while matching the performance in simpler domains. We include additional ablations and hyperparameter sensitivity analysis in Appendix A.7.
Hop
Walleer-run
Quadruped-wwakk
Quadruped-run
ME. PPN
X.C
110
100
200
MLP
FPN
200
MELP
00
0.0
03
0. 10
3.0
8.5
1,3
0.5
Figure 8: Learning curve with FFN applied to SAC on the DeepMind control suite. Domains are ordered by input dimension, showing an overall trend where domains with higher state dimension benefits more from Fourier features. We use (Fourier) feature-input ratio of 40: 1.
Walker-mun
Quadruped-run
Walker-mn
Quadruped-rum
500
500
MiFiimeet Ad thiee -Ct
FFN
300
(bspr10))
FPN
MLP (LPppoop-)
000
42
04
055
15
000
02
9.4
Figure 9: FFN needs only fraction of the com- pute to match the performance of the MLP base- line both Walker-run and Quadruped-run.
Figure 10: Increasing the gradient updates to sampling ratio causes Quadruped-rum to crash. On Walker-run it improves the performance.

---

Published as a conference paper at ICLR 2022
Walker-run (Weights)
Walker-run (Biss)
Quadrupec--tnt (Weights)
Quadropod-run (Bins)
FRIL
Lawil1
NL. Lo
KLL Lavet2
Wegg
00
02
00
0.2
1.0
00
0.5
100
Trams
3arm
Figure 11: Weight and bias changes in FFN and MLP during training, using SAC. While FFN's bias parameters undergo less change than MLP's bias parameters, the results are mixed when it comes to weight parameters the Quadruped environment.
Faster Convergence via Fourier Feature Networks The key benefit of the Fourier feature net- work is that it reduces the computation needed for good approximation. Off-policy reinforcement learning algorithms tend to be bottlenecked by optimization whereas on-policy algorithms tend to be bottlenecked by simulation time. This means with a reduced replay ratio, FFN can achieve faster wall-clock time. Figure 9 shows that we can indeed reduce the update frequency to 1 1/4 in Walker- walk and 1/6 in Quadruped-rum while still matching the performance of the MLP baseline. In control experiment (see Figure 10), we find that increasing the replay ratio causes the MLP crash on Quadruped-run, illustrating the intricate balance one otherwise needs to maintain. FFN brings additional improvements on learning stability besides faster convergence. We additionally visualize the value approximation error in Figure 25. FFN consistently produces smaller value approximation error compare to the MLP.
Hopp
Walker-nu
Quaddruped-walk
Quadropee--rn
MPP
H17
100
43
022
0.0
Figure 12: Value approximation error with FFN vs MLP using SAC. The divergence is especially prominent at the beginning of learning when the data is sparse. FFN reduces the variance in these regimes by making more appropriate biri-variance trade-off than the MLP baseline.
Improved Feature Representation Makes Learning Easier The random Fourier features sample from a Gaussian distribution in the frequency domain. This is a more uniform functional distribution compared to the MLP, which exponentially favors low-frequee fre.uency functions (Bietti & Mairal, 2019). Let W/ and B, refer to the concatenated weight and bias vectors. In this experiment we inspect the (e-norm of the changes of weights and biases w to their initial value, [[W, [WllWoll and |BB-Bolla We include all eight DMC domains in Appendix A. 11 and present the most representative domains in Figure 11. In seven out of eight domains, the first layer in both the MIP and the FFN experience roughly the same amount of change in the weights, but the FFN requires less weight change in the later layers. This indicates that LFF makes learning casier for the rest of the network. Quadruped- run is the only outlier where the LFF experience more weight change than the first layer of the MLP. Our hypothesis is that this is a harder task, and we are either underparameterized in terms of Feature dimension, or there is instribution-mislignnn focused on the weights. The biases change consistently less in the FFN than the MLP.
between the task and the model. So far we have
Hopp
Walker-run
Quadruped-walk
Quadruped-mo
1099
M.PPwerg)) PPN
589
Mrl Paucaa
100
MLP twrtgt
100-
00
022
0.0
0.2
0.4
10
80
0.5
1.5
Figure 13: Learning curves showing that FFN improves learning stability to the extent that learning can happen on some domains even without the target network. On the contrary, MLP consistently fails without target value network. FFN has consistently less variance than the MLP.
8

---

Published as a conference paper at ICLR 2022
Hopper-bop
Cheetal-tun
Quadruped-wall
Quadruped-tun
50
100
200
0.
0.0
02
02
0.
16
Orye
1m5
Figure 14: Control from pixels learning curve on the DeepMind control suite using Fourier-CNN (F-CNN) and DrQv2. We use feature-input ratio of 40 1. Performance with the F-CNN has much lower variance and is consistently at the top of the confidence range of the vanilla CNN.
Fourier Features Improve Off-Policy Stability Without target network, gradient updates through the TD objective causes the regression target itself to change, making the learning procedure less stable. Mnih et al. (2013) replaces the value network on the r.h.s of the Bellman equation with a copy that is updated at a slower rate, so that the TD objective is more stationary. In Figure 13 remove the target network. While the MLP baseline completely fails in all environments, the improved neural tangent kernel of the FFN sufficiently stabilizes Q learning. that only minor perfor- mance losses occur with seven out of cight domains. We offer the complete result in Appendix 12.
Improving the Convolution Kernel The filters in standard convolution neural network suffer the same spectral bias in the RGB and feature dimension as the state and action space above. We can in principle extend our spectral-bias fix to convolution networks by replacing the first layer with a x convolution with a sine non-linearity. In this experiment we build upon the DrQv2 implementation (Yarats et al.. 2021) and replace the CNN in both the actor and the critic with this Fourier-CNy (F-CNN) architecture according to Equation 14. The results indicate that while F- CNN's performance within the variance of CNN's performance, its variance is much lower (see Figure 14). Such technique has also been used by Kingma et al. (2021) to improve image genera- tion with diffusion models. For more detailed theoretical analysis of the CNN NTK, one can refer to Arora et al. (2019) and Li et al. (2019).
7 DISCUSSION
The inspiration of this work comes from our realization that techniques used by the graphics com- munity (Tancik et al., 2020; Sitzmann et al., 2020; Mildenhall et al., 2020) to learn high-fidelity continuous neural representations represent a new way to explicitly control generalization in neural networks. We refer to Achiam et al. (2019)'s 's pioneering work for the theoretical setup on off-policy divergence, which predates many of the recent analysis on spectral-bias in neural networks, and the (re- (re-Jintroduction of the random Fourier features as a solution to correct such learning biases. Functional regularization through gradient conditioning is an alternative to re-parameterizing the network. A recent effort could be found in Piché et al. (2021). A key benefit of reparameterizing the network is speed. We managed to reduce the wall-time by 32% on the challenging Quadruped environment, by reducing the replay ratio to 1/6 of the baseline.
Fourier features for reinforcement learning dates back as early as Konidaris et al. (2011). While this manuscript was under review, few similar publication and preprints came out, all developed inde- pendently. Li & Pathak (2021) focuses on the smoothing effect that Fourier feature networks have in rejecting noise. Our finding is that vanilla feed-forwer neural networks are on the biased-end of the bias-variance trade-off. In other words, we believe the main issue with neural value approx- imation is that the network underfits real signal, as opposed being overfit random noise. The rank of the neural representation describes the portion of the linear space occupied by the largest eigenvalues of the kernel regardless of the spatial frequency of those corresponding cigenfunctions. Therefore lower rank does not correspond to smoother functions. We additionally find that main- taining a Fourier feature to input feature ratio (D/d > 40) is critical to the expressiveness of the network, which allowed us to scale up to Quadruped without needing to concatenating the raw input as a crutch. Brellmann et al. (2022) is a concurrent submission to ours that delightfully includes the random tile encoding scheme from Rahimi & Recht (2007), and on-policy algorithm, PPO. Our intuition is that policy gradient needs to be considered a life-long learning problem, and locality in speed up learning by eliminating the need to repeatedly sample areas the
the policy network policy has experienced. We are excited about these efforts from the community, and urge the reader to visit them for diverse treatments and broader perspectives.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like to thank Leslie Kaelbling for her feedback on the manuscript; Jim Halverson and Dan Roberts at IAIFI for fruitful discussions over neural tangent kernels, and its connection to off- oif-policy divergence; Aviral Kumar at UC Berkeley for bringing into our attention the expressivity issues with off-policy value approximation; and the reviewer and area chair for their kind feedback.
This work is supported in part by the National Science Foundation Institute for Artificial Intel- ligence and Fundamental Interactions (IAIFI, https://iaiff.org org/)) under the Cooperative Agreement PHY-2019786; the Army Research Office under Grant W91 1--N-22----028; and the MIT-IBM Wat- son Lab and Research Collaboration Agreement No. 1771667 The authors also acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing high perfor- mance computing resources. The views and conclusions contained in this document are those of the authors and should not be interpreted
representing the official policies, either expressed im- plied, of the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REPRODUCIBILITY STATEMENT
We include detailed implementation details in Appendix A.4.
REFERENCES
Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q- learning. arXiv preprint arXiv:1903.08894, 2019.
Arora, S Du, W Hu, Z Li, R Salakhutdinov, and others. On exact computation with an infinitely wide neural net. arXiv preprint arXiv, 2019.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Mikhail Belkin, Daniel J Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. undefined, 2018.
Dimitri P Berrtekks. A counterexample to temporal differences learning. Neural Comput.. 7(2): 270-279, March 1995. ISSN 0899-7667, 1530-888X. doi: 10.1162/neco.11955. 1995.7.2.270..
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernlls. Adv. Neural Inf. Process. Syst. 32, 2019. ISSN 1049-5258.
David Brellmann, Goran Frehse, and David Filliat. Fourier features in reinforcement learning with neural networks, 2022. URL https:
wttp:/ewwwww..eeeooonueeeeeeeeee htt.:///ww......oooooooooo/////o jg.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization kernel regression and infinitely wide neural networks. Nat. Commun., 12(1):2914, May 2021. ISSN 2041-1723, doi: 10. 1000008/441467-00122330000
Yuan Cao, Zhiying Fang, Yue Wu, Ding-Dinuun Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. December 2019.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Adv. Neural Inf. Process. Syst., 32, 2019. ISSN 1049-5258.
Peter Dayan. The convergence of td(A) for general x. Mach. Learn., 8(3-4):341-36,, May 1992. ISSN 0885-6125, 1573-0565. doi: 10. 00.000//000992001.
10

---

Published as a conference paper at ICLR 2022
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning, In International Conference on Machine Learning, pp- 2627-2637. PMLR. 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Iteratively extending time horizon reinforcement learning. In Machine Learning: ECML 2003, pp. 96-107. Springer Berlin Heidelberg, 2003. doi: 10.1007/978-3 3-540-39857--8 11.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290 2018.
Hado Hasselt. Double q-learning. volume 23, pp. 2613-2621, 2010.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv: 103. 10427, 2021.
Tommi Jaakkola, Michael I Jordan, and Satinder P Singh. On the convergence stochastic iterative dynamic programming algorithms. Neural computation, 6(6):
1994.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. arXiv preprini arXiv: 07572, 2018.
Kenji Kawaguchi and Jiaoyang Huang. Gradient descent finds global minima for generalizable deep neural networks practical sizes, 2019.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. July 2021.
George Konidaris, Sarah Osentoski, and Philip Thomas. Value function approximation in reinforce- ment learning using the fourier basis. In Twenty-fifth AAAI conference on artificial intelligence, 2011.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parrmmtterizaioon inhibits data-efficien deep reinforcement learning. International Conference on Learning Repre- sentations, 2021.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrry Pennington, and Jascha Sohl-Dickstein. Deep neural networks gaussian processes. arXiv preprint 2017.
Kimin Lee, Michael Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In ICML,, 2021.
Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Adv. Neural Inf. Process. Syst., 34, 2021. ISSN 1049-5258.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kernels. November 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
Long-Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teach- ing. Mach. Learn., 8(3):293-321, May 1992. ISSN 0885-6125, 1573-0565. doi: 10.1007) BF00992699.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Marvin Minsky. Steps toward artificial intelligence. Proc. IRE, 49(1):8-30, January 1961. ISSN 0096-8390. doi: 10.1 1091109//PPRCC http//RRROC.191...000 162777777

---

Published as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33, 2020.
Andrew W. Moore.
Efficient memory-based learning for robot control. 1990.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. December 2019. URL http://
mexiv.org/aee abs 912 02292.
Alexandre Piché, Joseph Marino, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan. Beyond target networks: Improving deep q-learning with functional regularization. June 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. pp- 5301-5310, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Adv. Neural Inf. Process. Syst. 20, 2007. ISSN 1049-5258.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Adv. Neural Inf. Process. Syst., 21, 2008. ISSN 1049-5258.
Martin Riedmiller. Neural fitted q iteration-fiss experiences with data efficient neural reinforce- ment learning method. In European conference machine learning, pp. 317-328. Springer, 2005a.
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural rein- forcement learning method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo (eds.), Machine Learning: ECML 2005, pp. 317-328, Berlin, Heidelberg, 2005b. Springer Berlin Heidelberg. ISBN 978-3-540-31692-3..
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural net- works for learned functions of different frequencies. In H Wallach, H Larochelle, A Beygelzimer, dAlché-Bue, Fox, and R Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07710, 2020.
Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
R Sutton. Mountain car software, December 2000. URL http:://t///tt www..s....bbbaaa.......oo....
html.
Richard S Sutton. Learning to predict by the methods of temporal differences. Mach. Learn., 3(1):
9 44, August 1988. ISSN 0885-6125, 1573-0565. doi: 10.1007/BF001 15009.
Richard S Sutton and Andrew G Barto. Reinforeement Learning: An Introduction. MIT Press, October 2018. ISBN 9780262352703.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. NeurlPS, 2020.
12

---

Published as a conference paper at ICLR 2022
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bo- hez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm.ontrol :
Softwareandtasks orcontimuousconntrol 2020.
Gerald Tesauro. Practical issues in temporal difference learning. Adv. Neural Inf. Process. Syst., 4, 1991. ISSN 1049-5258.
John N Tsitsiklis. Asynchronous stochastic approximation and q-learning. Mach. Learn., 16(3):185- 202, September 1994. ISSN 0885-6125, 1573-0565. doi: N0.1007/BF00993306
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4)2279-29) 1992.
Christopher John Comish Hellaby Watkins. Learning from delayed rewards. 1989.
Greg Yang. Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes. October 2019.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. June 2020.
Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pp- 11762-11772. PMLR, 2021.
Greg Yang and Hadi Salman. A fine-grained spectral perspective on neural networks. arXiv preprint 1rX1v:1907.10599, 2019.
Zichao Yang, Andrew Wilson, Alex Smola, and Le Song. A la carte-learning fast kemels. In Artificial Intelligence and Statistics, pp. 1098-1106, 2015.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation pytorch. https: 2020.
//github. com hot.////aaaaaaarrtttprtrroeaaeaa
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 2021.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A. EXPERIMENTAL DETAILS ON THE TOY MDP
Offiine Data Generation We divided the 1-dimensional state space into 1000 discrete bins and used the midpoints of the bins as initial states. We then took both the actions from these initial states to get corresponding next states, We used the dataset these transitions (2000 total transitions) as offline dataset.
Optimization details We use 4-layer MLP with ReLU Activation, with 400 latent neurons. We use Adam optimization with a learning rate of le-4, and optimize for 400 epochs. We use gradient descent with batch size of 200. For a deeper network, we use 12-layer MLP, with 400 latent neurons but keep other optimization hyperparameters fixed. To show that longer training period help MLPs evade the spectral bias, we use a 4-layer MLP, with 400 latent neurons and train it for 2000 epochs. All the optimization hyperparameters are kept the same.
A.2 EXPERIMENT DETAILS ON MOUNTAIN CAR
Environment We use the implementation from the OpenAI gym (Brockman et al., 2016), and discretize the state space into 150 bins. This is relatively small number because otherwise the space complexity becomes infeasible large. We use a four-layer ML. as the Q function, and bandwidth parameter of 10.
A.3
EXPERIMENTAL SETUP ON DEEPMIND CONTROL SUITE
DMC domains We list the 8 DMC domains along with their observation space dimension and action space dimension in Table We also list the optimal bandwidth used by FFN for each of these envs.
Table 1: DMC domains in increasing order of state space and action space dimensionality
Name
Observation space
Action space
Bandwidth b
Acrobot-swingup
Box(6,)
Box(1.)
0.003
Finger-turn-bard
Box(12.)
Box(2.)
0.001
Hopper-hop
Box(15.)
Box(4,)
0.003
Cheetah-run
Box(17.)
Box(6.)
0.001
Walker-run
Box(24.)
Box(6,)
0.001
Humanoid-run
Box(67.)
Box(21.)
0,001
Quadruped-walk
Box(78.)
(12,)
0.0003
Quadruped-rum
Box(78.)
Box(12.)
0.0001
A.4 ARCHITECTURAL DETAILS
For the state space problems, parameterize the actor and the critic in the MLP baseline with three latent layers. For FFN (ours), we replace the first layer in the MLP a learned Fourier features layer (LFF). If is the input dimension, both MLP and FFN have [40 d, 1024, 1024] hidden dimension for each of their layers. Note that in the actor, d observation dimension whereas the critic, observation dimension actor dimension.
We build upon DrQv2 (Yarats et al., 2021), and derive conv FFN by replacing the first convolutional layer with 1xI convolution using the sine function non-linearity. We the initialization scheme described in Equation 14 for the weights and biases.
A.5 SUMMARY OF VARIOUS FOURIER FEATURES
Using Fourier features to correct the spectral- bias is a general technique that goes beyond a particular parameterization. Hence present comparison between
14

---

Published as a conference paper at ICLR 2022
vanilla MLP is stack of t linear layers with ReLU activation
(L)(x) = =f fo ReLU o.... f ReLU (x
Fourier features network (FFN) (Ours) uses sine activation with random phase-shift, to replace the first layer of the network with learned Fourier features
LFF(z) = sin(Wx + c). Way ~ (O( ~ ((-,,
that the Fourier features network (FFN) is
FFN(2) = f" ReLU LFF(2)
RFF (Tancik et al., 2020) that uses sine and cosine pairs concatenated together
F. (z:
x),, cos(2TWa.) z)], W1. (N
SIREN network (Sitzmann et al., 2020) that stacks learned Fourier layers through the entire network, using the Sitzmann initialization according to
lf(() = sin(Wa + c), Wa.5 U( V/ ((=,,T)
where the t-layer network
SIREN(+) lif' o.... -ff 10 (z).
It is critical to note what each layer has a distinct bandwidth scaling parameter. Hence instead of single scaling hyperparameter b for the random matrices, Siren has set (b;) that each need to be tuned.
We found that is important to use isotropic distributions for the weights. This means using Gaus- sian distributions orthogonal matrices for the weight matrix. Uniform distribution is anisotropic.
To stack LFF (which becomes SIREN), the network requires a "fan-out" in the latent dimensions accommodate the successive increase in the number of features. SIREN is difficult tune be- cause the initialization scheme is incorrect, as it depends on the inverse quadratic root of the input dimension. Correcting the initialization to depend inversely the input dimension fixes such issue.
A.6 EFFECT OF BANDWIDTH PARAMETER ON TOY MDP
The weight distribution in the random Fourier features offers a way to control the generalization (interpolation) in the neural value approximator. In Figure 15, we compare the learned function using Fourier feature networks initialized with different band-limit parameter b. With larger bs. the network recovers sharper details on the optimal Q function. We also provide the result for an FFN with the target network turned off.
Acdia
Accle
Acden
Valu
1.0
16
State
State (5.
State (0.
State
(a) RFF b
(b)
(c)
(d) No target network
Figure 15: Q-value approximation on the toy MDP with different bandlimit parameter b. (a c)
showing b -1133,5 5). (d) showing FFN without target network.
A.7 ABLATIONS AND SENSITIVITY ANALYSES
Sensitivity to bandwidth 6 Since FFN introduces bandwidth hyperparameter b, it is natural to ask how the choice of b affects FFN's performance. Figure 16 shows that FFN's performance does vary with choice of b. Furthermore, the best performing value of b differs with environment. This difference arises from the fact that the optimal value function for different environments have different spectral bias and hence requires different bandwidth à for FFN.
15

---

Published as a conference paper at ICLR 2022
Walker-run
Quadruped-run
500-
Ronuis
600
400-
EFN (3-0.001)
FEN (B-00008)
400
0.000
(B-0.001)
200-
FFN (B-0.0005)
200
FEN (B-0.0003)
(B-000001)
0.6
Frames
Figure 16: Learning curve with FFN for different values of B on Walker-run and Quadruped- We use SAC as the RL algorithm. We observe that different B values lead to different performances and hence, we must choose the B value carefully for each environment.
MLP Actor + FEN Critic Experiments we have presented so far use FFN for both the actor and the critic. Our intuition that the main benefit comes from using FFN in the critic. Results in Figure 17 shows that this is indeed the case using an MLP for the actor does not affect the learning performance, whereas using an MLP for the critic causes the performance to match those of the MLP baseline.
Sensitivity to (Fourier) Feature-Input Ratio In addition to the bandwidth b, we need to choose the Fourier feature dimension D for FFN. We maintained the feature-input ratio D to be 40. But can we get away with a lower feature-input ratio? Figure 18 shows that it is important to maintain feature-input ratio be least 40 and any reduction in the feature-input ratio hurts the performance of FFN.
Walker-run
Quadruped-run
Quadruped-run
100
Rroards
6:0
600-
MLP
400
MUP
MLP FFN (estio-40)
200
FFN (ucerr) (catik)
FFN (acor)
200-
FFN (eatio 30)
(abll)
Figure 17: Using FFN for critic is good using FFN for both actor and critic. However, using learned FFN only for actor is similar the MLP baseline. This indicates the gain mainly comes from better value approximation.
Figure 18: Learning curve with
different Fourier dimension ratio Quadruped-run. Lowering the ratio decreases the performance.
A.8 FULL RESULTS WITH SAC
We include the full set of results on eight DMC domains in Figure 19 using soft actor-critic (Haarnoja al., 2018).
A.9 FULL RESULTS WITH DDPG
The benefits FFN brings generalize beyond soft actor-critic to other RL algorithms. In this section present results based on deep deterministic policy gradient (DDPG, see Lillicrap al. 2015). Figure 20 shows learning curve of the FFN vs the MLP on eight DMC domains.
A. REDUCING VALUE APPROXIMATION ERROR
We include the full result value approximation error eight DMC domains in Figure 21. FFN consistently reduces the value approximation error w.rt the MLP baseline especially at earlier stages of training when insufficient amount of optimization has occurred.
16

---

Published as a conference paper at ICLR 2022
swinbot-swingup
Finger-tunn nurn
Happer-hop
Cheecab.cum
38090 3809 300R 4x00 52000
0.0
0.2
04
05
0.5
0.5
0.5
Walkeer-run
Quadruped-walk
Quadruped-run
300
MLP
PMN
1092
200
MLP
117
200
MIP
1T3
0.0
0.0
0.5
20
85
1.3
0.2
Figure 19: Learning curve with FFN applied to SAC on the DeepMind control suite. Domains are ordered by input dimension, showing an overall trend where domains with higher state dimension benefits more from Fourier features. We use (fourier) feature-input ratio of 40 1.
swinbu
Finger num
Happer hop
PPN
200-
100
200-
0.0
02
05
0.
02
04
60
0.2
04
06
0.
0.5
Quadruped-walk
Quadruped-rum
900
0'0-
FFN
580
77N
0.0
0.2
0.5
0.0
0.5
1.0 Tarms
Figure 20: Learning curve with FFN applied DDPG on the DeepMind control suite. Domains ordered by input dimension. The over trend agrees with results on soft actor-critic, where domains with higher state dimension benefits more from random Fourier features. The same feature-inpus input ratio of 40 is applied.
A.11 MEASURING PARAMETER CHANGES IN FEN AND MLP
We can visualize the amount of changes the FFN and the MLP experience in the weight space. Let W and Bf refer to the flattened weight and bias vectors from layer k in the critic after training over environment frames which corresponds the number of optimization steps times multiple. Figure 22 shows the evolution of [| Woll for all eight domains. Figure 23 shows the evolution of IBB Boll Both collected with soft actor-critic.
A.12 REMOVING THE TARGET NETWORK
We present the full result without the target network in Figure 24. While MLP completely flat-lines in all environments, FFN matches its original performance with target networks in Finger-turn, and Quadruped-walk, It suffers performance loss but still manages to learn in Cheetah-run, Walker-run, and Quadruped-rum Hopper-hop. Humanoid-run, and Acrobat-swing up are difficult for model-free algorithms, and FFN fails to learn without target network.
We can inspect the value approximation error. Without a target network. both MLP and FFN do poorly Hopper-hop, the result this domain is uninformative. In Walker-run, removing the target network reduces the value approximation error even further. In Quadruped walk, FFN without target network perform worse than FFN with target network, and has slightly higher approximation error. Both FFN baselines however have lower approximation errors than the MLP In Quadruped- without the target network, the approximation error diverges large value, then converges back. The source of this divergence remains unclear.
17

---

Published as a conference paper at ICLR 2022
Autobot-swinguup
Finger-tum hard
Hloppe--hop
Checlah-run
019
Porm Hummmood--uun
Feom Quadrupad-watd
Fream
Walker-run
Quadrrpped-run
F7N
MLP
17%
0.
00
0.2
0.3
1.0
00
02
04
Figure 21: Value approximation emror with SAC on the DeepMind control suite. FFN reduces the
emoor w.r. the MIP, especially during earlier stages of learning where the gradient updates is low.
Acrobot-swing swingup (Weights)
Finger-tum hard (Weight)
Hopper-hop (Weights)
Cheetah-run (Weights)
20-
213 Lentta]
EIN Li
PIN Layetta
158
EIN Lap-til
PIN Looet
ITN Lee-tt]
Ler-tta
19.
Lipell]
XLPLapuf1]
Walker-nun (Weights)
Humanoid-num (Weights)
Qundraped-wild walk (Weights)
Quadruped-rum (Weights)
128
FiN Lestll
129
ITN Lazett)
Lemtil
MirLapenti
Figure 22: Weight change in FFN and MLP during training RL control suite. The results mixed and FFN's weight parameters undergo less change than MLP's weight parameters only in some environments.
RL agents with SAC on the DeepMind
Actobot-swingup (Bins)
Fingar-tum bard (Bins
Hopper-hop (Biss)
Chectale-re (Bias)
ITN Lapeli! Lopelhng
Lowdl1
losecha)
Lome
KSP Laye??
105
105
Trene
Trens
Walkeer-ru (Bias)
Humanoid-me (Bias)
Quadruped walk (Bins)
Quadrupad-nm (Bias)
Liscel)
Lomil
MIP MiPlamp
KL
1m5
180
Figure 23: Bias change in FFN and MLP during training of RL agents with SAC on the DeepMind control suite. Given FEN's bias parameters have better initialization, they undergo less change than MLP's bias parameters.
18

---

Published as a conference paper at ICLR 2022
Acrobol-ss
Finger-lur bard
Hopper-lop
Cheetal-run
MLPP
NLP
MLP
MLP (e-gg)
NLL reoeg!
barkk
PPN
IPN
200-
10000 J8003 MooC Soao
03
Walleer-run
Homnanodd-run
Quadruped-waik
Quadrupeu-run
500
MLP
MLP
KLPox4g)
NLP readd"
FIN
ETN Juodg?
taaagn
16.7
ME.7 targg
400
360
20
Figure 24: Since FFN require fewer gradients for value function estimation, its performance doesn't t degrade much when target value networks are removed. On the contrary, MLP completely fails when target value networks are removed.
Hopper-bae hop
Walker-um
Quadruped-walk
Quadruped-run
100
MLP
MI
7.52
1.00
77%
23% mn-go
100
MLP
0.3
0.2
035
0.2
055
Figure 25: Value approximation error with FFN vs MLP using SAC. The divergence is especially prominent at the beginning of learning when the data is sparse. FFN reduces the variance these regimes by making more appropriate bias bias- variance trade-off than the MLP baseline.
19


Reviews:
[{"review_id": "h7-6TRMZu7Z", "reviewer": "Reviewer_TezL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper tackles an important problem and has an exemplary exposition of the problem and the proposed solution.\n- Its results suggest that much more could be done in RL to take into account the peculiarities of the functions being estimated to find good architectures (that may depart from their supervised learning counterparts).\n\nWeaknesses:\n- This particular parameterization is fairly close to prior work\n- The empirical evaluation on \"large scale\" problems is limited\n\nOn the empirical evaluation, I think this is an important point since, as the authors point out, there is no study in this paper of the generalization properties of RFFs. This is particularly important since we know, e.g. from the SIREN paper, that sinusoidal features are particularly good at memorization.\nThis may be a fundamental limitation of generalization/function approximation. How much can we get out of generalization when high-frequency features are involved? Results on this question would be potentially very impactful--see for example Hooker et al. (this paper is very much orthogonal to the present work, no need to cite it, but is relevant to the general question of high-frequency things).\n\nCharacterising Bias in Compressed Models, Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, Emily Denton, 2020.\n\n\nComments:\n- Be careful when using citations, some are missing parentheses around them or just the year. Inline citations where the citation is an object in the sentence (e.g. \"Newton (1687) showed that X\") should use \\citet, whereas parenthesized citations should use \\citep (e.g. \"It has been shown that X (Newton 1687)\".\n- The text legends in Figures 8-11 could be larger\n- An \"ablation\" is when something is removed. It doesn't seem like anything is removed in the first part section 4.3. Rather, different hyperparameters are tested to attempt to understand their impact.\n", "summary_of_the_paper": "This paper proposes a novel parameterization for MLPs, Random Fourier Features (RFF), which amount to using a sinusoidal activation, and initializing parameters to capture different frequencies (by initializing $w_{ij}$ with a variance proportional to a bandwidth hyperparameter $b$) and phases ($b_j \\sim U(-\\pi,\\pi)$).\nThis particular parameterization is motivated by the need, in RL, to learn value functions which have been found to potentially require high frequency components (whereas vanilla MLPs have a low-frequency/simplicity bias). The paper offers a very detailed exposition of why this is the case and of why RFFs can help.\nRFFs are then used on 3 standard control problems, showing some improvement when using DDPG with (vs without) RFFs. The choice of bandwidth as well as the number of features required are tested.", "summary_of_the_review": "The pedagogical aspect of this paper is certainly an appreciable contribution. This paper also contributes to understanding how certain biases of DNNs can be dealt with in deep RL, but only within a limited scope. While it certainly refines our intuition and could help the field focus its research in the right areas, the novelty of the concepts introduced in this paper is fairly limited.\n\nI think that, as it is, this paper meets the bar for acceptance, but its potential seems much higher.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bG3JCmxETX2", "reviewer": "Reviewer_cq8o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Discussion period update: After reading the other reviews (note: no author response), my overall score has not changed.\n\n-This work has limited novelty. The core architecture is basically pulled from existing work. One difference is a change in weight initialization, but the initialization schemes are not directly compared within the experiments. Additionally, if I understand correctly, both schemes still require a hyperparameter (with appropriate choice of hyperparameter, the schemes are the same). The other difference is that this work focuses on DRL.\n\n-The baselines include a decent set of non-Fourier baselines. However, this work does not compare to architectures from Sitzmann 2020 and Tancik 2020 (though these works are cited). As a minor comment, including \"dense+deep\" and \"wide\" would be better, as would including more discussion on the number of parameters vs depth of different baselines.\n\n-This work includes claims about \"evading the spectral bias\" and, in some sections, uses performance to \"show\" the efficacy of the proposed method. However, improved performance does not necessarily mean the bias issue has been reduced.\n\nOther comments:\n\n-The MLP example (paragraph 2) in the introduction is presented in an unclear way. Insufficient information is provided to show that the training process was not the reason for poor performance.\n\n-The third paragraph of the introduction is better placed in a later section.\n\n-For Figure 2 and Figure 7, showing the learned function does not convey the difference between the true and learned functions. Showing the absolute difference between the two would be more informative.\n\n-Much of Section 2 (\"Preliminaries\") is not relevant to this work's contributions. This section should be substantially condensed.\n\n-Reorganizing Section 3.1 would improve clarity. Additionally, the first paragraph largely repeats content from earlier in the paper.\n\n-The Mountain Car experiment is better placed elsewhere in the text.\n\n-If I understand correctly, when varying bandwidth, the number of parameters in the network also changes. This should be considered when performing the ablation studies. Additionally, the SIREN-style selection of width should be explicitly compared to the proposed method.\n\n-Related Work is best placed earlier in the paper.\n\nThis work would benefit from another editing pass:\n\n-\"Given that a more gradient updates\" -> \"Given that more gradient updates\"\n\n-\"as pointed in Fu et al.\" -> \"as pointed out in Fu et al.\"\n\n-\"is called the spectral norm\" -> \"is the spectral norm\"\n\n-\"in figure 4\" -> \"in Figure 4\"\n\n-\"Xiavier-like\" -> \"Xavier-like\"\n\n-\"deepr reinforcement learning\" -> \"deep reinforcement learning\"\n\n-\"In figure 7e\" -> \"In Figure 7\"\n\n-\"its performance vary a lot\" -> \"its performance varies a lot\"", "summary_of_the_paper": "This work proposes Random Fourier Networks (RFNs), MLPs with random Fourier featurization within the first layer. RFNs are supposed to learn high-frequency components of the value function more quickly than standard MLPs. The authors present an initialization scheme for the Fourier layer that is better-suited for high-dimensional inputs. RFNs are compared to non-Fourier-feature baselines on high-dimensional environments.", "summary_of_the_review": "This work overstates its novelty. Using sinusoidal activations on RL tasks is of limited novelty, and no comparisons are made to related methods / few changes are made that are specific to RL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z9Owirm7ReY", "reviewer": "Reviewer_fVzS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe motivation is clear and sound. The problem discussed is well-defined and the introduction to the problem makes sense.\n\nThe proposed method RFN was shown to be effective on both toy tasks and high-dimensional robotic control tasks with DDPG. In particular, RFN +DDPG show a significant performance gain in the challenging humanoid control task. Also, RFN can theoretically be applied to any neural network for function approximation.\n\nConcerns / Questions:\n\nIt is impressive to see DDPG+RFN works on Humanoid in contrary with the original DDPG’s terrible performance. However, we also know that more advance algorithms like SAC or TD3 can also get much better performance than DDPG (similar to RFAC) without resolving the spectral bias issue. Does this mean that spectral bias is not critical to performance if other aspects of the RL algorithm are good? \n\nWhile the empirical results are good, the novelty is a bit weak. My understanding is that the authors borrowed the RFN from recent studies and made an investigation of the hyper-parameters so as to fit in RL. Please correct me if I have any misunderstanding.\n\nFor the RFF layer, it was written that “We find this initialization scheme work well across all domains, with minimum requirement on tuning, even when stacked as learnable units.”. I am a bit confused here. Are the weights and bias of RFF layer learnable or not in your implementations?\n\nThe format of references is not professional nor consistent. Some of them has hyperlinks and some not. Also, Please carefully confirm the published venue of each citation. For example, the Adam paper was published on ICLR, please do not use the arxiv version. \n\nFor the MuJoCo experiments, I would like to see RFAC’s performance on other tasks such as hopper, walker2D, etc., if possible. These results may give people more understanding of RFAC’s effectiveness on (relatively) simpler and lower-dimensional control tasks.  \n\nFor the performance curves (figure 8-12). How many random seeds were tested? What was the shaded area (STD of S.E.M.?) In figure 9, the authors claimed that “its performance varies on Humanoid with different values of b”. But from the plot I cannot see clear statistically significant difference.\n\n\nSmaller problems:\n1.\tThe abbreviation TD needs to be explained at the first usage.\n2.\t\"Note that DDPGtakes 1 gradient step\" -> DDPG takes\n3.\tIt is a bit weird to put Related Works in the end. How about putting it before the Discussion?\n4.\tComma and period should be properly used in the equations.\n", "summary_of_the_paper": "The paper discussed the problem of spectral bias in MLPs (i.e., dependence on the higher-frequency part of input feature is under-fitted) within RL domain. The authors first described the problem with toy examples, which clearly explained this problem. Then it was suggested to use random Fourier feature network (RFN) to transform the input to frequency domain and then feed to the network, to resolve this issue. \n\nExperiments demonstrated effectiveness of RFN to learn a complicated Q-function in low-dimensional state space. And it was also shown that RFN contribute to performance gain with DDPG on high-dimensional robotic tasks. The authors also did an “ablation study” (I would rather call it “sensitivity analysis” instead) to empirically show how the performance depends on the choice of hyper-parameters.\n", "summary_of_the_review": "The paper discussed an important problem in RL that was previously under-investigated. Overall, I think the paper successfully addressed it by suggesting the usage of a random Fourier feature network. However, the novelty is limited since the main contribution borrows the methodology in existing papers, together with the aforementioned concerns, I am not convinced that the paper meets the acceptance criteria.\n\n### Post-rebuttal\nI really appreciate the authors' work for providing new results on DMC tasks. The empirical results are much more convincing and comprehensive now. I would like to raised up my score because I think the current results have successfully supported the motivations. However, now the only and the largest concern is that necessity of a new round of review for such a major revision (unfortunately this is not possible now). After all, I set my recommendations to weak accept since I cannot (and should not) access the newest revised version", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper points out an interesting and, to me unexpected, problem when learning Q-functions to do with spectral bias. Figures 1 and 2 are quite striking. The diagnosis and proposed solution elegantly combines ideas from NTKs and NeRFs. The proposed random Fourier actor-critic performs well in practice. The main problem reviewers had in the end is that the authors added substantial new empirical results too late to review thoroughly.
Decision: Accept (Poster)