Functional Regularization for Reinforcement Learning via Learned Fourier Features
Authors: Alexander Cong Li
Deepak Pathak
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
deep learning
neural tangent kernel
regularization
architecture

Abstract: We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL. We perform infinite-width analysis of our architecture using the Neural Tangent Kernel and theoretically show that tuning the initial variance of the Fourier basis is equivalent to functional regularization of the learned deep network. That is, these learned Fourier features allow for adjusting the degree to which networks underfit or overfit different frequencies in the training data, and hence provide a controlled mechanism to improve the stability and performance of RL optimization. Empirically, this allows us to prioritize learning low-frequency functions and speed up learning by reducing networks' susceptibility to noise in the optimization process, such as during Bellman updates. Experiments on standard state-based and image-based RL benchmarks show clear benefits of our architecture over the baselines.

Functional Regularization for Reinforcement Learning via Learned Fourier Features
Alexander C. Li Carnegie Mellon University al alexanderl 1@cmu. edu
Deepak Pathak Carnegie Mellon University dpathak0cs. cmu. edu
Abstract
We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL. We perform infinite-width analysis of our architecture using the Neural Tangent Kernel and theoretically show that tuning the initial variance of the Fourier basis is equivalent to functional regularization of the learned deep network. That is, these learned Fourier features allow for adjusting the degree to which networks underfit or overfit different frequencies the training data, and hence provide controlled mechanism to improve the stability and performance RI. optimization. Empirically, this allows us prioritize learning low- low-frequency functions and speed up learning by reducing networks' susceptibility to noise in the optimization process, such as during Bellman updates. Experiments standard state-based and image-based RL benchmarks show clear benefits of our architecture over the baselines!.
Introduction
Most popular deep reinforcement learning (RL) approaches estimate either value Q-value function under the agent's learned policy. These functions map points in the state or state-action space to expected returns, and provide crucial information that is used for improving the policy. However, optimizing these functions can be difficult, since there are no ground- -ouud truth labels to predict. Instead, they trained through bootstrapping: the networks are updated towards target values calculated with the same networks heing optimized. These updates introduce noise that accumulates over repeated iterations of bootstrapping, which result in highly inaccurate value Q-value estimates [44, 46). As result, these RI. algorithms may suffer from lower asymptotic performance or sample efficiency.
Most prior work has focused making the estimation of target values more accurate. Some examples include double Q-learning for unbiased target values [16, 47], or reducing the reliance the bootstrapped Q-values by calculating multi-step returns with TD(A) [41]. However, it is impossible to hope that the noise in the target values estimated via bootstrapping will go to zero, because cannot estimate the true expectation over infinite rollouts in practical setups. Hence, argue that it is equally important to also regularize the function (in this case, the deep Q-network) that is fitting these noisy target values.
Conventional regularization methods supervised learning can be associated with drawbacks in RL. Stochastic methods like dropout [40] introduce more noise into the process, which is tolerable when ground-truth labels are present (in supervised learning) but counterproductive when bootstrapping. An alternative approach is early stopping [50], which hurts sample efficiency in reinforcement learning because a minimum number of gradient steps is required to propagate value information backwards to early states. Finally, penalty- methods like L₃ [45] and L2 regularization [17, 22] can help in
Code available
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input
Output
Controlling smoothing with learned Fourier features
sin (x)
d 0.01
0.1
4
3-
train points
x
cxs(2nBB)
-1.0 -0.5
0.0
0.5
1.0
Figure I: Left: the proposed learned Fourier feature (LFF) architecture. B is a matrix trained through backpropagation, and is used to create learned set of Fourier features. The network then passes the Fourier features through alternating linear layers and ReLU nonlinearities, as is done vanilla MLPs. Right: tuning the initialization variance of Bij (0 o²) controls the rate at which target frequencies are learned. Higher a fits higher frequencies faster, while lower a smooths out noise. This architecture be used as functional regularization for Q-function S x A R, value function V S > R. policy π S IRd)i((A), or model T : x A
RL [26]. but regularizing the network in weight space does not disentangle noise from reward signal and could make it difficult to learn the true Q-function. This leads to ask: what is the right way to regularize the RL bootstrapping process?
We suggest that the impact of target value noise can be better reduced by frequency-based functional regularization: direct control over the frequencies that the network tends to learn first. If the target noise consists of higher frequencies than the true target values, discouraging high-frequency learning can help networks efficiently learn the underlying Q-function while fitting minimal amounts noise. In this work, we propose architecture that achieves this by encoding the inputs with learned Fourier features, which we abbreviate as LFF. In contrast to using fixed Fourier features [34, 42], we train the Fourier features, which helps them find appropriate basis even in high dimensional domains. We analyze our architecture using the Neural Tangent Kernel [18] and theoretically show that tuning the initial variance of the Fourier basis controls the rate at which networks fit different frequencies in the training data. Thus, LFF's initial variance provides a controlled mechanism to improve the stability and performance of RL optimization (see Figure 1). Tuned prioritize learning low frequencies, LFF filters out bootstrapping noise while learning the underlying Q-function.
funccition
We evaluate LFF, which only requires changing a few lines of code, state-space and image-space DeepMind Control Suite environments [43]. We find that LFF produces moderate gains in sample efficiency
based RL and dramatic gains image-basct RL. In addition, empirically demonstrate that LFF makes the value function bootstrapping stable even in absence target networks, and confirm that most of LFF's benefit comes through regularizing the Q-network. Finally, we provide a thorough ablation of our architectural design choices.
Preliminaries
The reinforcement learning objective is to solve Markov Decision Process (MDP), which is defined as a tuple AS R.P.R.,, 7). . S and A denote the state and action spaces. P(s'|s, a) is the transition function, R(s, a) the reward function, and aim to find an optimal policy -² (als) that maximizes the expected sum of discounted rewards. Q-learning finds the optimal policy by first learning a function Q' (s, such that (* a) a.- [[ss, a) +y max. (s' (s',a')] The optimal policy is then x (a|s) arg max, (s, a). To find Q", we repeatedly perform Bellman updates Q. which uses the Q-network itself bootstrap target values on observed transitions as,,, [6]. The most basic way to estimate the target values is target
[0, 1] is the discount factor. We
+ max. Q(s', a²), but popular line of work in RL aims to find more accurate target value estimation methods [12, 16, 32, 41, 47]. Once we have these target value estimates, we update our Q-network's 's parameters via gradient descent on temporal difference emror: AO = V, (s((ss0 a) targel)². Our focus on using the LFF architecture prevent Q-networks from fitting irreducible, high- high-frreuency noise in the target values during these updates.

---

True Q-function
Noisy targets
MLP
LEE, 0.003
Figure 2: Filtering noise with LFF input embedding. The bootstrapped targets in RL are a mix between signal and noise. Right side: fit different networks these noisy targets and display their predictions. While the MLP overfits, LFF learns the Q-function and ignores almost all the noise.
3 Reinforcement Learning with Learned Fourier Features
We present a visualization the noisy target value problem in Figure 2. The target value estimates can be noisy due stochastic transitions, replay buffer sampling, or unintended generalization across state-action pairs due to function approximation [48]. We simulate this in Figure by adding noise the optimal Q-function of small gridworld. MIPs susceptible fitting the noise, resulting in inaccurate Q-values that could diverge after repeated bootstrapping. contrast, our LFF architecture controls how quickly low- and high-frequency signals are learned. Tuned properly, LFF filters out the noise and learns the ground truth Q-function almost perfectly (Figure 2, right).
The problem is that MLPs provide control over how quickly they learn signals of different frequencies. Prior work in computer vision found this a problem when MLPs blurred desired high frequencies in low-dinensional (3-5 dimensions) graphics regression problems [28, 42]. They fixed this blurring problem by transforming the input using a random Fourier feature embedding embedding ((x) = sin 22m Bx)]] cos(2m Br.), where B is a disuriee//2 x dingui matrix, || denotes concatenation, and sin and act elementwise. The embedding ((x) directly provides mix of low- and high-frequue freuffrry functions a MLP fa can use to learn a desired function. Tancik et al. [42] use this to improve fidelity in coordinate-based graphies problems.
of the input [34, 42]. Specifically, the idea is to map low-dimensional input
(2m
Intuitively, the row vectors are responsible for capturing desired frequencies of the data. If they capture only low frequencies, then the MLP will be biased towards only learning the low- ow-frequency signals in the data. Conversely, the Fourier features capture sufficient high-frequence features, then MLP can fit high frequency functions by computing simple nonlinear combinations of the features. In these -iwdimensional graphics problems, initializing fixed entries Bij ~ N (0, o²) with large o was enough learn desired high frequency functions; training B did not improve performance.
In the following section, we propose our learned Fourier feature architecture for deep RL, which allows practitioners to tune the range of frequencies that the network should be biased towards learning. We propose several key enhancements that help Fourier features learn in high-dimensional environments. Our work uses learned Fourier features to improve deep RL, in contrast to prior work focused on simple environments with fixed, hand-designed Fourier features and linear function approximation [20. 21]. Although we focus on prioritizing low-freque wwwfrequency signals to reduce bootstrap noise, we also present cases in Appendix F.1 where biasing networks towards nigh-frequency learning with learned Fourier features enables fast convergence and high asymptotic performance in RL.
3.1 Learned Fourier Feature Architecture
Standard MLPs can be written as the repeated, alternating composition of affine transformations L,(z) = W.x + b, and nonlinearity 7, which is usually the ReLU T(x) m))
fo(x) T.,, I,
I1 [,
(1)
We propose a novel architecture based on Fourier features, shown in Figure 1. We define a new layer:
FB sin(2rrz))
(2)
where B is a drourier/2 dinput matrix and || denotes concatenation. drouricr is a hyperparameter that controls the number of Fourier features can learn; increasing drourier increases the degree to which the model relies on the Fourier features. Following Tancik et al. [42], we initialize the entries Bi N o0, where 2 is hyperparameter. Contrary prior work, B is trainable parameter.

---

The resulting LFF MLP can be written:
Algorithm LFF PyTorch-like pseudocode.
fo = Ln T o L1 o Fz(2)
(3)
class LFF():
def ...iiit. (self, input_eize, output__izze size, n_hidden=1, Hidden_din-221 aigna=1.0, r_4in-256):
We can optimize this the same way we optimize a standard MLP, e.g. regression would be:
create
b_chape (input__ciz, i_din self (normal (xeroa (Haao(() sigus (eeepbb))))))))
arg min 0.8
1 O... 0...0 (.((ss y:)² (4)
create rest of netuork
melf.mpp MLP(In_din==f_dd inpat_
out, dit.dins-outpuu..e size,
We propose two key improvements to random Fourier feature imput embeddings [34, 42]: train- ing B and concatenating the input x to the Fourier features. We hypothesize that these changes help preserve information in high- dimensional RL problems, where is increas- ingly unlikely that randomly initialized B pro- duces Fourier features well-suited for the task. Training B alleviates this problem by allowing the network to discover them on its own. Appendix G shows that training B does change its values, but its variance remains close the initial o². This indicates that is an important knob that controls the network behavior throughout training. Concatenating the Fourier features is another key improvement for high dimensional settings. preserves all the input information, which has been shown to help in RL [38]. We further analyze these improvements in Section 6.4.
a_hiddee-a
def forvard(aelf, x):
proj (2 np.pt) matmal self.B) (ai((pii()))f)) coe(pro)), x], dim--1) return self .mp. forward(f)
norual: sample from Gaussian with specified mean and std dev;
natmul: matrix multiplication; cat: concatenation.
Theoretical Analysis
By providing periodic features (which are controlled at initialization by the variance g²), LFF biases the network towards fitting desired range of frequencies. In this section, we hope to understand why the LFF architecture provably controls the rate at which various frequencies are learned. While Tancik et al. [42] analyze different weightings of full set of Fourier basis frequencies, we examine the effect of the initialization variance of randomly initialized Gaussian Fourier features.
We draw upon linear neural network approximations in the infinite width limit, which is known as the neural tangent kernel approach [18]. This approximation allows us to understand the training dynamics of the learned neural network output function. While NTK analysis has been found to diverge from real-world behavior in certain cases, particularly for deeper convolutional neural networks [2. 10], it has also been remarkably accurate in predicting directional phenomena [4, 42]. We provide background on the NTK in Section 4.1, and discuss the connection between the eigenvalues of the NTK kernel matrix and the rate at which different frequencies are learned in Section 4.2. We then analyze the NTK and frequency learning rate for Fourier features in networks with layers (Section 4.3) or more (Section 4.4).
4.1 Neural Tangent Kernel
We can approximate a neural network using first-order Taylor expansion around its initialization tg:
fa()) = (x) + (z) (0 00)
(5)
The Neural Tangent Kernel [18] line of analysis makes two further assumptions: fo is an infinitely wide neural network, and it is trained via gradient flow. Under the first condition, trained network stays very close to its initialization, so the Taylor approximation is good (the so-called "lazy training" regime [8]). Furthermore, fo, (x) 0 in the infinite width limit, so neural network is simply linear model over features Φ(x) = V oo (x)) This gives rise to the kernel function:
2,) (Vofe, (I;), Vofo. (zz))
(6)
The kernel function & is similarity function: if A: Iz)) is large, then the predictions fo (x:;) and fo(r;) will tend to be close. This kernel function is deterministic and does not change over the course of training, due to the infinite width assumption. If we have n training points (zz, y1). & defines
PSD kernel matrix K E pax= where each entry Ki k(I,, When train this infinite width

---

Infinite Karnel Functions,
Effect kix.x)
Fourier Spectrum
w65th=1024
2.0
p-6.1
1.-
2-3
NTK Imit
o=1
Nomslged Eggmaatee
d 10
1.5
1.5
0.8
d 100
0= 30
0.6-
10.
1.0-
0.5
0.5
0.2
0.0
-1
-0.5
e
0.
0.5
1
-1.7
-0.5
D.T
0.5
-0.3
0 0.5
1
Anglo otsect e of
Angle offict & ofx
Elgenvector Froquency M
(a) Quality of NTK approximation
(b) NTK kernel function
(c) Eigenvalue spectrum
Figure 3: NTK of the 2-layer Fourier feature model. We plot the NTK k(r, 0) for points x that lie on the unit circle. In (a) and (b), 0 [-,, л| denotes the offset from reference point with 0g O. Note that our NTK is shift invariant, so these figures are valid for any reference point On- Left: we compare the NTK infinite-width limit the kernel function of 10 randomly initialized 2-layer Fourier feature networks with width 1024. The NTK limit is quite accurate for realistically wide networks. Middle: NTK kernel function & for varying settings of the Fourier feature variance o². Larger enables sharp. local learning, while smaller a induces smoother function learning. Right: The y-axis shows eigenvalues of the NTK kernel matrix K, and the x-axis indicates the frequency of the corresponding eigenvector. High frequencies are not fit with low since their eigenvalues vanish. Increasing increases the higher frequencies' eigenvalues, so they can be learned faster.
neural network with gradient flow on the squared error, we precisely know the model output at any point in training. At time t, we have training residual:
fo. (x) y gKt (fo (z)
where fo, (x) is the column vector of model predictions for all z;, and is the column vector of stacked training labels (see Appendix A.1 for proof sketch). Eq. is critical because it describes how different components of the training loss decrease over time. Section 4.2 will build on this result and examine the training residual in the eigenbasis of the kernel matrix K. This analysis will reveal that each frequency present in the labels will be learned at its own rate, determined by K's eigenvalues.
4.2 Eigenvalues of the LFF NTK
Consider applying the eigendecomposition of K = QAQ" to Eq. 7, noting that e Qe- the matrix exponential is defined exX = Ek=o X Q and Q" will repeatedly cancel in the middle of x since Q is unitary.
(fo, y) e nA (fo, (x) -)
(8)
Note that the ith component of the residual (So, (x) y) decreases with rate e -
Consider the scenario where the training inputs x; are evenly spaced on the d-dimensional sphere gd-1. When & is isotropic, which is true for most networks whose weights are sampled isotropically. the kernel matrix K is circulant (each row is a shifted version of the row above). In this special case, K's eigenvectors correspond to frequencies from the discrete Fourier transform (DFT), and the corresponding eigenvalues are the DFT values of the first row of K (see Appendix A.2). Combining this fact with Eq. 8, where we looked at the residual in K's eigenbasis, shows that each frequency in the targets is learned at its own rate, determined by the eigenvalues of K. For ReLU MIP, these eigenvalues decay approximately quadratically with the frequency [4]. This decay rate is slow, so MLPs often fit undesirable medium and high frequency signals. We hypothesize that LFF controls the We examine LFF's s kernel matrix eigenvalues in Sections 4.3 and 4.4 and verify this hypothesis.
learning rate by tuning the kernel matrix K's eigenvalues.
4.3 NTK Analysis of 2-layer Network with Fourier Features
To simplify our LFF NTK analysis, we consider a two layer neural network f .dd - R:
2 W
sin Bx)
f(x) = V
cos Br)
5

---

Fourier Spectrum of the NTK
Finite Infinite Width Fourier Spectrum
Normallzze Eigenvalus
10
MLP
Normalized Eigenvalue
10
n= 0.01
0.01
0.1
0
0.1
1
10
10-
10-
7/2
12/2
x/2
Eigenvector Frequency w
Eigenvector Frequency w
Figure 4: Left: we compare the NTK eigenvalue spectrum (which determines the frequency-specifi, learning rate) of deep networks with Fourier features to those of a vanilla MLP. Right: we initialize finite-width LFF networks with hidden layers of 1024 units each and compare their kernels (dashed) to the corresponding NTK infinite-width limit (solid). We find that the NTK is accurate (note the log scale) and that decreasing indeed results in lower convergence rates for higher frequencies.
where each row of B a vector by_ E glxd, and there are m rows of B. W ~ (N, 1) and By (0, d²), where is hyperparameter. Note that concatenating x is omitted for this two-layer model. This is because any contribution from concatenation goes to zero as we increase the layer width m. Lemma determines analytical expression for the LFF kernel function k(x, x¹).
Lemma 1. For x. x' E 5d-1 with angle = cos cos- (xr T x'), we have the NTK kernel function::
r x' x'1 2
0² |1 - I'113
((x,x') =
exp
(10)
Proof: see Appendix A.3. This closed form expression for k( elucidates several desirable properties of Fourier features. a directly controls the rate of the exponential decay of k, which is the similarity function for points x. and x'. For large a, k(æ, x²) rapidly goes to apart, so their labels only affect the learned function output in a small local neighborhood. This intuitively corresponds to high-frequency learning. In contrast, small ensures A(2,, a²) is large, even when x and æ' are relatively far apart. This induces smoothing behavior, inhibiting srequnryy learning. We plot the NTK for varying levels of a in Figure 3(b) and show that directly controls the frequency learning speed in Figure 3(c). Figure 3(a) also verifies that the NTK limit closely matches the empirical behavior of realistically sized networks initialization.
and x' get farther
4.4 NTK of Deeper Networks
Figure 3(c) shows that larger initialization variance o² corresponds to larger eigenvalues for high frequencies in the 2-layer model. This matches empirical results that small a leads to underfitting and large a leads overfitting [42]. However, Figure 3(c) indicates that only extremely large a, on the order of 10 103, result in coverage of the high frequencies. This contradicts Tancik et al. [42], who fit fine- mne-griindd image details with o E [1, 10). We suggest that the 2-layer model, even though it accurately predicts the directional effects of increasing or decreasing a,, fails to accurately model learning in realistically sized networks. Manually computing the kernel functions of deeper MLPs with Fourier feature input embeddings is difficult. Thus, we turn to Neural Tangents [30], library that can compute the kernel function of any architecture expressible in its API.
We initialize random Fourier features of size 1024 with different variances o² and build an infinite- width ReLU MLP on top with 3 hidden layers using the Neural Tangents library. As in Figure 3, we take input data x evenly spaced on the 2D unit circle and evaluate the corresponding kernel function (xx,)) between the point (1, 0) and the point x = (cos 8 sin 0). Figure shows the eigenvalues of Fourier features and vanilla MLPs in this scenario. We see the same trend, where increasing a leads to larger eigenvalues for higher frequencies, as we saw in Figure 3. Furthermore, Figure shows that this trend also holds for the exact finite-width architectures that we use in our experiments (Section 6). These results now reflect the empirical behavior where a E [1, 10) results in high frequency learning. This indicates that deeper networks are crucial for understanding the behavior of Fourier features.

---

Chestah
Norobot
Feger Tur (hwrd)
Walkey
131
500
Anras Ratur
700
Aotagg Rature
2000
200-
0.50
0.25
0,30
0.26
100
0.10
0.25
0.30
0.76
1.00
0.0
0.2
0.4
3.10
0.35
0.53
0.76
1.0
Step
Stepo
Stepe
Steps
Quacmuped Run
reped
Hopper Hop
Humonood
Mataage fetum
Autugg fetmm
000
0.35
1.5
Sheps
622
622
Steps
X0F
MPP-sspectal
MLP P-dropouttt
MLP+WD
MLP+FR
MLP
LFF (aurs)
Figure 5: Off-policy State-based Evaluation: Soft Actor Critic (SAC) experiments DM Control environments. We emphasize that these results are produced using the same hyperparameters (e.g. leaming rate, Polyak averaging parameter, and batch size) tuned for MLPs. These results show that plugging in our LFF architecture can yield more sample-efficient learning on most environments.
5 Experimental Setup
We treat the learned Fourier feature network as a drop-in replacement for MLP and CNN architectures. We show that just adding Fourier features improves the performance of existing offte-of-theear methods on existing standard benchmark environments from DeepMind Control Suite [43]. We will release the code, which involves only changing a few lines of code in existing RL algorithms.
State-based LFF Architecture Setup We use soft actor-critic (SAC), an entropy-regularizzed off- policy RL algorithm [14], to learn 8 environments from the DeepMind Control Suite [43]. We keep the default hyperparameters fixed, varying only the architecture for the policy and Q-function. Our LFF architecture uses our learnable Fourier feature input layer, followed by 2 hidden layers of 1024 units. We use Fourier dimension drourier of size 1024. We initialize the entries of our trainable Fourier basis with B, (0, o²). with a 0.01 for all environments except Cheetah, Walker, and Hopper, where we use MLP with three hidden layers. The first MLP hidden layer is slightly wider, about 1 1100 units, to compensate for the extra parameters in LFF's first layer due to input concatenation. Learning curves are averaged over 5 seeds, with the shaded region denoting standard error.
0.001. To make the parameter count roughly equal, we compare against an
Image-based LFF Architecture Setup We test image-based learning on 4 DeepMind Control Suite environments [43] with SAC RAD [24], which uses data augmentation improve the sample efficiency of image-based training. The vanilla RAD architecture, which uses the convolutional architecture from Srinivas et [39], denoted as "CNN" Figure To apply LFF images, we observe that computing Bx at each pixel location is equivalent to a 1x1 convolution without bias. This Ixl convolution maps the the RGB channels at each pixel location from dimensions to drourier/2 channels. We then compute the sin and cos of those channels and concatenate the original RGB values, so our image goes from H x W x 3 to x x (diowrier + 3) embedding. The Ix] conv weights are initialized from (0, o²) with Finger and Quadruped. As we did in the state-based setup, we make the CNN baseline fair by adding an additional Ixl convolution layer at the beginning. This ensures that the "CNN" and "CNN+LFF" architectures have the same parameter count, and that performance gains are solely due to LFF.
0.1 for Hopper and Cheetah and = 0.01 for
Results
We provide empirical support for the approach by investigating the following questions:
1. Does LFF improve the sample efficiency of off pollcy state-based or image-based RL?
2. Do learned Fourier features make the Bellman update more stable?
3. Does LFF help more when applied to the policy or the Q-function?
4. Ablation: How important input concatenation or training the Fourier basis B?
7

---

Cheerat (pinls)
(pioda)
Hopper (psea)
Quadssped Ran (soak)
Acaras Ratur
Antua Rutum
-00
60-
0.00
0:20
0.50 Steps
0.75
1000
0.0
0.00
0.23
3.50
0.75
1.00
0.00
0.23
3,50
0.75
1.00
Staps
Stepe
Staps
Figure 6: Off-policy Image-based Evaluation: SAC experiments on learning DMControl envi- ronments from pixels. LFF can yield dramatic improvements in sample-effin aamll-efficiency over CNNs.
Quadruped Walk, scale11
Quadhuped Walk, scalo=10
Quad Walk, nole scalo-30
Averge Raam
Rue Retum
Just Retum
0.00
03.35
01333
0.25
0.10
03.55
0.53
0.25
Stgps
X3P
Stapo
x37
Stugo
Figure 7: State-based with Added Noise: We add zero- Gaussian noise to the targets. As the
standard deviation of the added noise increases, LFF maintains its performance better than MLPs.
6.1 LFF Architecture for Off-policy RL
We show the results of using the LFF architecture for state-based RL with SAC in Figure 5. LFF does clearly better than MLPs in 6 out of 8 environments, and slightly better in the remaining 2. Figure shows even stronger results on image-based based RL with SAC and RAD. This is especially promising because these results use the hyperparameters that were tuned for the MLP or CNN baseline. We find that the return consistently starts increasing much earlier with LFF. We hypothesize that LFF reduces noise propagation due to bootstrapping, so less data is required to overcome incorrect targets. SAC can use these more accurate Q-values to quickly begin exploring high-reward regions of the MDP.
For the state-space experiments, we also test several baselines:
MLP with weight decay, tuned over the values {10-3,3 3 10 1,-,,,010 1,-4,3 10 10 50- Weight decay helps learning in most environments, but it can hurt performance (Acrobot, Hopper) or introduce instability (Cheetah). Weight decay strong enough reduce overfitting may simultaneously bias the Q-values towards 0 and cause underestimation bias.
MLP with dropout [40]. We add a dropout layer after every nonlinearity in the MLP. We search over [0.05, 0.2) for the drop probability, and find that lower is better. Dropout does help in most environments, although occasionally at the cost of asymptotic performance.
MLP with functional regularization [31]. Instead of using a target network to compute target values, we use the current Q-network, but regularize its values from diverging too far from the Q-values calculated by a lagging snapshot of the Q-network.
MLP with spectral normalization [13]. We add spectral normalization the second-to-lase layer of the network, as is done in [13]. but find that this works very poorly. It is likely necessary to tune the other hyperparameters (learning rate, target update frequency, Polyak averaging) in order to make spectral normalization work.
Overall, LFF consistently ranks around the top across of the environments. can combined with weight decay, dropout, functional regularization for more gains, and has simple plug-and-pil and-play advantage because a single set of parameters works over all environments.
6.2
Do learned Fourier features improve the stability of the Bellman updates?
Our key hypothesis is that standard ReLU MLP Q- f-functions tend to fit the noise in the target values, introducing error into the Q-function at some (s, a). Bootstrapping with this incorrect Q-value to calculate target values for other (s',a') yields even noisier targets, propagates the error to other states, and causes instability or divergence (see Appendix D for more details). To further test whether LFF solves this problem by filtering out the noise, we train SAC agent state-based DMControl
8

---

Tam (fard)
Cheetah
Quadivrre
Humanoid Run
700
Anangs Rature
Anmrg Ratur
[oun]
0..0
0.25
0.20
8.70
1,00
0.50
01233
0.00
0.23
1.00
0.300
01.37
0.20
0.70
1.00
0.00
0.20
0.50
0.55
110
Stape
Staps
Steps
Steps
Figure Effect LFF on Stability of Bootstrapping: We train SAC, foregoing target network, by bootstrapping directly from the Q-network being trained. The dashed line shows the LFF performance with a target network after 1M steps. We find that the LFF network is remarkably stable, and even learns faster on Quadruped Walk than when using target networks. However, LFF fails to learn on Humanoid, indicating that higher dimensional problems still pose problems.
Firger Turn (sar)
Weller Ran
Quadiaped Rus
Harraoodd Run
830-
mm
Average Recum
Pecaars
Anarae
MIP
LFF Q
LFF Polac
0.10
11:31
1.75
0.0
03.35
11.51
0.75
1.00
Steps
Stegs
50ags
Figure 9: LFF Policy vs Q-function. Walker and Quadruped results indicate that only using LFF for the Q-network is just as good as using LFF for both networks. In contrast, using LFF for the policy network is about as bad as the MLP baseline. This suggests that LFF primarily improves off- offlpolicy learning by reducing noise in the Q-network optimization.
environments with either of the following modifications: testing the Q-function's robustness by adding Gaussian noise to the targets, or removing target networks altogether.
Gaussian noise added to targets In each bootstrap step, we add zero-mean Gaussian noise with standard deviation 1, 10, or 30 to the targets. LFF maintains higher performance even at significant noise levels, indicating that it is more robust to bootstrap noise. Full results are in Figure 12.
No target network Target networks, updated infrequently, slow down the propagation of noise due to bootstrapping [29]. LFF fits less noise to begin with, so it should work even when the target network is omitted. Here, we bootstrap directly from the network being trained. Figure 8 shows that MLPs consistently fail to learn on all environments in this setting, while the LFF architecture still performs well, except when the problem is very high dimensional. LFF even manages to learn faster in Quadruped Walk than it does when using a target network, since there no longer Polyak averaging [25] with target to slow down information propagation. Omitting the target network allows us to use updated values for Qs aa), instead of stale values from the target network. This result is in line with recent work that achieves faster learning by removing the target network and instead penalizing large changes to the Q-values [37].
Overall, Figure 7 and 8 validate our theoretical claims that LFF controls the effect of fig-frequency noise on the learned function, and indicates that LFF successfully mitigates bootstrap noise in most cases. Tuning the SAC hyperparameters should increase LFF sample efficiency even further, since we can learn more aggressively when the noise problem is reduced.
6.3 Where Do Learned Fourier Features Help?
In this section, we confirm that our LFF architecture improves RL performance by primarily pre- venting the Q-network from fitting noise. We train state-based SAC with an MLP policy and LFF Q-network (LFF Q-net in Figure 9), or with an LFF policy and MLP Q-network (LFF Policy). Fig- ure 9 shows that solely regularizing the Q-network is sufficient for LFF's improved sample efficiency. This validates our hypothesis that the Bellman updates remain noisy, even with tricks like double Q-networks and Polyak averaging, and that LFF reduces the amount of noise that the Q-network accumulates. These results suggest that separately tuning for the Q-network and policy networks may yield further improvements, as they have separate objectives. The Q-network should be resilient to noise, while the policy can be fine-grained and change quickly between nearby states. However,

---

(hard)
Walker
Quadiuped
Hamaniid
Anaagg Rature
Antra Raurr
No concal
unform FV
(aun)
200
0.50
0.35
0B
1.56
1.00
0.20
0..23
0.33
0.20
100
0.00
0..5
0.33
0.76
1.00
0.00
0.50
0.25
3.00
Stape
Steps
Steps
Staps
Figure 10: Ablation analysis: We train SAC on DMControl environments with three variants of our architecture: LFF, LFF with fixed Fourier features, and LFF without input concatenation. Lower dimensional environments like Finger and Walker are more forgiving for fixed Fourier features or omitting input concatenation. However, Humanoid absolutely requires both modifications to learn.
for simplicity, we use LFF for the Q-networks and the policy networks. Finally, we also train vanilla MLPs where we concatenate the input z to the first layer output. LFF outperforms this variant, confirming that concatenation is not solely responsible for the improved sample efficiency.
6.4 Architectural Ablations
LFF features two key improvements: learning the Fourier feature basis B and concatenating the input J. to the Fourier features. We perform an ablation on several DMControl environments with SAC in Figure 10 to investigate the impact of these modifications.
We first find that training the Fourier feature basis B is critical. Across all four enviromments, learning is impacted by using a fixed, randomly initialized B. This is because finding the right Fourier features initialization is unlikely in our high dimensional RL problems. Training B allows the network to discover the relevant Fourier features on its own. The relationship with dimension is clear: as the input dimension increases, the performance gap between LFF and fixed Fourier features grows.
Concatenating the input x to the Fourier features is also important. It maintains all of the information that was present in æ, which is critical in very high dimensional environments. If the Fourier basis B is poorly initialized and it blends together or omits important dimensions of Jr, the network takes long time to disentangle them, if at all. This problem becomes more likely the observation dimension increases. While LLF can learn without concatenation in low-dimensional environments like Walker and Finger, it has a much harder time learning in Quadruped and Humanoid.
Finally, we test an alternative approach to initializing the values of B. B, which has shape (k dimpu) x diggut- is now initialized B (I, cl, c²I...., 1IT where is the identity matrix, is an integer, and 0 < c < is tuned multiplier. This parallels the axis-aligned, log-uniform spacing used in NeRF's positional encoding [28]. but we additionally concatenate x and train B. We find that this initialization method, dubbed "Log-uniform FF" in Figure 10, is consistently worse than sampling from (0, o²). This is likely because the initialization fails to capture features that are not axis-aligned, so most of the training time is used to discover the right combinations of input features.
7 Conclusions and Future Work
We highlight that the standard MLP or CNN architecture in state-based and image-based deep RL methods remain susceptible to noise in the Bellman update. To overcome this, we proposed embedding the input using learned Fourier features. We show both theoretically and empirically that this encoding enables fine-gra fine-grained control over the network's spequcc learning rate. Our LFF architecture serves as a plug-and-ples addition to any state-of-the- method and leads to consistent improvement in sample efficiency on standard state-space and image-space environments.
One shortcoming of frequency- based regularization is that does not help when the noise and the signal look similar in frequency space. Future work should examine when this is the case, and test whether other regularization methods are complementary to LFF. Another line of work, partially explored in Appendix F, is using LFF with large to fit high frequencies and reduce underfitting in model-based or tabular reinforcement learning scenarios. We hope this work will provide new perspectives existing RL algorithms for the community to build upon.
10


Reviews:
[{"review_id": "HGQYICQdhEG", "reviewer": "Reviewer_Susc", "summary": "This paper proposes to augment the input space of deep value functions with a finetuned random projection followed by two periodic activations, both cos and sin. An interesting interpretation of this parameterization, Learned Fourier Features (LF2), is that it is an implicit Fourier decomposition of the input space, and that the variance used to initialize the random projection is monotonically related to the dominant frequencies that these random features capture (i.e. $\\sigma^2 \\to 0$ captures lower frequencies).\n\nIt is hypothesized that deep value networks tend to capture high-frequency features, which lead to poor Bellman updates, and thus that LF2 should help by reducing the capture of these high-frequency noise features. \nIn practice, LF2 does improve performance on some control tasks of the DMControl benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although it proposes an interesting method, this work does not really prove its hypothesis.\nThe central hypotheses of this work are that (A) RL methods are susceptible to target noise and (B) LF2 reduces noise in the Bellman update. (A) is used as a premise here and is never shown to be true, and although it is \"common wisdom\" in the deep RL community, it is contrary to some existing empirical literature. (B) is also never measured directly, and so never shown to be true. \nWhile it appears correct to claim that LF2 improves performance in some cases, it is impossible to link that improvement to the noise reduction hypothesis.\n\nThe theoretical part of this paper is interesting, and is probably a valuable contribution. Unfortunately, results in this section do not trivially transfer to hypotheses (A) & (B).\n\nWhile I am confortable with deep RL, I am only superficially familiar with Fourier features literature. Training Fourier features appears to be something that isn't normally done, and while lots of papers have proposed regularizations to value functions, I am unaware of frequency-domain methods. The paper itself is well written and was easy to understand.\n\nAlthough I find the method interesting and in-line with many intuitions that deep RL researchers have, this paper does not teach us much, and relies too heavily on proxies to \"prove\" its hypothesis. More work needs to be done for this paper to be impactful, and to really teach us something about deep RL.\n\n\nSpecific comments:\n- l.32 Farebrother et al. [1] successfully apply dropout to DQN\n- Section 3, Some unusual notation: using \\gamma for the frequency mapping, and \\tau for the activation function. In RL, \\phi is typically used for the former, and in deep learning, \\sigma or `a` for the latter.\n- l.245 'with the shaded region denoting 68% confidence interval'. 68%? This seems like a rather odd choice. 90 and 95% confidence intervals are more usual. I wonder what justified this choice? You may want to refer to papers such as [2] for guidance.\n- l.248 'LF2 does clearly better than MLPs in 6 out of 8 environments'. My _subjective_ reading of Figure 5 does not suggest so, but a statistical test would provide a quantitative assessment of superiority. How statistically significant are these improvements?\n- Figure 8, I again doubt that any of these curves are significantly different from each other.\n- l.285 'This validates our hypothesis that Bellman updates remain noisy [..] and that LF2 reduces [noise]'. I'm not convinced by this experiment. Not only are the results of Figure 8 probably marginal, noise has not been measured. It would be much more convincing to actually measure this noise, if it exists at all (current literature suggests instead that deep value networks _underfit_ severely, thus what is badly propagated is not noisy memorized estimates but underfit overgeneralized value estimates, see [3]; [6] may be relevant as well, and touches on the correlation between Bellman noise and agent performance in Atari). \n- I find the choice of using SAC as a baseline to improve over quite odd. Presumably, the proposed method stabilizes the value function, but SAC is a policy gradient method -- one can only expect marginal improvements from improving the value function. Why not compare to a value-based method?\n- Baselines: A number of functional regularizations for deep RL have been proposed recently, but none appear here. For references, see e.g. [3,4] and Section 5 of [5].\n\n\n[1] Generalization and Regularization in DQN, Jesse Farebrother, Marlos C. Machado, Michael Bowling, 2018  \n[2] Deep Reinforcement Learning that Matters, Henderson et al. 2017  \n[3] Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning, Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine, 2020  \n[4] Temporal Regularization in Markov Decision Process, Pierre Thodoroff, Audrey Durand, Joelle Pineau, Doina Precup, 2018  \n[5] Beyond Target Networks: Improving Deep Q-learning with Functional Regularization, Alexandre Piché, Joseph Marino, Gian Maria Marconi, Christopher Pal, Mohammad Emtiyaz Khan, 2021 (Yes, this paper was only recently submitted to arxiv, but its Section 5 might contain good works to refer to)  \n[6] Interference and Generalization in Temporal Difference Learning, Bengio et al. 2020  \n\n\n=======  \nUpdate  \nI thank the authors for their answers, updates and additional experiments, I think that these more convincingly explain why the proposed method is advantageous. My main problem with the paper was connecting the hypothesis behind the method with the improvements, but this connection is much better now. I will update my score. To be clear, validating all hypotheses should be a main feature of papers like this one, not just showing improvements. I encourage the authors to feature this prominently when updating this paper.\nIt does additionally seem like the improvements in the paper are consistent. Their magnitude is limited, but could be valuable in certain environments where target noise can become a significant issue.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors on superficially address the limitations of their method. More could be done, for example, experiments could show the kinds of environments/observation spaces where their method fails (a failure setup similar to the one in Figure 2 should be easy to create).\n\nThere are no mentions of potential negative societal impacts, but this is fairly abstract work. As an exercise, the authors may want to situate the impact of RL as a field (to which they contribute to) on society.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BTMX405u4Q", "reviewer": "Reviewer_jV7a", "summary": "In this work, the authors analyze a layer for deep networks based on learnable Fourier features and propose to employ it as a way to improve performance in architectures used for Deep Q-Learning (DQL). Specifically, they introduce this layer to control the noise as a function of the frequency in the input domain, and parametrize the granularity of such features by means of a learnable parameter. \n\nThe authors analyze the behaviour of such layer using a neural tangent kernel approximation and empirically evaluate its noise-filtering properties in various DQL experiments, showing its benefits to the task.\nOne of the main claims of the paper is that by controlling the variance of the stochastic initialization of the proposed layer, they can control the frequencies at which the noise will be filtered, with subsequent improvements in terms of sample complexity and stability.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the idea of using base layers that learn a representation in some harmonic domain is not new (e.g. Alexandridis 2013, Zuo 2008,  Oyallon 2014-2017), to the best of my knowledge no prior application to target networks is present in the literature. Still, I suggest to expand the introduction with some of this references, mentioning that the base idea is not new.\n\nWhile the theoretical analysis is interesting in principle, it does not offer a strong insight on the proposed method. Specifically, Neural Tangent Kernel analysis makes the assumption that the network does not change significantly from the initialization, which is in-line with the idea that the initialization variance of the Fourier layers matters. However, if that is true, is it even worth training those layers? I invite the authors to investigate more the impact of early layers focusing on certain representations, either fixed or learned.\nAnother suggestion would be to use alternative tools for analyzing the behaviour of the network, not at initialization but at convergence (e.g. Approximate Inference Turns Deep Networks into Gaussian Processes - 2018), or studying the stability of the Bellman operator when using such layers (e.g. see Towards Characterizing Divergence in Deep Q-Learning - 2019).\n\nWhile the analysis has shortcomings, the proposed approach is investigated thoroughly in the experimental section, which constitutes the more relevant part of the work. The experimental set-up look adequate and the results are good, albeit not groundbreaking in terms of performance. I appreciate that the authors not only validated the overall performance in terms of reward but also the stability during training.\n\nOverall, the paper is clear and well-structured. Because I deem the fundamental questions investigated by this paper to be very relevant for the RL community, either in theoretical or empirical form, I think its shortcomings could be overshadowed by its positive effect on the RL community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors could stress more the limitation of their analysis and that it's just a sketch mostly caused by the current state of theory in DL.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7ccT87NFZgv", "reviewer": "Reviewer_aHKm", "summary": "The paper proposes to use input features concatenated with learned Fourier feature (LF2) basis to control the degree of fitting various frequencies in the training data. Theoretical analysis under NTK regime with 2 layer network shows that the initial variance of Fourier basis controls this degree. In value-based RL, the updates using target Q-values can result in noisy optimization and LF2 can reduce dependence on noise by prioritizing low-frequency functions.  Empirically, LF2 leads to small sample-efficiency gains on a subset of DM control environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The paper can be considered as a small modification on top of existing ideas. Building on prior work[1] that introduced a fixed fourier feature basis to capture high frequency functions, the paper proposes LF2 which learns the fourier feature basis  as randomly initialized features can be quite limited. Additionally, it utilizes the architecture trick to concatenate the input features with these fourier features. \n\nAlthough the related work is cited generally well, the paper doesn’t have an explicit section but could benefit from one. One point which is not discussed much is why the inherent spectral bias of deep networks towards learning low-frequency functions [1] is not sufficient for learning in deep RL. For example, recent work shows that this spectral bias can lead to too low effective rank of the learned features in deep RL [2] with theoretical results in the NTK regime. [3] also confirmed such findings for online RL.\nWhile Figure 2 takes an simulation attempt at it but can possibly be alleviated with deeper networks like ResNets or just using L2-regularization. Also \n\n**Quality**: The submission seems to be technically sound. I did not go through the proofs but the lemme presented in the paper made sense. The empirical evaluation currently seems to indicate LF2 only results in tiny improvements and could be somewhat stronger. Some concerns I have are listed below:\n\n- The empirical results in Figure 5 indicate that LF2 gains are marginal compared to standard MLPs (and might have just resulted from randomness in evaluation on some tasks). Furthermore, l2-weight decay performs reasonably well on most of the domains (and even outperforms LF2 on certain domains). Same is found to be true for PPO-based agents in the appendix. \n\n- The results on image-based domains in Figure 6 seem moderate too and are evaluated only in a smaller subset of 4 environments reported in Figure 5 (which may seem to be cherry-picked?).\nFurthermore, no other baseline is evaluated on such image-based domains including the l2-reg baseline in Fig 5. If the improvement is coming from learning smoother functions, another way to impose such constraints is to impose spectral normalization which is also reported to work quite well in RL [3] (and even on DM control tasks[4]). \n\n- The ablation study in Sec 6.2 cripples the standard SAC agent by removing the target network and adding a noise value of “11” and shows that LF2 works well. However, in practice, target networks are almost always used. So, how does LF2 compare to standard SAC when the target network is added back – are target networks sufficient for handling the noise? Also, why was a specific noise value of 11 picked – why not smaller/larger (or better a range of values on log scale)?\n\n- How much of the improvement in LF2 is just coming from concatenation of the input?  It was not clear to me whether the ablation studies answer this question (as there was no baseline which uses concatenated inputs with MLP/conv architecture with the same parameter count).\nSpecifically, what if I use the same architecture for the baseline as used currently but with the inputs concatenated on the first layer output as done for the LF2 architecture?\n\n**Clarity**: The submission seems to be well written and organized. A minor comment: Instead of reporting the 68% confidence intervals, it might be better to report the standard deviation as 68% CIs may not have a true coverage of 68% as the reported results are only with 5 seeds.\n\n**Significance**: The results seem to indicate that LF2 architecture may be better than standard MLPs on certain hard tasks such as humanoid. However, it is unclear when LF2 should be used or why it should be preferred over other existing methods like using ResNets or spectral normalization. That said, the example pointed in the appendix about learning overly complex Q-functions was quite neat and shows that LF2 might be useful in certain scenarios (but the main paper focuses on the other scenario). Overall, the paper in its current form may not be that exciting to the community.\n\n[1] Rahaman, Nasim, et al. \"On the spectral bias of neural networks.\" International Conference on Machine Learning. PMLR, 2019.\n[2]. Kumar, Aviral, et al. \"Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning.\" International Conference on Learning Representations. 2020.\n[3] Gogianu, Florin, et al. \"Spectral Normalization for Deep Reinforcement Learning: an Optimization Perspective.\" ICML (2021).\n[4] Bjorck, Johan, Carla P. Gomes, and Kilian Q. Weinberger. \"Towards Deeper Deep Reinforcement Learning.\" arXiv preprint arXiv:2106.01151 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-- Small gains in empirical experiments\n-- Lack of evaluation of relevant baselines listed in main review (such as deeper nets / spectral normalization / l2-normalization)\n-- Adhoc selection of environments for reporting gains \n\n====================================================================================================\nUpdate\n\nBased on the author response, I have updated the score to 6 (from initial score of 5) and increased confidence to 4 (from 3) as the new experiments and ablations show the proposed method indeed leads to consistent performance gains even though they may not be large. The authors also provided some empirical experiment on a toy gridworld domain, thus providing some evidence the premise of the paper for noise amplification in RL. Assuming that authors would incorporate the discussion in their camera ready (especially the part related to underfitting and noise amplification), I can recommend publishing this work at NeurIPS.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "1bHPP6Xj5-4s", "reviewer": "Reviewer_N6ix", "summary": "This paper introduces a novel data transformation for deep reinforcement learning which enables faster learning of low-frequency information and empirically improves the performance of deep RL agents on 8 state-based DM control tasks and 4 image-based DM control tasks. The paper shows theoretically the convergence rate of the proposed transformation for neural tangent kernels to desired frequencies.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I recommend to accept this paper. [Str1] The proposed transformation has reasonable empirical support and is sufficiently general to likely be applicable to a wide variety of problems. [Str2] The goal of adjusting the learned representation of deep RL algorithms to avoid the need for additional stability tricks (like target nets) is important to the community, and understanding the types of representation which lead toward stable algorithms is useful scientifically. \n\nA few weaknesses which currently prevent me from increasing my scoring---but I believe could be addressed easily during the discussion period--- [W1] There are a few highly related works whose citations were missed. [W2] The theoretical and empirical results do not support some of the conclusions drawn in the paper, especially that functional regularization is the cause of the improved performance. [W3] A few assumptions / assertions made during the motivation of the work seem unsupported (both in this paper and in the literature more broadly). [W4] The experiments have a few odd choices.\n\n---\n\n[W1] Two very important papers which should be cited and discussed:\n\n1. Representations for Stable Off-policy Reinforcement Learning [Ghosh and Bellemare, 2020]\n2. Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks [Ghiassian et al., 2020]\n\nThe first heavily discusses the role of (learned) representations on stability in deep RL. It shows empirically and theoretically several properties of representations that yield provably convergent (thus stable according to their def'n) deep RL algorithms. \n\nThe second proposes a highly similar approach as this submission. Instead of learned fourier features being used to separate the data on the frequency domain, Ghiassian et al. separate the features spatially using tile-coding fed into a neural network. Ghiassian et al. have similar results showing that this spatial separation significantly reduces the need for target networks (and thus improves stability according to this submission's def'n).\n\n---\n\n[W2] A central conclusion (and the title of the paper) indicate that the cause of the improved performance is _functional regularization_. However, I do not believe that any empirical or theoretical results actually show that this is the case. For the theory results, the only claims that can be supported are that the network converges to specific frequencies faster than others, which is a bit different than showing a form of regularization (note that faster convergence does not imply a different final solution, nor a constraint on solution space). Because the proposed algorithm additionally concatenates the original features, the function class is actually strictly _larger_ for the proposed transformation and so the difference in performance _can't_ be functional  regularization.\n\nI highly recommend reconsidering the title of the paper and the claims made throughout to more closely match what the paper can actually provide evidence for. Noise filtration, easier tuning of hyperparams, etc. all seem like more plausible and directly supportable claims.\n\n---\n\n[W3] This is a slight continuation of W2, but there are a few other instances where claims are not directly supported. During the motivation (roughly lines 31-39) the paper discusses weaknesses in prior approaches: dropout, weight-space regularization, early stopping. It isn't clear, however, that any of these approaches are actually harmful. Early stopping is perhaps intuitively harmful, but why is l1 or l2 regularization more harmful for bootstrapping than for the standard supervised prediction problem? This is less clearly true to me. The paper would benefit greatly by either (a) supporting these claims with citations or (b) supporting these claims empirically by comparing to a standard MLP + l2 regularization (for instance).\n\n**edit:** After reading other reviews and checking the paper again, I notice that I had missed that Figure 5 does include a baseline with l2 regularization. The core of my point still stands, however, that there are several instances of not quite supported claims throughout the paper.\n\n--- \n\n[W4] Some of the empirical decisions are not well motivated and may suggest some overclaiming and unfairness.\n\n1. Why use 5 seeds and 68% confidence intervals? This implies that most of the results are 32% likely to be incorrect (and for 8 domains this means the first experiment has approx a 93% chance of at least one claim being wrong). And with 5 seeds, the assumption that variance is appropriately estimated (and so the confidence intervals are accurate) is a pretty strong assumption.\n2. The statement on lines 247-248 appears totally incorrect. I count 4 wins (Cheetah, Quad Run, Quad Walk, Hopper), 3 ties (Acrobot, Finger, Walker) and 1 loss (Humanoid). The proposed still looks good considering these results, but the paper should be much more careful about avoiding overclaims.\n3. The environment subselection is a little concerning. How did you pick these 8 domains from the approx 28 domains in DMControl? Similarly, how did you pick the 4 image based domains? Of the 4 chosen image domains, 3 were where the proposed algorithm performed best for state-based experiments. This appears to be biasing results in favor of the proposed algorithm via domain cherry-picking.\n\nFinally, along the same lines of W2 and W3, section 6.2 is not directly measuring the conclusions it is trying to draw. By designing an experiment comparing two algorithms without target networks, you can make claims about the impact of target networks on performance. You cannot make claims about \"stability\" of the algorithm, nor can you support claims about the noise in the targets when not using target networks. If you want to make claims about stability and noise, you need to measure stability and noise.\n\nThat said, I actually believe section 6.2 has really interesting results about the interaction between your representation and the need for target networks. Keeping the same empirical results and changing the wording to be about target networks (instead of noise and stability) would make this section considerably stronger.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have sufficiently addressed both limitations and societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considered the learned Fourier features in RL as an alternative parametrization to the vanilla MLP for Q network. The author argued that the benefits of the Fourier features can **i)** reduces high-frequency noise; **ii)** the noise is the major performance killer in RL. Then, the authors showed the better performances of the proposed parametrized Q in off-policy RL tasks on DeepMind Control Suite from both state and image. 

Besides the novelty of using Fourier features raised by several reviewers (jV7a, aHKm,  Susc),  the major issue of this paper is that the claimed benefits of Fourier feature layer are indeed the main reason leading to better performances (N6iX and Susc). Specifically, the empirical study does not provide convincing support of claim, while the theoretical justification does not offer strong insight (Reviewer N6ix) and actually may not able to explain the empirical results. 

 - As reviewer Susc mentioned, the motivation is clearly justified: **i)** reduces noise; **ii)** the noise is the major performance killer in RL.


 - The difference between vanilla MLP and Fourier feature is in fact different parametrization families. With square loss in Eq. 4, the optimal solution actually is $E[y|x]$ if the parametrization is realizable. All parametrization is actually able to reduce the noise. How can the author guarantee the superior comes from uncertainty canceling, not **approximation error** reduction is never discussed. 


 - In the theoretical analysis part, the author calculate NTK, which relies on the assumption in the **limit case** and the neural network weights are not far away from initialization. In the experiment, only around 1000 random feature is used. As many recent papers [1, 2 and to name a few] suggest the situation is different in finite case. With this gap in mind, it is not clear to me the phenomena from kernel can be straightforwardly extend to random feature. To support the claim, the author may first compare the NTK vs. random feature. 


- The authors also provide the contraction proof for kernel feature. **i)** this proof is not clear to me whether the relu layer is used; **ii)** in fact, how to characterize the benefits of the contractition vs. noise cancellation should be discussed.

Let me quote the suggestion from Reviewer Susc "validating all hypotheses should be a main feature of papers like this one, not just showing improvements." As a scientific paper, we should make careful claim with supportive evidence. The paper indeed reveals some interesting phenomena, but with some overclaims without convincing support. 

Minor:

The term "functional regularization" might not be appropriate and not aligned with its vanilla meaning. In this paper, it is in fact parametrization family selection. 

Since most of the reviewers agreed to accept this paper, I will still recommend acceptance for this submission. However, I would suggest the authors can revise the paper,  taking the comments in reviews and meta-review into account. Specifically, reduce the over-simplified justification and the misleading claim, just make the claims which the empirical evidence can support to avoid the potential misleading of the whole community. 

[1] Fort, Stanislav, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel." arXiv preprint arXiv:2010.15110 (2020).

[2] Allen-Zhu, Zeyuan, and Yuanzhi Li. "What can ResNet learn efficiently, going beyond kernels?." arXiv preprint arXiv:1905.10337 (2019).
Decision: Accept (Poster)