Fourier Features in Reinforcement Learning with Neural Networks
Authors: David Brellmann
David Filliat
Goran Frehse
Venue: Accepted by TMLR ()

Abstract: In classic Reinforcement Learning (RL), the performance of algorithms depends critically on data representation, i.e., the way the states of the system are represented as features. Choosing appropriate features for a task is an important way of adding prior domain knowledge since cleverly distributing information into states facilitates appropriate generalization. For linear function approximations, the representation is usually hand-designed according to the task at hand and projected into a higher-dimensional space to facilitate linear separation. Among the feature encodings used in RL for linear function approximation, we can mention in a non-exhaustive way Polynomial Features or Tile Coding. However, the main bottleneck of such feature encodings is that they do not scale to high-dimensional inputs as they grow exponentially in size with the input dimension.

Published in Transactions on Machine Learning Research (09/2023)
Fourier Features in Reinforcement Learning with Neural Net- works
David Brellmann U2IS, ENSTA Paris, Institut Polytechnique de Poris
david
David Filliat
daviilfillatteennsteerrrrr
U2IS, ENSTA Paris, Instifut Polytechnique de Paris
Goran Frehse
gomn.frehseelenstee.o
U2IS, ENSTA Paris, Institut Polytechnique de Poris
Reviewed on OpenReview: https: // openreview. net/ forum? LN-
Abstract
In classic Reinforcement Learning (RL), encoding the inputs with a Fourier feature map- ping standard way to facilitate generalization and add prior domain knowledge. In Deep RL, such input encodings are less common since they could, in principle, be learned by the network and may therefore seem less beneficial. In this paper, Multilayer Perceptrons (MI.P) that indicate that even in Deep RL, Fourier features can lead to significant performance gains in both rewards and sample efficiency. Furthermore, we ob- serve that they increase the robustness with respect to hyperparameters, lead to smoother policies, and benefit the training process by reducing learning interference, encouraging spar- sity, and increasing the expressiveness of the learned features. However, a major bottleneck with conventional Fourier features is that the number of features increases exponentially with the state dimension. As remedy, propose simple, light version that only has linear number of features yet empirically provides similar benefits. Our experiments cover both shallow/ /eep, discrete/contimouss and on/off-polie RL settings.
present experiments
1 Introduction
In classic Reinforcement Learning (RL), the performance of algorithms depends critically on the way the states of the system are represented as features. Choosing appropriate features for a task an important way of adding prior domain knowledge since eleverly distributing information into states facilitates generalization. For linear function approximations, the representation is usually hasd-designed according to the task at hand and projected into higher-dimensionall space to facilitate linear separation (Sutton & Barto 2018). Tn RL, such feature encodings for linear function approximation have been proposed in the form of, e.g., Polynomial Features (Lagoudakis Parr 2003) or Tile Coding (Albus] 1971). However, the main bottleneck of these feature encodings is that they do not scale to high-dimensional inputs as they grow exponentially size with the input dimension.
In recent years, interest in Deep RL has grown in part because Neural Networks (NN) are able to learn feature representations. This allows algorithms to learn complex tasks from raw sensory data without prior knowledge (Mnih et al.] 2015] Schulman et al. 2017 Lillicrap et al.] 2015 Haarnoja et al. 2018). In Decp Learning, it is common to apply min-max normalization (Bishop et al. 1995) or batch normalization (Toffe) & Szegedy 2015). However, preprocessing inputs with hand-designed features are less common since such features could, in principle, be learned by the network and thus may scem less beneficial. In recent work, it has been shown that preprocessing inputs with Random Fourier Features (Rahimi & Recht 2007) help Multilayer Perceptrons (MLP) to control the frequencies that the network tends learn first Tancik et al.

---

Published in Transactions on Machine Learning Research (09/202))
Fouriaa Expanss jon
Q(o,a) = - (8,00)
Φ ((s,
Figure 1: Example of a 3-layer MLP with preprocessing via Fourier Expansion. The state 8 is processed with a Fourier function expansion before being passed into the MLP with the action a. Learned features (ss, a) are used for the prodiction of the Q-value Q(s, a).
2020 Wang et al. 2021 and improves the training performance for NN (Mehrkanoon Suykens 2018 Mitraa & Kaddoum 2021p. In Deep RL, has been observed that Tile Coding can improve performance, sample efficiency, and robustness to hyperparameter variations by mitigating lcarning interference (Ghiassian &
Huizhen Yu 2018 Ghiassian et al. 2020 Liu et al. 2019 In this paper, empirically study preprocessing inputs with function expansion based on the Fourier series in Deep RI, as illustrated in Figure 1 The study is based on kinematic observation- benchmarks, where observations are expressed as state vectors whose components are the agent's kinematic quantities. Although the advantages of Fourier Features have been investigated in the case of standard RT. (Konidaris et al. 2011); ; the best of our knowledge, their use has not yet boen studied in deep RL. Concurrent work proposes tuning the scale of learnable Fourier features one means of ensuring that high-frequency components of the value function are captured (Li & Pathak 2021 Yang et al.) 2021). Another recent work studied the use of the Fourier series with MLP in computer vision (Benbarka et al. 2022).
Our main contributions in this paper can be summarized as follows:
While Fourier Features are standard in classic Reinforcement Learning, we suggest that Fourier Features are beneficial in kinematic observation-based RI. problems with Neural Networks. We observe significant performance gains in both rewards and sample efficiency and extend the range of usable hyperparameters. In our experiments, Fourier Features outperform other common types of input preprocessing.
We empirically investigate the effects of Fourier features on the learning process and show that Fourier features lead smoother Neural Networks, mitigate learning interference, promote sparsity, and increase the expressivity of learned features.
We propose light, scalable version of Fourier Features to avoid the exponential explosion of traditional Fourier Features while maintaining much of their benefits.
The remainder of the paper structured as follows. We first discuss some of the limitations of Neural Networks in Deep RL. Then, we present our Fourier Light Feature encoding and demonstrate its advantages. Finally, analyze in more detail the underlying reasons for these gains.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Optinnal Q-function
Evaltaation Curve Day Exampte SLR
1.0
6.0
Nei Stee
Quadee
9.
NFQI (400
Ostiraal Palaraase
04
Acrogg Rote
3.0
NFOI (1000
NFQI (40044)
Action Action
NFOI (34REN)
50
NFQI (URNNNN:)
NEWI PLE (4000)))
5000
1000C
15000
2000
2500
Spaadds
(a) Dynamics & Optimal Value Function
(b) Evaluation Curves
Figure 2: Example of a Toy MDP with simple forward dynamics and complex value function, adapted from Dong et al. (020)). MLPs without function expansion underperform on the toy MDP. Evaluation curves are averaged over 10 training runs. Shading indicates the 95% confidence interval (CI). The tested architectures are 1-layer MLP with 400 hidden neurons, 2-layer MLP with 400 hidden neurons, and 4-layer MLP with 400, 2048, and 4096 hidden neurons.
2 Limitation of the Expressiveness of Neural Networks in Deep RL
Neural networks have seen widespread success in deep learning, so one might expect that they perform equally well in reinforcement learning. This, however, is not always the case, A study by Dong et al. (2020) suggests that Neural Networks have limitations when predicting Q-functions in RI. The authors show, both theoretically and empirically, that there are many Markov Decision Processes (MDPs) whose optimal Q- functions and policies be very complex with high-frequency components and variations, even in the case of simple dynamics in one-dimensional continuous state space. Such complexities arise from the recursive applications of the Bellman optimality operator.
To illustrate the limitations of Neural Networks for approximating value functions in Deep RI, we will compare the evaluation curves and the Q-function of different MLP architectures, with and without Fourier Features, using the following toy MDP distribution from Dong et al. (2020). The state space is S = [0, 1), the discrete action space is {0, 1}, and the reward is proportional the state, i.e., r(s) = s. The dynamics are randomly sampled from the space of picco-wise lincar functions with fixed number 1: of "kinks." For our example, shown in Figure 2a we use k 2 and uniformly sample the value of each kink in this dynamic function between 0 and 1. Figure 2a also shows the optimal Q-function (computed with dynamic programming), which is complex and has both high- and low-frequency components.
To facilitate the analysis, we use the Neural Fitted Q-Iteration (FQI) algorithm from Riedmiller (2005
The basic idea underlying Neural FQT is the following: instead of updating the neural value function on online, the update is performed offline on a set of transition experiences. During the k-th fitting iteration, FQI trains the Q-function, Qk: to match the target values, 3/4 = R 7P= which are generated using the previous Q-function, QK-1- Figure 3 shows the value function anted for different MLP architectures All of them underfit the optimal value function and fail to learn high-freet freuuerreeency components, even in cases that are over-parameteriieed Indeed, the spectral bias of Neural Networks towards low-frequency functions, i.e., functions that vary globally without local ffuctuations, has been studied in detail in (Rahaman et al.) 2019 Bietti & Mairal 2019; Cao et al.] 2021 Xu et al.] 2022). Such a bias may prevent MLPs from correctly learning high-frequency components of Q-functions that are complex and have high-frequency components, like the one in our example. This phenomenon is further exacerbated by the fact that Deep R.L algorithms based on TD errors generalize poorly and memorize experiences as they are trained (yyle et al. 2021 2022 Nikishin et al. 2022). Since low-frequency components are learned first by MLPs, they struggle to learn higher-frequency components. This problem can be alleviated by adding a functional exlension to the MLP, such as Fourier features. Using Fourier features in the above example improves the performance and adds high-frequency components, as shown in Figure 3 and Figure 4 In the next sections, we will focus on a specific functional expansion based on the Fourier Series add high-frequency components to the predictions of Neural Networks.
3

---

Published in Transactions on Machine Learning Research (09/2023)
Optimell Dxfenctioon
SPQU (RMIN
NIQQ
NEQI (90RG-4)
NPQU (2AAMN
NPQB (a116e))
NXQ0 FT.
Qudse
Nation D Action
05
Cancri CancrSStecs Stree
Figure 3: MLPs without function expansion underfit in comparison to the optimal value function the toy MDP from Figure 2a Results are averaged over 10 training runs. Shading indicates the 95% confidence interval (CI). The tested architectures are 1-layer MLP with 400 hidden neurons, 2-layer MLP with 400 hidden neurons, and 4-layer MLP with 400, 2048, and 4096 hidden neurons.
Optimal Q fuaction
NEQI (410-21)
NEQU (400
NFQI (3000)4)
NEQI (294994)
NEQI (409644)
NFQI FLF (4003))
Arglink
1.
1,0
1.5
1.0
0.5
0.5
0L5
0.5
03
20
16
>0
30
21
40
30
41
20
40
Preqency (Bt))
(a) For Action
Optinad Q-function
NIQQ (4008:2)
NFQ8 (-006944)
NFQ8 (204864))
NEQI (4096)-4)
NEQQ FLF (480G1)
Auveiaal
0.4
04
0.4
0.4
0.4
0.4
02
D22
20
Preganay [BQ]
(b) For Action
Figure 4: Fourier Spectrum of the averaged predicted Q-function on training runs depicted in Figure 3 The tested architectures are I-layer MLP with 400 hidden neurons, 2-layer MLP with 400 hidden neurons, and 4-layer MLP with 400, 2048, and 4096 hidden neurons. Those Fourier spectrum do not include the Fundamental frequency since it is much higher than other frequencies.
3 Applying Fourier Features to Reinforcement Learning with Neural Networks
Fourier Features (FF) were introduced in RL by Konidaris al. (2011) to approximate linear functions by using the terms of the multivariate Fourier series features. Tn practice, Fourier features are easy to use and perform better for linear function approximation than other popular feature encodings such as Tile Coding or Polynomial Basis (Konidaris et al. 2011]. Formally, they are generated by the order-m Fourier Feature function expansion FF 0, 1, + R", mapping a normalized state 8 E R" into y-dimensional feature space (Konidaris et al. 2011 with p = (m + 1)". For SiSp p, the feature i is given by
(F, COS(TT Tct,,
TT)))
(1)
where each cocfficient vector c takes a value in {O...., mm (one-to-onc). Examples of FF are given in Figure 5
The inner product s T determines the frequency along dimension and creates interactions between state variables. The major bottlenock of Fourier features that their dimension grows exponentially with the dimension n of the state space. To remedy this, we propose the following subset of Fourier features that do not join state variables during preprocessing and scale linearly with the state dimension.
4

---

Published in Transactions on Machine Learning Research (09/2023)
c-(0,00 0]
c-[0,]]
c-11.0]
[-|1,1] c-[0,5]
c-15.0]
c-15,2]
1.0
10
1,0
1.0
1.11
1.0
03-
0.5
0.5
150
0.5 0.0 015
0.0
05 0.11
00
100
0U0
Figure 5: Example of Fourier Features over variables. Darker colors indicate a value closer to 1, and lighter colors indicate value closer to -1. Note that c c= [0, 0] results in constant functiin. When [0, ky or [Kx, 0] for positive integers h₂ and Ky, the basis function depends only of the variables, with the value of the non-zero component determining frequency. Only when = [kg, ky] does it depend on both; this basis function represents an interaction between the two state variables. The ratio between ka and ky describes the direction of the interaction, while their values determine the basis function's frequency along each dimension.
W
W
W
Figure 6: Example Fourier Light Features.
We call order-m Fourier Light Features (FLF) the n(m Fourier Features generated by the order- Fourier Light Feature functional expansion FLF [0, pa(m+1) mapping a normalized state s $1.....,pl snl" E [[-1,]] 1]" into a n(m 1)-dimensional feature space using subset of decorrelated Fourier Features
FLF(s) [FF(s))]... FF(s.))",
with Fs() = [1, cos(TT:)) c)s()).))) E [1, m]. The choice not to mix state variables in this version of Fourier features is motivated by the fact that Fourier features are not directly used for predictions but rather serve as a preprocessing for the data injected into the NN. We let Neural Networks choose how state variables will be mixed during the learning of the features that will be used for predictions. Examples of order-7 FLF are shown in Figure
Overall Performance We apply Fourier Features (FF-NN) and Fourier Light Features (FLF-NN) on the off- off-policy Deep-Q Network (DQN) algorithm (Mnih et al. 2015) for the discrete action environments and on the on-policy Proximal Policy Optimization (PPO) algorithm Schulman et al. 2017) for contimuous action environments and compare to the algorithuns without encoding (NN). Both discrete and continuous environ- ments provide kinematic observations, i.e. observations are expressed as a state vector whose components are the agent's kinematics quantities. For the normalization of the states, we compute the min/max nor- malization when state variables bounded and a nonlinear normalization based on arctan otherwise. All alg are applied to Neural Network with the ReI.u activation function. For computation time reasons and fair comparison, only the learning rate and Fourier order are re-optimized for FF-NN and FLF-NN. Other hyperparameters are taken from Stable Baselines Zoo Rafiin (020)). See the appendix for further experimental details. Figure 7 shows the averaged returns per episode for DQN on four discrete-action en- vironments from OpenAT Gym (Brockman al., 2016). Figure X shows the averaged returns per episode of PPO on five continuous-action control tasks from Mujoco (Todorov al. 2012). For the latter, we only test FLF because the number of standard Fourier Features explodes due to the higher state dimension. In all discrete tasks, except for the LunarLander-v2 task, FLF and FF enhance sample efficiency, i.e. FLF and FF have better performance in terms of cumulative rewards with fewer environment interactions. In the LunarLander-v2 lask, their performance does not deteriorate. It is worth noting that in the MountainCar- v0 task, FLF and FF' significantly increase the final cumulative reward, as traditional Neural Networks in this experiment are unable to converge to the optimal policy. The increase of final cumulative reward for FLF and FF is not observed in other discrete environments, as they are relatively simple. In all continuous fasks, FLF considerably outperforms the baseline in terms of both cumulative rewards and sample efficiency.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobet-v1
CarlPole-v
LunarLander--2
MountainCar Car-V0
Averge Ketum For Eppccd
-100
300
1000
430
-120
-204
-500
-100
-400
-1000
NN
FR-N
FLF-N
0
20000 40000 60030
30000 100000
150000
50000
(00000
150200
50000
100000
150000
Timestops
Timesiops
Timessope
Tanestogs
Figure 7: The use of Fourier features improves performance and sample efficiency of DQN on discrete control tasks. Evaluation learning curves of NN (blue), FF-NN (orange), and FLE-NN (green); reporting episodic return versus environment timesteps. Results averaged over 30 training (different seeds), with shading indicating the 95% confidence interval (CI).
HallCChotab--2
Hopper-v2
InvertedBuhhieeenadllee
Swimmee-v2
Walker2d-12
1500
30000
Averge Reuu For Expood
4000
1750
5000
3000
6000
2000
4020
1500
100
1000
1000
250
3000
30
500
NN
RLF-N
0
Timesiops
166
Timestpo
Tineslops
166
Timestgo
lch
Timeslips
Figure 8: The use of Fourier Light features improves the performance and sample efficiency of PPO on continuous control tasks. Evaluation learning curves of NN (blue) and FLF-NN (green), reporting episodic return versus environment timesteps. Results are averaged over 10 training with shading indicating the 95% confidence interval (CI).
Where the dimension is small enough so that can apply FF, we obtain similar performance by FF and FLF.
Robustness to Hyperparameter Changes RL algorithms can be very sensitive to hyperparameter changes (Henderson et al. 2018 Islam et al. 2017). The following experiments indicate that Fourier features reduce the sensitivity to hyperparameters. F Figure 9a illustrates how the performance varies with the learning rate, keeping other hyperparametery constant. FF-DQN and FLF-DQN require a smaller learning rate, but perform well over a larger range compared to raw inputs. Experience replay buffers (Lin 1992 Mnih et al. 2015) and target networks (Mnih et al. 2015) were introduced in RL mitigate interference problems and have become critical in the training of many deep RL algorithms including DQN. However, it is at the cost of higher computational and memory costs and slower offline learning (Plappert et al. 20188). Zhang & Sutton (2017) highlighted difficulties in properly tuning the buffer size where either too small or too big buller can have a negative effect on performance. In Figure 9h and [c we vary only the bufler size or larget update frequency while keeping other hyperparameters fixed. In the cases where standard DQN shows large performance variations for different buffer sizes and frequencies, we observe that FF-DQN is both better and less sensitive. This indicates a more stable learning process, with potentially less interference (see also Section 4.3 and makes Fourier Features even more interesting for nonstationary and online problems.
4 Observed Effects on Training Neural Networks
In this section, we look different metrics order investigate why Fourier features help the Neural Network to learn better and faster. We study the effects of Fourier features on the sparsity and expres- siveness of the NN, which in turn reduce catastrophic interference. Catastrophic interference occurs

---

Published in Transactions on Machine Learning Research (09/2023)
MountataCar-vi
MountainnCar
-100
Avomge Cumpbaive
400
-120
Avenee Cinstahi
900
-120
-1-0
-140
300
-100
NN
-301
FT-NN
FF-N
100000
20000
20000
100000
20000
303031
Learning
Buffer
(a) Learning Rate Variations over 71 10 trainings
(b) Buffer Size Variations over n trainings
CurtPole-v1
DormatainCCrr-10
Avenag Canelatie Rowird
-100
-140
-180
301 Tarps Liptane Fraquzncy
Tirgel Upeber Pragacncy
(c) Target Update Variations over 5 trainings
Figure 9: Fourier Features are more robust to learning rate, buffer size and target update frequency. Cumulative reward over different hyperparameter variations, for NN (blue) and FF-NN (orange) on MountainCar-vo and CortPole-vl. Results are averaged over trainings and shading indicating the 95% confidence interval (CI).
when the learner 'forgets" what it has learned in the past by overwriting previous updates to better fit the learned function to recent data (McCloskey & Cohen 1989 French 1991). Such interference can signifi- cantly slow down lcarning and even prevent the network from converging to an optimal solution. In sparse representations, only a few features are aclive (nonzero) for any given input, so each update only impacts a few weights and less likely to interfere with other updates (Liu al. 2019: Hernandez-Garre
& Sutton 2019 Ghiassian et al. 2020 Pan al. 2020). Another beneficial effect of sparsity is the promotion of locality, where similar inputs should produce similar features. It may be thus easier for the agent. to make accurate predictions for an explored local region as the local dynamics are likely to be simpler function than the global dynamics. The authors showed that learning sparse representations improves performance (Liu et al., 2019 Hernandez-Garcia. & Sutton 2019 Ghiassian et al. 2020 Pan et al. 2020). Functional expansion may help the learning of sparse representations as was observed by Ghiassian et al. (2020) who empirically shows that applying Tile Coding before passing data into Neural Network improves the learning of sparse features. Enforcing sparsity can also promote expressiveness through the identification of key altributes by encouraging the input to be well-described by small subset of attributes. To achieve good performance, NN needs to extract expressive and fine-grained local features. This is particularly truc when consecutive raw inputs are similar and small differences between inputs may lead to different actions. Kunar et al. 22020 Luo et al. (2020); Lyle et al. (2021) identified in RL an implicit under aaader--armmterization of Neural Networks where under tatleerpaaatteriattiion
defined as an excessive aliasing of learned features, i.e., learned features are mapped into much smaller subspace than the feature space that could be generated by the NN. Consequently, they observed the neural network behaves underr-arrameterized network, generates less rich features, and leads to poorer performance.
4.1 Sparsity of Learned Representations
We measure sparsity with two proxy measures: normalized overlap and instance sparsity (Liu al., 2019 Hernandez- Garcia & Sutton, 2019 Pan et al.] (020). In the following, we take into account the number of dead neurons, İ.e. neurons that have a zero response value for any input, and call the other neurons

---

Published in Transactions on Machine Learning Research (09/2023)
Table 1: Fourier features and Fourier Light features promote sparsity on discrete control tasks. Sparsity scores with percentage of dead neurons, normalized activation overlap and instance sparsity obtained for DQN fed with raw imputs (NN), Fourier features (FF-NN) and Fourier Light features (FLF-NN), averaged across environment timesteps. Averages are taken across all timesteps and margins of error of the 95% confidence interval (CI) are computed over 30 trainings. Lower sparsity scores are better and better scores are in bold.
Architecture
MountainCar-vo
Acrobot-v1
CartPole-vl
LunarLander-v2
Dead Neurons
0.47 0.09
0.0
0.07 + 0.02
0.0
NN
Normalized Overlap
0.72 0.08
0.49 = 0.04
0.63 + 0.04
0.30 0.01
Instance Spansity
0.78 0.07
0.64 + 0.02
0.66 + 0.03
0.46 0.02
Dead Neurons
0.0
0.01
0.0
0.0
FF-NN
Normalized Overlap
0.37 0.06
0.05 + 0.02
0.52 d: 0.02
0.23 0.03
Instance Spansity
0.57 0.05
0.13 + 0.04
0.60 + 0.02
0.40 + 0.02
Dead Neurons
0.0
0.0
0.0
0.0
FLF-NN
Normalized Overlap
0.10
0.16 :
0.79 + 0.07
0.39 0.04
Instance Spansity
0.62 0.08
0.30 + 0.03
0.85 +: 0.06
0.55 0.04
alive. In our experiments, we use NNs with ReLU activation function, wherein dead neurons may occur. Let d be the number of neurons in the penultimate layer. The normalized activation overlap, as proposed by Sernandez-Garciaa & Sutton 2019). is
d
ai (eeraa((((s1,a))) a2)) m
When the normalized overlap between two representations is zero, there is no interference between their corresponding inputs. The normalization with the number of neurons alive avoids misleadingly low scores in cases where only few neurons are alive. The instance sparsity is the percentage of active units in the feature vector ó(s, a) for a given input value (8, a) {Liu et al.) 2019). We estimate these sparsity measures during the training over the same learned feature matrix (D) (0($1.01) ai),...,((($$............ az))² every 1,00 environment timesteps, where D == ((1), (1), ..., (8b, a datatset of == 3, 000 state-action pairs. State-action pairs in DD are drawn i.i.d from rollouts obtained with sub-optimal pre-trained policies and random policios. This construction of D aims to cover state-action pairs likely to be used during the DQN learning. Consequently, our estimations of the percentage of dead neurons are less restrictive than the true percentage of dead neurons. Nevertheless, we believe that measuring sparsity scores over 'D makes more sense since it removes neurons only active in parts of the state space that are less likely to be visited by the agent. Our results are summarized in Table [ the corresponding curves as a function of environment. steps can be found in Appendix C In all tasks, FF results in lower (and thus better) normalized overlap and instance sparsity. There are no dead neurons FF/FLF, suggesting a better use of the Neural Network capacity. However, FLF increases the sparsity in one instance (CartPole-vl), even though the learning performance of FLF-NN is better than NN in all instances. Hence, sparsity does not seem to be the only beneficial effect of FF FF/FLF.
4.2 Expressiveness of Learned Representations
To measure expressiveness, we compute the effective rank sranks of the learned feature matrix & built on the set D. normalized by the number of neurons in the pemultimate layer like (Kumar et al., 2020 This estimates the proportion of the sum of the k highest singular values a (4) 2 ... 2 mmin(b.a) ④ 2 0 of +
that capture 1 8 (usually d = 0.01) of the sum of all singular values:
1
Et o.(
srank, ( min k:
2 0
(4)
E (
minthbdd
where b is the number of state-action pairs used to build and d is the number of neurons in the penultimate layer of the NN. Intuitively, this quantity represents the number of "effective' unique components of the
8

---

Published in Transactions on Machine Learning Research (09/2023)
Acrebot-v1
CariPole-v1
Lamarl Lamaraande--22 Lander-22
MauntainCar--0
0.5
0.8
0.940
0.6
0.75
0.4
0.85
0.70
NN
0.6
022
FF-NN
0.60
FLF
200000
60020
500000 100000
150020
55000
1000000 150003
50000 3000000
150000
Timesiops
Timesops
Timestaps
Timestops
Figure 10: The use of Fourier features and Fourier Light features enhances the expressiveness of the learned features on discrete control tasks. Learning curves of the normalized effective rank sranks ( for NN fed with raw inputs (blue), Fourier features (orange), and Fourier Light features (green); averaged over 30 trainings with shading indicating the 95% CI.
feature matrix + that form the basis for lincarly approximating the targets. When the network aliases inputs by mapping them to a smaller subspace, & has only a few active singular directions and srank; ( takes thus a small value. Several works pointed to an implicit under-paramettriaatio with the measure of low effective rank during the training. They explained this phenomenon in RL because direct and accurate targets are not available Kumar et al.] (2020); Luo et al. (2020): Lyle et al. (2021). We must approximate them with bootstrapping, iee by sequentially fitting outputs to target value estimates generated from the function Iearned in previous iterations. Such targets can not serve efficient guidance for NNs to extract expressive representations Figure 10 shows the normalized effective rank over the environment timesteps of training. The learned features are more expressive for FF FF/FLF in all instances, which may induce better use of the network capacity and explain better performance. These results are consistent with the absence of dead neurons for FF FF/F /FFF reported in Table Features learned with FLF-NN are more expressive than those with FF-NN in most instances. In Deep RT, because targets are estimated with boostrapping a decrease in the effective rank of learned features can lead a sequence of NNs with potentially decreasing exprossivity and results in degenerate behaviors, generalization problems and drops in performance (Kumar et al. 2020; Luo et al. 2020 Lyle et al. 2021). In Figure 10. the curves with FF FF/FLF and raw data have a similar general trend in time, but in most instances, the decrease in the effective rank is less pronounced with FF/FLF. This suggests more stable learning process with less catastrophic interference for FF-NN and FLF-NN.
4.3 Learning Interference
The learning interference for a state-action pair (s,a) at time t with parameters W and loss L is defined as [oppez-aax & Ranzato 2017 Riemer et al., 2018):
LI: = WWW+(WWiiss L(W;: s, s,a).
Assuming small learning rate n and using the Taylor series expansion on the loss L, we have for an update made to W with (8t, a:):
L. R WWW.(Ws,,,.. aF (W++1 W.)
V 8c, a V (WW,(W,s,e)
where the quantity V LW, 8,, a.)
V W. 8, a) is the gradient alignment (Bengio et al. 2020: Lopez-
Paz & Ranzato 2017 Riemer et al.)
2018
Schaul et al. 2019 The positiveness or negativeness of this
quantity determines whether the update is constructive (i.c. positive generalization) or destructive (i.c. interference) on (s, a). To investigate interference, we estimate the stiffness of the gradient alignment FFott
ct al. 2020
p(s, (,,a,, 8', .') cos LWwL(W; 8, a), Vw s a'))
with parameters W, loss L, and cosine similarity cos(u, (u, v) u" v/ /|||||||||| Based on stiffness, we consider three proxy measures for gradient interference evaluated on the DQN experience replay buffer B:
9

---

Published in Transactions on Machine Learning Research (09/2023)
Table 2: Fourier features and Fourier Light features mitigate learning interference on discrete control tasks. Interference measures with Average of Stiffness (AS), Average of Interference (AI), and Interference Risk (IR) averaged across all timesteps for DQN fed with raw imputs (NN), Fourier features (FF-NN), and Fourier Light features (FLF-NN) discrete control tasks. The symbol (f) indicates that lower (higher) score is better. Best interference measures are in bold.
Architecture
MountainCar-vo
Acrobot-v1
CartPole-vi
LunarLander-v2
AS
0.24
0.09
0.22
0.06
NN
A1
-0.83
-0.60
-0,92
-0.56
IR
0.91
0.92
0..9
0.94
AS
0.10
0.03
0.05
0.06
FF-NN
A1
0.47
-0.37
-0.73
0.49
IR
-0.87
00.00
-0.98
00.2
AS
0.05
0.04
0.05
0.04
FLE-NN
0.86
0.67
0.87
0.79
0.98
0.94
CartPole-v1
Lamurl.anee
10
300
100
10
10
NN
FLF-NN
5000I 100000
150XI
50000 T000000
150000
50000 100000
150020
500001
T0000
15X000
Timestaps
Timesags
Timestops
Timestops
Figure 11: Preprocessing inputs with Fourier features or Fourier Light features may improve the smoothness of the Neural Network. Lower and upper bounds on the Lipschitz constant L of NN over training timesteps, for NN fed with raw inputs (blue), FF (orange), and FLF (green). Bounds are averaged over 30 trainings. A lower score is better.
Average Stiffness (AS):
Average Interference (AT): (e.a.)) (er))))( s' a²) lø(s, a, a, < 0], which only considers (neg- atively) interfering samples and determines the average of interference,
Interference Risk (IR): CVaRus(o(s,a,s/
= Esp(s. a,
s', a')))(lsss a, s's
VaRno 8", a')) A p(s,as s a') ) < 0] which is the conditional value at risk of interference where VaRao(p(s, a, 8 d')) is the 0.9-quantile of the distribution of p(s, a, s',a),
We estimate interference every 1, 000 timesteps on 64 randomly drawn samples from the experience replay buffer and average them over 30 trainings. Our results averaged across all timesteps are reported in Table 2 and curves showing the evolution of interference during the training can be found in Appendix [ In all cases, AS shows that update with state-action pair has less impact other NN predictions with FF/FLF compared to raw inputs. This is confirmed by higher (better) AI and IR scores. Our observations indicate that Fourier features help to generalize appropriately without overgeneralizing and lead to a more stable training and better performance. Interestingly, FLF-NN seems to have even less interference than FF-NN, while we observed sparser representations for FF-NN Scction [.1, Such results suggest that even if sparsity mitigates catastrophic interference, FLF may have other beneficial effects that reduce catastrophic interference.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3: Increasing the FLF order improves the metrics. The table shows Spearman's rank correlation coefficient rss between different metrics and the order of the FLF. The p-value of the hypothesis test indicates high confidence in the result in almost all cases. The metrics are dead neurons in % (DN), normalized overlap (NO), instance sparsity (IS), Average of Stiffness (AS), Average of Interference (AT), Interference Risk (TR); Lipschitz Lower Bound (LLB), Lipschitz Upper Bound (LUB), averaged across all timesteps for 5 trainings with DQN fed with Fourier Light features (FLF-NN), over an order varying from 1 to 30. J and f indicate the direction in which the metric is better.
Mountain Car-vo Car-vO
Acrobot-vl p-value
CartPole-vi
LandrLander-v2
Metric
p-value
p-valne
rg
p-valno
DN
0.876
2.198
10
0.882
1.173 x 10
0.658
7.768
10-5
0.800
2.188 x 10-10
NO
-0.991
4.528
10-26
-0.33
1.143
10
-D.77
6.692
10-7
-0.562
1.000
IS
0.902
2:215
10-26
0.724
6.082
x 10-6
0.75
1.795
10-6
0.131
4.836
10
sranikg(
U.766
8:290 x 10-7
0.988
7.749
10-36
0.984
2.374 x 10-22
1.0
0.000
AS
0.996
1.256
10-31
0.955
2.547
10-16
0.153
4.201
10-1
00962
6.575 x 10-18
AI
0.984
3.122%
10-2s
0.968
1,990
10
0.501
4.780
10
0.922
2.538 x 10-15
IR
0.086
1.256
10-31
0.996
6.515
10
0
5.566
10-5
0.989
1.000 x 10
LLB
-0 962
2.380
10-17
-0.352
5
x 10-2
-0 0.828
1.610
10
-0.977
5.119 x 10-21
LUB
0.575
8.863 x 10-4
0.268
1,521 x 10
0.06
7.228
10-5
0.98
7.797 x 10
4.4 Smoothness of Neural Network
In Neural Networks, larger weights are associated with poorer relative performance new data (Neyshabur ctt al. 2015 Bartlett et al. 2017 Neyshabur et al. 2017). The idea follows Occam's 's razor that models with small weight norms are simpler and perform better than complex models. Not allowing individual weight norms to grow can also discourage large changes in output during the training. Similarly, in Reinforcement Learning, regularization approaches that enforce small weight norms, such as weight decay, lend to produce better results (Farebrother et al.] 2018 Liu et al.) 2020; Cobbe et al.) 2019). A common approach is to normalize weights to ensure that the learned layers are 1-Lipschitz, thereby improving the smoothness of the model, improving convergence (Salimans & Kinginal 2016 Gogianu et al., 2021 and reducing the generalization gap (Rosca et al. 2020; Gouk et al. 2021 Wang et al. 2019). The exact computation of the Lipschitz constant for a NN is NP-hard (Scaman & Virmaux 2018). but lower bounds and upper bounds can be estimated. The lower bound is obtained by computing for each state-action pair (s, a) in a dataset of 300, 000 stale-action pairs the norm of the gradient. of the Q-value with respect to the state-action pairs and we report the largest norm encountered (Rosca et al., (2000). Whereas establish the upper bound, we compute the Lipschitz constants of each layer in isolation and multiply them :Gouk et al.) 2021). Under the la and l norm, the upper bound of the Lipschitz constant of an MLP is given by the spectral norm and the maximum absolute column sum norm measure of the weight matrix (Neyshabur) 2017 Gouk et al. 2021 We measure these bounds every 1, 000 timesteps of the training and we average results over 30 trainings.
In three out of the four tasks shown in Figure II NNs with FF features have a lower Lipschitz constant. Additional metries, based on the I,, 12, and Tao norm of different layers, indicate that NNs with FLF can lie between NNs with FF and simple NNs, sometimes even surpassing NNs with FF; see Appendix D
4.5 Correlations of the metrics with the FLF order
We finally investigate the correlation between the above metrics and the order of the Fourier Light features. The Spearman's rank correlation coefficients and p-values over an order varying from to 30 in Table 3 indicate that in almost all tasks, increasing the FLF-order is strongly correlated with better metric and that this correlation is significant. In the CartPole-vl task, the correlations are weaker, but closer look at the graphs, shown in Figure 12 suggests that this is due to outliers and saturation in the metric.
When looking at the final reward over varying FLF orders, shown in Fig 13 we observe that increasing the order increases the reward up to a certain point, beyond which performance degrades. Fourier order can be considered as an additional hyperparameter for Deep RL algorithms. Only for the LunarLander-y2 task, the correlation bet the reward and the FT.F-order is unclear.
11

---

Published in Transactions on Machine Learning Research (09/2023)
Normaltzed Overlap
Instunce Spursity
15
14 15 Order
20
Order
% of Dead Deal Neuzons
0150
00
0.80
0.75
0.7U
115
Onder
-0.4
-05
2
10
14
15
16
17
18
10
20
21
22
23
25
34
5
17
-0.8
-0.
Onder
(a) Normalized overlap (NO), instance sparsity (IS), dead neurons in % (DN), srank, Average of Stiffness (AS), Average of Interference (AT), Interference Risk (IR), for CartPole-vl task
2000
200
1500
1000
10 13 15 16 E7 18 Order
26
2 6 00 12 15
16 20
Onder
(b) Lipschitz Lower Bound (LLB) and Lipschitz Upper Bound (LUB) for Acrobot-vl task
Figure 12: Selected metrics over varying FLF orders, for two discrete control tasks
12

---

Published in Transactions on Machine Learning Research (09/2023)
CartPolext
MountainCarsti
100
-120
-110
Avergy Retten par Eptoock
80
-115
300
80
Avengg Notunn
-170
-125
300
-160
-180
-135
20
30
10
20
30
10
201
30
20
30
Order
Order
Order
Order
Figure 13: Cumulative rewards over varying FLF orders, averaged across all timesteps for 5 trainings with DQN fed with Fourier Light features. The red line indicates the performance for DQN without any prepro- cessing.
5 Conclusion
We studiod the effect of Fourier feature encoding on Deep RL for of discrete and continuous control problems. We found that Fourier features provide a systematic increase in the final performance, sample efficiency, learning stability, and robustness to hyperparameters. A detailed empirical analysis of the reason for this behavior showed that Fourier features improve the sparsity, expressiveness, and smoothness of the NN, and reduce catastrophic interference during learning. Additionally, a light version of Fourier features with only a linear number of features compared to the input size leads to similar benefits.
We have just begun to understand how Fourier features and more broadly input preprocessing may improve neural network training. Our work suggests thus several promising future research directions. The first is investigate whether Fourier features behave as implicit regularizer and decrease the generalization gap. Second, we plan to further study the correlations between performance, sparsity, expressiveness, learning interference, and smoothness of neural networks.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, Optuna: A nexl-gemeration hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, 2019.
James Albus. A theory of cerebellar function. Mathematical biosciences, 1971.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normaalieed margin bounds for neural networks. Advances in Neural Information Processing Systems, 2017.
Nuri Benbarka, Timon Höfer, Andreas Zell, et al. Seeing implicit neural representations fourier series. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2041-2055 2022.
Emnanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning. PMLR, 2020.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter opti- mization. Advances in neural information processing systems, 2011.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
Christopher M Bishop et al. Neural networks for pattern recognition. Oxford university press, 1995.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Lars Buitinck, Gilles Louppe, Mathicu Blondel, Fabian Pedregosa, Andreas Muelerr, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Ar- naud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, 2013.
Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In IJCAT, 2021.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning. PMLR, 2019.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International Conference Machine Learning, pp. 2627-2637. PML.R,
2020.
Jesse Farebrother, Marlos Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
Stanislay Fort, Pawel Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks, 2020.
R. French. Using semi-distributed representations to overcome catastrophic forgetting in connectionist net- works. Proceedings of the AAAT Conference on Artificial Intelligence, 1991.
Sina Ghiassian and Richard S. Sutton Huizhen Yu, Banafsheh Rafice. Two geometric input transformation methods for fast online reinforcement learning with neural nets. arXiv, 2018.
Sina Ghiassian, Banafsheh Rafice, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. arXiv preprint arXiv:2105.0522 052446, 2021.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep rein- forcement learning that matters. In Proccedings of AAAI Conference on Artificial Intelligence, (AAAI-18), 2018.
J. Fernando Garran and Richard S. Sutton. Learning sparse representations incrementally in deep reinforcement learning. arXiv, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning. PMLR, 2015.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reprodueibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv.1708.04133, 2017.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6988 2014.
arXiv preprint
George Konidaris, Sarah Osentoski, and Philip Thomas. Value function approximation in reinforcement learning using the fourier basis. In Twenty-fifth AAAT conference on artificial intelligence, 2011.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parametteiiation inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2020,
Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Rescarch, 2003.
Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 2021.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Hoess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement. learning. arXiv preprint arXiv:1509.02971, 2015.
Long Ji Lin. Self improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn., 1992.
Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White. The utility of sparse representations for control in reinforcement learning. Proccedings of the AAAI Conference on Artificial Intelligence, Jul. 2019.
Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization-an empirical study on continuous control. In International Conference on Learning Representations, 2020.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual lcarning. Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
Xufang Luo, Qi Meng, Di He, Wei Chen, and Yunhong Wang. 14r: Promoting deep reinforcement learning by the indicator for expressive representations. In LJCAI, 2020.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2021.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11560-14558 PMLR, 17 23 Jul 2022. URL https::/ mlr press/v162//yle222 html
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot. learning, 2021.
Michael McCloskey and Ncal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
Siamak Mehrkanoon and Johan AK Suykens. Deep hybrid neural-kernel networks using random fourier features. Neurocomputing, 2018.
Rangeet Mitra and Georges Kaddoum. Random fourier feature based deep learning for wireless communi- cations. arXiv preprint arXiv:2101.055 .05254, 2021.
Volodymyy Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement. learning. Nature, 2015. ISSN 00280836.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.0195; 2017.
Bchnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring generalization in deep learning. Advances in Neural Information Processing Systems, 2017.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning, pp. 16828-16847. PMLR, 2022.
Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. In International Conference on Learning Representations, 2020.
Andy Patterson. Pyfixedreps. httt::://gihuub..ccmmmmmm///////
2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018.
Antonin Raffin. RI baselines3 zoo. https://github.ccom http://hauuu.oom//////////a/aaaa
2020.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3.
2019.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceedings of the 20th International Conference on Neural Information Processing Systems, 2007.
Aravind Rajeswaran, Kendall Lowrey, Emannel Todorov, and Sham Kakade. Towards generalization and simplicity in continuous control. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
Martin Riedmiller, Neural fitted q iteration first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018
Mihaela Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. A case for new neural network smoothness constraints. In Proceedings on 'I Can't Believe It's Not Better!" at NeurIPS Workshops. PMLR, 2020.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 2016.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient esti- mation. In Proceedings the 32nd International Conference on Neural Information Processing Systems, 2018.
Tom Schaul, Diana Borsa, Josoph Modayil, and Razvan Pascanu. Ray interference: a source of platcaus in deep reinforcement learning. arXiv preprint 1rXiv:1904.11455 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 11f.:1777006477 06347, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.10739 2020.
Norman Tasfi.
Pygame learning environment.
https:///itthub.commm
LeGame-Learning -awee--laaaannn--iorooeetttt 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Huan Wang, Stephan Zheng, Caiming Xiong, and Richard Socher. On the generalization gap in reparame- terizable reinforcement learning. In International Conference on Machine Learning. PMLR, 2019.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks. Computer Methods in Applied Mechanics and Enginecring, 2021.
Zhi-Qhn John Xu, Yaoyu Zhang, and Tao Luo. Overview frequency principle/speccral bias in deep learning. arXiv preprint arXiv:2201.07335 2022.
Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approximation. In International Conference on Learning Representations, 2021.
Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. arXiv preprint ar Xrvi 1712.01275 2017.
A Additonal Experiment on Ant
This section shows the following additional empirical result on a larger scale domain which cannot be put in the main body due to space limitations. For this experiment, we apply only FLF to PPO and compare perfor- mance to PPO without encoding We only test FLF because the number of standard Fourier ures explodes due to the higher state dimension. PPÒ is applied to Neural Network with the ELu activation function. For computation time reasons and fair comparison, only the learning rate and Fourier order are re-optimized for FLF-NN. Other hyperparameters are taken from Nivida Isaac Gym Environments (Makoviychuk et al., the averaged returns per episode of PPO Ant continuous-action control task incor- porating 60 kinematic observations from Nivida Isaac Gym Environments (Makoviychuk et al.) 2021). As in Section FLF have better performance in terms of cumulative rewards with fewer environment interactions.
2021). Figure[44hhor
B Comparison With Other Input Preprocessing
FF/FLF provide clear benefits, but it is natural to ask if other simple input preprocessing could provide similar advantages. In this section, we propose to empirically compare the performance of Neural Networks fed with Fourier features and with other traditional hand-designed features used in standard RL. We first give a brief description of the different preprocessing we investigate in our experiments and then we perform a comparison on MountainCar-vl and CartPole-vi. We compare Fourier features with three standard input preprocessings:
Polynomial Features (PF-NN). Polynomials are one the simplest families of features used for interpolation and regression. The feature vector consists of all polynomial combinations of the state variables with a degree less than or equal to a specified degree (Lagoudakis & Parr 2003 Sutton &
Barto 2018).
17

---

Published in Transactions on Machine Learning Research (09/202))
Ani
5000
Ps
6000
2000
BLEAN
250
500
750
1000
1250
1500
1750
Denaioos
Figure 14: The use of Fourier Light features improves the performance and sample efficiency of PPO on Ant. Evaluation learning curves of NN (blue) and FLF-NN (green), reporting episodic return versus environment timesteps. Results are averaged over 30 training with shading indicating the 95% confi- dence interval (CI).
Random Fourier Features (RFF). Random Fourier Features were initially introduced to approx-
imate an arbitrary stationary kernel-invariant by exploiting Bochner's theorem (Rahimi & Recht, 2007). Recent works have shown promising results where RFF's boost the performance of deep neu- ral networks (Mchrkanoon Suykens 2018), reduce the probability of misclassification (Mitra & Kaddoum 2021). can facilitate ML.s to learn high-frequency functions (Tancik et al., 2020 Tn
RL, Rajeswaran et al. (2017) used them with Natural Policy Gradicnt to outperform performance obtained with NNs. The i-th feature of the Random Fourier Feature mapping RFF IR" > is
(F) 2//pcos scc,, cb)bbb)
where of b (0, 2x), and p is the number of features want generate. The term 2/ is used as a normalization factor to reduce the variance of the estimates. RFFs and Fourier features have very close definition, except that the vector e creating interaction between state variables is sampled from a normal distribution. RFFs are studied to understand if it is rather the structure or the choice of c that can explain the good performances.
Tile Coding (TC). Tile Coding Allbs 1971 Sutton & Barto 2018) is a generalization of state aggregation, which we cover the state space with overlapping grids (tilings) where each grid divides the state space into small squares (tiles). The representation of a state for each tiling is a one-hot vector of dimension the number of tiles that has one for the tile where the state in and otherwise. Concatenation of one-hot vectors for each tiling forms Tile Coding features. A nice property of Tile Coding is that generalization occurs to states other than the one trained if those states fall within any of the same tiles. Ghiassian et al. (2020) already proposed to preprocess inputs of Neural Network with Tile Coding to promote sparsity of learned representations and obtain better performance.
In the experiments shown in Figure 15 the ranking on final average rewards is: PF-NN < RFF-NN < NN < TC-NN < FF-NN ~ FLF-NN where means lower performance. None of the other input preprocess- ings achieve the performance of FF-NN//LF-NN extensive search. It is even worse since PF and RFF degrade performance. TC-NN does better than NN for the MountainCar-vo but has slightly the same performance for CartPole-vI. These results suggest that applying input preprocessing to NN does not necessarily improve performance and can even degrade them. In the following, we evaluate the elfects on the Neural Network using the metrics from Section 4 Sparsity measures, as reported in Table 4 suggest that standard input preprocessings degrade sparsity. Even Tile Coding, reported promote sparsity in (Ghiassian et al. 2020), produces less sparse representations than standard DQN. Tile Coding, just as FF/FLF, does not have dead neurons, while PF-NN and RFF-NN increase the number of dead neurons.
even though we tuned their hyperparameters through an
18

---

Published in Transactions on Machine Learning Research (09/2023)
CurPPle--VI
MeuntainCar-v0
1100
Averge Nater Fo Riuunde
-120
NN
-10
FLE-N
-100
PP-N
TC
20000
40000
12000
130030
20000
40000
50000
130000
140000
100000
Figure 15: Fourier Features/ Paatees/Forrice Fourier Light features outperform other standard input prepro- cessings on discrete control tasks. Evaluation learning curves NN (blue), FF-NN (orange), FLF-NN (green), PF-NN (red), RFF-NN (purple) and TC-NN (brown) reporting episodic return versus environment timesteps. Results are averaged 30 trainings with shading indicating the standard deviation.
Table 4: Input preprocessings do not necessarily promote sparsity in discrete control tasks. Sparsity scores with dead neurons in % (DN), normalized overlap (NO) and instance sparsity (IS) obtained for DQN fod with raw inputs (NN), Fourier features (FF-NN), Fourier Light fcatures (FLF-NN), Normalizod inputs (NI-NN), Polynomial features (PF-NN), Random Fourier features (RFF-NN) and Tile Coding (TC- NN) discrete control tasks averaged across all timesteps. Averages and margins of error of the 95% C are over 30 trainings. Lower scores better.
Task
NN
FF-NN
FLF-NN
PE-NN
RFF-NN
TC-NN
DN
0.47 +: 0.09
0.0
0.0
0.66 +: 0.08
0.48 + 0.04
0.0
MorntainCar-v0
NO
0.72 + 0.08
0.37 + 0.06
0.43 + 0.10
0.80
0.87 + 0.06
0.77 + 0.13
IS
0.78 + 0.07
0.57
0.05
0.62 + 0.08
0.84
0.90 + 0.05
0.86 + 0.09
DN
0.07 + 0.02
0.01
0.0
0.23 0.02
0.88 0.07
0.0
CartPole-vl
NO
0.63 + 0.04
0.52
0.02
0.79 + 0.07
0.73 0.07
0.58 0.03
0.66 + 0.06
IS
0.66 + 0.03
0.60 + 0.02
0.85 + 0.06
0.75 + 0.05
0.61 + 0.03
0.70
Expressiveness is indicated by the learning curves of effective rank in Figure 16 PF-NN and RFF-NN generate poorer features than NN fed with raw inputs. As expected given the absence dead neurons, TC-NN produces richer features than NN and is on par with FF/FLF-NN.
Interference scores, shown Table 5 indicate that PF-NN, RFF-NN, and TC-NN highly interfere during the training with poor Average of Interference (AI) and Interference Risk (IR) scores. These results are consistent with the poor sparsity scores.
Smoothness, as measured by the Lipschitz bounds shown in Figure 17 is improved by all input preprocess- ings, with TC/FF /FLF giving the best performance, followed by PF FF/RFF. In the shown instance, smoother networks correlate with better learning performance.
C Sparsity Learning Curves for DQN on Discrete Control Tasks
Normalized overlap as function of environment steps corresponding is reported in Figure 18 The average of these scores across timesteps can be found in Table || and average them over 30 trainings.
We estimate sparsity scores every 1,000 timesteps
19

---

Published in Transactions on Machine Learning Research (09/2023)
Carthil-11
MruntaisCur-10
0.5
115
0.4
NN
0.3
FLE-N
02
TESN
0
20000
.0000
50000
5000
100000
120000
100000
150000
20000
20000
60000
50000
10000
100000
140000
160000
Figure 16: Learning curves of the normalized effective rank sranks (+) for NN fed with raw inputs (blue), Fourier features (orange), Fourier Light features (green), Polynomial features (red), Random Fourier Features (purple) and Tile Coding features (brown) on discrete control tasks. Results are averaged over 30 training with the shade that indicates the 95% CI.
Table 5: Interference measures with Average of Stiffness (AS), Average of Interference (AI), Interference Risk (TR), Percentage of Interference within a batch (PercT) and Average Q-Interference (AQT) averaged across all timesteps for standard input preprocessings on discrete control tasks where 4 and f mean lower and higher the score better, respectively.
Tasks
NN
FF-NN
FLF-NN
PF-NN
RFF-NN
TC-NN
AS
+
0.24
0.21
0.14
AI
0.83
-0.47
0.54
-0.1
-0.88
0.86
MountainCar-vd
IR
0.91
-0.87
0.87
0.95
0.93
0.93
Percl
1
0.38
0.44
0.47
0.39
0.43
0.44
AS
0.05
0.13
0.49
0.07
CartPole-vl
AI
f
0.92
0.73
0.84
0.97
-..5
IR
-0.99
0.98
0.87
-0.99
-0.97
Perel
4
0.39
0.47
0.47
0.43
0.25
0.46
D Interference Learning Curves for DQN on Discrete Control Tasks
Evolution of Interference During the Training. We estimate interference every 1, 000 timesteps and average them over trainings. For each point in the learning curves, we randomly draw samples from the experience replay buffer to estimate the interference. All interference measures defined in Section 4.3.
CartPole-v1
MountalnCar-v0
10'
1100
10
107
10
307
102
10P
0
20000
40303
60000
100000
120000
140000
TAND0O
20000
40000
60000
40000
100000
120000
140000
160000
Timestaps
Timestane
Figure 17: Lower and upper bounds on the Lipschitz constant of NN over training timesteps, NN (blue), FF-NN (orange), FLF-NN (green), PF-NN (red), RFF-NN (purple), and TC-NN (brown). Bounds averaged over 30 trainings with shading indicating the 95% Cl
20

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobet-11
CartPole-1I
LomerLande-22
MountaisCa--1
1.0
0.5
0.50
0.8
Nomaltzrl
0/3
0.6
0.40
0.25
0.5
0.25
0.4
0.3
FFNNN
30000
4000C
00000
50000 103000
55000
.0000
100000
150000
30303
100000
153000
Tinestop
Timestpp
Figure 18: Learning curves of the normalized overlap for NN (blue), FF-NN (orange), and FLF-NN (green) with DQN on discrete control tasks, reporting normalized overlap versus environment timesteps studied in Table Results are averaged over 30 trainings (different seeds), with shading indicating the 95% CI.
MannbaaCa--1
111
0.5
Nomellzee (horti
007
0.5
0.5
FRAN
0,5
0.5
PIFNV
0.4
RET NN
23000
4000
50200
SC00O
20000
1400X)
155000
20009
40001
55000
S000
[0000)
1200C0
[4003]
155000
Timaslops
Timalopa
Figure 19: Learning curves of the normalized overlap for NN (blue), FF-NN (orange), FLF-NN (green), PF- NN (red), RFF-NN (purple), and TC-NN (brown) with DQN on discrete control tasks, reporting normalized overlap versus environment timesteps studied in Table 4 Results are averaged over 30 trainings (different seeds), with shading indicating the 95% CT.
Accobol-r1
CartPrle-s
Lancclander-42
Maundain Car-d))
015
0.30
0L5
0.25
008
144
0.20
366
0.7
0.15
062
NN
61
FU-NN
Timcrage
Timasiopa
Timostpo
Timedags
Figure 20: Learning curves of the Average Stiffness (AS) for NN fed with raw inputs (blue), Fourier features (orange), and Fourier Light features (green) studied in Table 2 Results averaged over 30 trainings with shading indicating the 95% Cl.
21

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobator1
CanPalest
Lamar andkra
Monatalol
0.0
-0.45
-42
-02
-0.4
0.4
-063
-044
-66
0.6
-0.3
MUJ
-88
-0.
NN
M1-NN
-0090
-1.
Timssipp
Timostpp
Timesiaga
Timesips
Figure 21: Learning curves of the Average Interference (AI) for NN fed with raw inputs (blue), Fourier features (orange), and Fourier Light features (green) studied in Table Results are averaged over 30 trainings with shading indicating the 95% CI.
CanPPlood
Laweelando-43
MountainCar-st
-2.00
-114
8
4.555
-999
IT-NN
-1.0
20003
4000
60000
50000
(00000
1550C0
50300
100000
150000
50000
.00000
15000)
Tinesiqgs
Timolage
Timodape
Timcslage
Figure 22: Learning curves of the Interference Risk (IR) for NN fed with raw inputs (blue), Fourier features Results are averaged over 30 trainings with
(orange), and Fourier Light features (green) studied in Table 2 shading indicating the 95% CI.
CartPrle-r1
MeunträsCar-pp
1.0
0.5
066
115
2
NN
104
FLE-X
012
PENN
TU XN
50000
12003)
100000
133000
140000)
300000
20000
80000
10300
100300
130000
1A000
Timeciags
Figure 23: Learning curves of the Average Stiffness (AS) for NN fod with raw inputs (bluc), Fourier features (orange), Fourier Light features (green), PF-NN (red), RFF-NN (purple), and TC-NN (brown) studied in Table Results are averaged over 30 trainings with shading indicating the 95% CL.
22

---

Published in Transactions on Machine Learning Research (09/2023)
Carthho-11
MouatainCur-10
0.5
0.0
-112
-02
-144
2
NN
FLE-N
RFFAN
TCNN
20000
10000
00000
20000
100000
120000
140000
160000
20000
20000
60000
90000
100000
120000
140000
160000
Tieessopss
Timeskops
Figure 24: Learning curves of the Average Interference (AI) for NN fed with raw inputs (blue), Fourier features (orange), Fourier Light features (green), PF-NN (red), RFF-NN (purple), and TC-NN (brown) studied in Table
Results are avcraged over 30 trainings with shading indicating the 95% CI.
CariPiles11
MouatsinCarv00
0.0
8
BENN
FLF-NN
PF
REF
-1.0
20000
10000
(003)
193337
100000
120000
140000
100000
20000
0000
09330
99330
100000
120000
140000
10000
Timeekeps
Timessrye
Figure 25: Learning curves of the Interference Risk (IR) for NN fed with raw inputs (blue), Fourier features (orange), Fourier Light features (green), PF-NN (red), RFF-NN (purple), and TC-NN (brown) studied in Table
Results are averaged over 30 trainings with shading indicating the 95% CT.
E Smoothness of Neural Networks for DQN on Discrete Control Tasks
Estimation of the Smoothness of the Neural Network. The exact computation of the Lipschitz constant for NN is NP-hard (Scaman & Virmaux 2018), but lower bounds and upper bounds can be estimated. The lower bound is obtained by computing for each state-action pair (s, a) in dataset 300, 000 state-action pairs the norm of the gradient of the Q-value with respect to the state-action pairs and we report the largest norm encountered [Rosca et al., 2020). Whereas to establish the upper bound, we compute the Lipschitz constants of each layer in isolation and multiply them (Gouk et al. 2021). Under the le and I norm, the upper bound the Lipschitz constant of an MLP given by the spectral norm and the maximum absolute column sum norm measure of the weight matrix (Neyshabur, 2017 Gouk al. (021). We measure these bounds every 1, 000 timesteps of the training and we average results over 30 trainings.
Simplicity of the Neural Network. Liu et al. (2220) observed that many tasks, smaller policy weight norms correlate with better generalization ability. Figure 26 Figure 26 Figure 28 visualize the l2, 4 lint weight norms on both layers, respectively. In most cases, FF and FLF reduce the la and I weight norms, especially in the second layer.
F Reproducing Experiments
The code for reproducing all experiments is available on GitHuu at httpp://gihhho..co///rridddeeeed Fourier FaariierFeatures.. in RL with NN
Discrete Control Tasks. We compare the performance of DQN on five discretc-action environments from OpenAI Gym (Brockman al. 2016):: Acrobot-v1, CartPole-vi, LamarLander-v2, MountainCar-vvl, and Catcher-vl [Tasfi 2016). We ML.P architecture with single hidden layer.
23

---

Published in Transactions on Machine Learning Research (09/2023)
Acmibat11
CartPele-r1
LanarLanddr--2
MountainCar-e
130
2x 10!
201
NN
201
FLFNN
20000
40000
0XX0
50000
100000
150000
59000
100000
190000
50000
10000)
150000
Tiresteps
Timessops
Timestops
Timessog
In the First. Layer
Arrobut-c1
CartPaierl
Lanort ander--2
Mountain wrs-tl
211
15
10
10
40
NN
10
HF-NN
20000
150000
50000
100003
Timasiops
Timolape
(b) In the Second Layer
Figure 26: Learning curves of the l2 weight norm for NN fed with raw inputs (blue), Fourier features (orange), and Fourier Light features (green), averaged over 30 trainings with shading indicating the 95% CT.
Continuous Control Tasks. We compare performance on three continuous-action control tasks from and Swimmer-v2. Because
Mujoco [Todorov et al.) 2012): HalfCheetah-v2, InvertedDotblaaenntuunsss of the higher state dimension, were only able to test Fourier Light Features. We MLP with two hidden layers.All experiments are averaged over 10 runs, with offline evaluations performed on the policy every 1, 000 environment timesteps.
Common Settings. The first for discrete action domains are from OpenAl Gym Brockman et al. 2016) with version 0.18.0 and continuous action domains from Mujoco [Todorov et al. 2012). The discrete action environment Catcher-vl is from "aGaae-Tearning-Fniiioo Environment (Tasfi 2016). For Deep Reinforcement
Learning implementations, we adopt the code from Stable (Raffin
2019 with version 0.10.0
based Pytorch 1.8.0. We use Adam optimizer (Kingma & Ba 2014), Xavier initializer (Glorot & Bengio
2010), and ReLU activation functions across all experiments. We evaluate offline each algorithm every 1, 000 training/environmen environment timesteps.
Hyperparameter Sampling on Discrete Control Tasks. Since hyperparameters for the discrete con- trol tasks were not included at the time of experiments, we tune hyperparameters with Optuna 2.4.0 (Akiba et al. 2019 for each discrete control task. We sample hyperparameter values in Table using the TPE sTree-structured Parzen Estimator) algorithm (Bergstra et al., 2011). Since TPE can be very sensitive the scores of the first trials, we run 5 independent hyperparameter rescarch where one rescarch is composod of 500 trials. Each trial corresponds TPE, and the score of each trial is based the return of 100 rollouts of the learned policy. Because Deep RL. algorithms are highly unreliable (Henderson et al. 2018 Islam et al.) 2017), we take the 15 best hy- perparameter settings returned by Opluna and run 5 additional trainings of 150, 000 timesteps. The best hyperparameter setting we take is the one with the better final average return over the trainings.
training of 120, 000 timesteps with hyperparameters sampled by
For scarching the learning rate and Fourier Order of FF and FLF, we adopt the same stratogy by doing just one research with Optuna. For the Fourier Light order, we sample values between and the Fourier Light order corresponding to the number of traditional Fourier features found previously for the research for FF.
24

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobot-rl
CartPolevil
LutarLander-12
MountainCa--1
100
1250
IIF
1000
800
80
600
(8)
150
10P
NW
300
10
FFFNN
20000
60000
50000
50000
100000
150000
50000
100000
199000
53900
100000
150000
Timestops
Timestepe
Tinestops
Tinesteps
In the First. Layer
Accobut-v1
CarPalr-rl
Lansel ander-12
MeontainCar-ve
10
107
21
30*
NN
HF-NN
10-1
20000
69000
.0000
100000
Timesiops
Tincolau
(b) In the Second Layer
Figure 27: Learning curves of the I weight norm for NN fed with raw inputs (blue), Fourier features (orange), and Fourier Light features (green), averaged over 30 trainings with shading indicating the 95% CT.
Table 6: Sampling Values for DQN
Hyperparameter
Range
Number of Hidden Layers
Nunber of Neurons per Hidden Layer
{16, 32, 64, 128, 256 I
Batch Size
{ {16, 32, 64, 100, 128, 256, 512]
Replay Buffer Size
{1e4, 5e4, 1e5, 1e6]
Discount Factor
{0.9, 0.95, 0.98, 0.99, 0.995, 0.999, 0.9999}
Learning rate
[le 5, {0.9, 0.95, 0.98, 0.99, 0.995, 0.999,
Target Update Frequency
Train Frequency
{1,4, 8, 16, 128, 256, 1000)
Exploration Fraction
[0.0.5]
Final Value of Random Action Probability
[0, 0.20
Fourier Order (FF)
{1, 31,2,344,5)
Hyperparameter Sampling Continuous Control Tasks. On continuous tasks, we keep the hy- perparameters from the original codebase, StableBaselines--2zo0 (Raffin 2020). For scarching the learning rale and Fourier Light Order in continuous control lasks, we adopt the same strategy the one detailed in the previous subsection by doing just one research of 500 trials with Optuna. In this setting, each trial corresponds to a training of 800, 000 timesteps with hyperparametors sampled by TPE. Similarly, the best hyperparameter setting we take is the one with the better final average return over trainings 800, 000 timesteps. The range of values for the Fourier Light order in continuous control tasks between 1 and 50.
Normalization. Before being passed into the Fourier features mapping or Fourier Light features mapping, input data need be normalized by normalizer v) R= [0, (Section 3) We compute either the min/max normalization when state variables are bounded or we apply nonlinear transformation based
25

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobet-11
CurPole-vI
2
MountainCar-d
200
200
FLFNN
20000
10000
GOO00
S5000
100000
150000
30000
100000
150000
50000
100000
150000
Timesipp
Thmessops
Timestops
Tinesops
In the First. Layer
Acrobot-vl
CartPole-ct
Lamarl aadorl.nndee-22
Mountoin ac-v
1350
107
10'
130
10'
-00
NN
200
HF-NN
40000
(00000 15000)
(b) In the Second Layer
Figure 28: Learning curves of the lint weight norm for NN fed with raw inputs (bluc), Fourier features (orange), and Fourier Light features (green), averaged over 30 trainings with shading indicating the 95% CT.
arctan where:
(8) arctan
5:=ti C,
+ 0.5
(8)
with t the normalizer shift parameter and c the normalizer scale parameter. t and с are arbitrarily chosen obtain (8max) = 0.9 and th: (8min) 0.1 where 8max,i and 8min, are respectively the maximum and minimum state variable values observed over 5 5, 000 rollouts generated by 5 suboptimal policies.
Training timesteps. For DQN, LunarLander-vl and MountainCar-vo, For Catcher-vl, fix the episode length to 500 timesteps. For PPO, run 2e8 limesteps for HallCheetah-v2 and Swimmer-v2 and 1e8 timesteps for Inverted v2.
160, 000 timesteps for Acrobot-v1, CartPole-vi, Catcher-vl,
Implementation Details for Standard Input Preprocessing. For generating Polynomial features we use Scikit-Learn (Buitinck et al. 2013) and for Tile Coding features we adopt the implementation Pattersson (2020). For Random Fourier features and Tile Coding fcatures, inputs are first normalized with the same normalization defined for Fourier Teatures/Pourier Light features. In our first experiments, we noticed that just changing the learning rate not sufficient to achieve convergence. We then keep the same MLP architecture and found another hyperparameter setting with Optuna by using the same hyperparameter ranges reported in Table 6 According to the experiments performed by Ghiassian et al. 2020 we set the number of tilings to and the number of tiles to 4.
26


Reviews:
[{"review_id": "X1qP1NAQZy", "reviewer": "Reviewer_1EW8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The empirical analysis is very well done. There are a few minor improvements that would help solidify claims in the paper, but overall I find the claims to be meaningful and well-supported.\n\nStrengths:\n* Fig 2b and Fig 3 make a very convincing case that FQI cannot reasonably approximate the optimal value function, no matter the size of the network. Using the proposed FLF preprocessing step and the smallest neural network, the proposed architecture learns a far more accurate value function estimate.\n\n* The proposed FLF representation substantially reduces the computational cost of FF by eliminating all non-orthogonal feature directions---effectively forming an m-order basis. The paper then leans on neural networks to learn the (nonlinear) recombination of these bases into more complex feature spaces. This seems like a very reasonable simplification of FF and the experiments well-justify the approach.\n\n* Fig 7 was surprising! I would have expected FF and FLF have much lower stable rank due to the fact that they have far more inputs than the standard neural network. More inputs would suggest some of the work of combining features is done in the preprocessing step and the NN is now being used more purely as a filtration (i.e. small stable rank). Instead, this result suggests that the varied features from preprocessing allows the NN to find more complex combinations of these features and retain a more rich representation (high stable rank). This is a great result.\n\n* Table 3 is great. It is nice to see all claims tested statistically and that nearly every claim is significant.\n\n---\n\n**Note:** I was a reviewer for this paper at ICLR 2022 and highly appreciated the direction of the work, but felt there were several limitations of the empirical study. All of those limitations have been very well addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically investigates the use of fixed Fourier Feature representations as a preprocessing step for neural network function approximation in reinforcement learning. The paper investigates the DQN and PPO learning systems equipped with two different preprocessing steps alongside their standard implementation and show across several domains that the fourier feature preprocessing step significantly improves performance, increases smoothness of the representation (measured by Lipschitz constant), increases the stable rank of the weight matrices, reduces the number of dead neurons, and decreases sensitivity to hyperparameters (stepsize, buffer size, target net refresh rate).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Bottom of page 2: It isn't clear to me that the claim \"Such complexities arise from the recursive applications of the Bellman operator.\" is true. Because we typically use discounting, recursively applying the Bellman operator is a filtration process---typically smoothening out the values. The lack of smoothness here likely is coming from the Bellman optimality operator, which has the maximization over actions which can cause high complexity. Anyways, this is a trivial wording nitpick.\n\n* Figure 3 would benefit from a different statistic than standard deviation. It isn't clear how shaded regions with std.dev. should be visually interpreted. Typically a shaded region communicates uncertainty: \"I think the mean is somewhere within this reason, but due to statistical noise I cannot know where\". Std.dev., however, is a measure of statistical variation and not uncertainty. A 95% CI would be reasonable here. However, I think this is actually an ideal place to use a tolerance interval instead, which would show both the variation and uncertainty and make the claim even more clear.\n\n* Figure 5 would benefit from more samples in order to draw more clear differences between algorithms for Hopper, InvertedPendulum, and Walker. \n\n* Throughout the paper uses 95% confidence intervals (which is wonderful) but frequently only 10 samples. I suspect these are percentile bootstrap confidence intervals. This needs to be stated explicitly somewhere. If these are PBCI, then likely 10 samples is insufficient to compute the confidence intervals and so these are likely to be anti-conservative. It would be useful to increase the number of samples to 30 for these instances. Perhaps Figure 6 can continue to use only 10 samples, since we can additionally lean on the structure of the sensitivity curve.\n\n* The bolded claim in Figure 8 is probably too strong for the given evidence. It would be good to qualify the claim a little. In Acrobot, the NN may have better smoothness than either FF or FLF, it isn't clear. The only overly clear result is MountainCar followed by Cartpole.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J01OrIBslQ", "reviewer": "Reviewer_mfm9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The results provided are quite convincing with respect to the claim that Fourrier features can be beneficial.\n\nWeaknesses:\n- Some technical elements are not described with all the necessary details. As it is currently, key elements are unclear to me (see below).\n- (minor) The authors mention that the code will be made available. Could it be shared at this point such that reviewers can check it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses how encoding inputs with a Fourier feature mapping can improve the performance of deep reinforcement learning models. Experiments show that it can lead to performance gains and increased robustness with respect to hyperparameters. The authors propose a specific version of Fourrier features that only has a linear number of features. The experiments cover a variety of RL settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The toy environment from Figure 2 is not very clearly described. First, the description of the MDP does not seem to be recalled in the paper but the reader is sent to another paper (would it be possible to describe it, at least in one or two sentences?).Then, for this toy environment, what would the Fourrier features look like given that the state is apparently only one scalar between 0 and 1? I do not understand the discussion in the last paragraph of Section 2 that discusses the idea of \"high-frequency\" components that are important, etc.\n- From Figure 4, it is unclear whether the blue (baseline) is significantly worse in 3 out of 4 of the experiments. Can it be clarified?\n- It would be interesting to provide a visualization of a few states and the corresponding Fourrier features for some of the environments to provide a better idea about what is fed to the the network in at least some of the cases. It would also clarify whether the pixels are used for the states or whether it is the low-dimensional state. It seems that only a few discrete features are provided (e.g. see Before Equation 2)? Does that mean for instance that for mountain car, only two features are used?\n- Some other important technical elements are not clearly mentioned, or at least not in the main text, which makes the reading not as informative as it could be. For instance, the ReLU activation functions are only mentioned in the appendix while it is a key element to discuss the \"dead neurons\" with zero activation.\n- FQI is mentioned then DQN is but not much detail are provided at least for FQI. \n- Figure 22, 23 (among others have a standard deviation that is zero for some of the curves? Is that normal?\n- (minor) Sometimes the word \"Figure\" is used to refer to a figure, sometime \"Fig\".", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DBdTY9IFd2", "reviewer": "Reviewer_Yrod", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, it felt easy to read, although I was already quite familiar with the cited papers, such as Dong et al., Lyle et al., or Kumar et al.'s works. I wonder if readers less familiar with this literature might benefit from a bit more hand holding.\n\nThe work in the paper is good science, it builds upon past investigations to thoroughly examine a proposed method and gain insights into its benefits.\n\nThe main weakness I can see is the scale of the tackled problems; I appreciate results at this scale, and I don't see it as a barrier to acceptance, but I think applying FLF to one larger scale domain would be beneficial. In particular, as the authors point out, a sufficiently large DNN eventually can model any function. Orthogonally to image domains like Atari or ProcGen (where convolutions are already Fourier-like), it may be a valuable contribution to identify the _types_ of (larger scale) domains where Fourier features are particularly important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper has two core contributions:\n- a low-rank version of Fourier features (FF), Fourier Light Features (FLF)\n- a thorough investigation of the use of Fourier features in (small scale) deep RL\n\nMore specifically, the proposed FLF assume that input dimensions are in some sense independent in the frequency domain. The intuition is that this provides a better basis for a non-linear model (like an MLP) to then decide how input dimensions are interrelated. This means that instead of $O(m^n)$ features we have $O(mn)$ features, for $m$ the number of frequency components, $n$ the input dimensionality.\n\nThe investigation reveals that FF & FLF:\n- can more easily model high-frequency value functions\n- increase sparsity in MLP representations\n- increase the effective rank (and thus presumably the expressivity) of representations\n- sometimes have some beneficial effects on interference \n- learn functions with a lower (approximate) Lipschitz constant, i.e. smoother functions\n- induces an intuitive capacity knob, $m$, which is predictive of improved performance (and of the above metrics)\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some comments:\n- \"state-based RL\", do the authors mean \"observation based RL\"? I'm not sure what \"state\" refers to here, presumably all RL is based on states?\n- it would be nice to have consistent font sizes on figures (ideally figure legends are at most 2 pts below the main text's)\n- Using $n(m+1)$ input features instead of $n$ input features in an MLP adds an extra $mn$ learnable parameters. This is non-negligible. I don't see an experiment that accounted for this discrepancy between FNF and NN (nor FF actually), unless I've misunderstood the setup.", "broader_impact_concerns": "No specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have done a great job in addressing the concerns raised by the reviewers. The paper is well-written and easy to understand. At the end of the rebuttal there is no remaining concerns about this paper. As a result, the reviewers unanimously agreed to accept this paper.
Decision: Accept (poster)