Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel data transformation for deep reinforcement learning which enables faster learning of low-frequency information and empirically improves the performance of deep RL agents on 8 state-based DM control tasks and 4 image-based DM control tasks. The paper shows theoretically the convergence rate of the proposed transformation for neural tangent kernels to desired frequencies.
main_review: I recommend to accept this paper. [Str1] The proposed transformation has reasonable empirical support and is sufficiently general to likely be applicable to a wide variety of problems. [Str2] The goal of adjusting the learned representation of deep RL algorithms to avoid the need for additional stability tricks (like target nets) is important to the community, and understanding the types of representation which lead toward stable algorithms is useful scientifically. 

A few weaknesses which currently prevent me from increasing my scoring---but I believe could be addressed easily during the discussion period--- [W1] There are a few highly related works whose citations were missed. [W2] The theoretical and empirical results do not support some of the conclusions drawn in the paper, especially that functional regularization is the cause of the improved performance. [W3] A few assumptions / assertions made during the motivation of the work seem unsupported (both in this paper and in the literature more broadly). [W4] The experiments have a few odd choices.

---

[W1] Two very important papers which should be cited and discussed:

1. Representations for Stable Off-policy Reinforcement Learning [Ghosh and Bellemare, 2020]
2. Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks [Ghiassian et al., 2020]

The first heavily discusses the role of (learned) representations on stability in deep RL. It shows empirically and theoretically several properties of representations that yield provably convergent (thus stable according to their def'n) deep RL algorithms. 

The second proposes a highly similar approach as this submission. Instead of learned fourier features being used to separate the data on the frequency domain, Ghiassian et al. separate the features spatially using tile-coding fed into a neural network. Ghiassian et al. have similar results showing that this spatial separation significantly reduces the need for target networks (and thus improves stability according to this submission's def'n).

---

[W2] A central conclusion (and the title of the paper) indicate that the cause of the improved performance is _functional regularization_. However, I do not believe that any empirical or theoretical results actually show that this is the case. For the theory results, the only claims that can be supported are that the network converges to specific frequencies faster than others, which is a bit different than showing a form of regularization (note that faster convergence does not imply a different final solution, nor a constraint on solution space). Because the proposed algorithm additionally concatenates the original features, the function class is actually strictly _larger_ for the proposed transformation and so the difference in performance _can't_ be functional  regularization.

I highly recommend reconsidering the title of the paper and the claims made throughout to more closely match what the paper can actually provide evidence for. Noise filtration, easier tuning of hyperparams, etc. all seem like more plausible and directly supportable claims.

---

[W3] This is a slight continuation of W2, but there are a few other instances where claims are not directly supported. During the motivation (roughly lines 31-39) the paper discusses weaknesses in prior approaches: dropout, weight-space regularization, early stopping. It isn't clear, however, that any of these approaches are actually harmful. Early stopping is perhaps intuitively harmful, but why is l1 or l2 regularization more harmful for bootstrapping than for the standard supervised prediction problem? This is less clearly true to me. The paper would benefit greatly by either (a) supporting these claims with citations or (b) supporting these claims empirically by comparing to a standard MLP + l2 regularization (for instance).

**edit:** After reading other reviews and checking the paper again, I notice that I had missed that Figure 5 does include a baseline with l2 regularization. The core of my point still stands, however, that there are several instances of not quite supported claims throughout the paper.

--- 

[W4] Some of the empirical decisions are not well motivated and may suggest some overclaiming and unfairness.

1. Why use 5 seeds and 68% confidence intervals? This implies that most of the results are 32% likely to be incorrect (and for 8 domains this means the first experiment has approx a 93% chance of at least one claim being wrong). And with 5 seeds, the assumption that variance is appropriately estimated (and so the confidence intervals are accurate) is a pretty strong assumption.
2. The statement on lines 247-248 appears totally incorrect. I count 4 wins (Cheetah, Quad Run, Quad Walk, Hopper), 3 ties (Acrobot, Finger, Walker) and 1 loss (Humanoid). The proposed still looks good considering these results, but the paper should be much more careful about avoiding overclaims.
3. The environment subselection is a little concerning. How did you pick these 8 domains from the approx 28 domains in DMControl? Similarly, how did you pick the 4 image based domains? Of the 4 chosen image domains, 3 were where the proposed algorithm performed best for state-based experiments. This appears to be biasing results in favor of the proposed algorithm via domain cherry-picking.

Finally, along the same lines of W2 and W3, section 6.2 is not directly measuring the conclusions it is trying to draw. By designing an experiment comparing two algorithms without target networks, you can make claims about the impact of target networks on performance. You cannot make claims about "stability" of the algorithm, nor can you support claims about the noise in the targets when not using target networks. If you want to make claims about stability and noise, you need to measure stability and noise.

That said, I actually believe section 6.2 has really interesting results about the interaction between your representation and the need for target networks. Keeping the same empirical results and changing the wording to be about target networks (instead of noise and stability) would make this section considerably stronger.
limitations_and_societal_impact: The authors have sufficiently addressed both limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors analyze a layer for deep networks based on learnable Fourier features and propose to employ it as a way to improve performance in architectures used for Deep Q-Learning (DQL). Specifically, they introduce this layer to control the noise as a function of the frequency in the input domain, and parametrize the granularity of such features by means of a learnable parameter. 

The authors analyze the behaviour of such layer using a neural tangent kernel approximation and empirically evaluate its noise-filtering properties in various DQL experiments, showing its benefits to the task.
One of the main claims of the paper is that by controlling the variance of the stochastic initialization of the proposed layer, they can control the frequencies at which the noise will be filtered, with subsequent improvements in terms of sample complexity and stability.
main_review: While the idea of using base layers that learn a representation in some harmonic domain is not new (e.g. Alexandridis 2013, Zuo 2008,  Oyallon 2014-2017), to the best of my knowledge no prior application to target networks is present in the literature. Still, I suggest to expand the introduction with some of this references, mentioning that the base idea is not new.

While the theoretical analysis is interesting in principle, it does not offer a strong insight on the proposed method. Specifically, Neural Tangent Kernel analysis makes the assumption that the network does not change significantly from the initialization, which is in-line with the idea that the initialization variance of the Fourier layers matters. However, if that is true, is it even worth training those layers? I invite the authors to investigate more the impact of early layers focusing on certain representations, either fixed or learned.
Another suggestion would be to use alternative tools for analyzing the behaviour of the network, not at initialization but at convergence (e.g. Approximate Inference Turns Deep Networks into Gaussian Processes - 2018), or studying the stability of the Bellman operator when using such layers (e.g. see Towards Characterizing Divergence in Deep Q-Learning - 2019).

While the analysis has shortcomings, the proposed approach is investigated thoroughly in the experimental section, which constitutes the more relevant part of the work. The experimental set-up look adequate and the results are good, albeit not groundbreaking in terms of performance. I appreciate that the authors not only validated the overall performance in terms of reward but also the stability during training.

Overall, the paper is clear and well-structured. Because I deem the fundamental questions investigated by this paper to be very relevant for the RL community, either in theoretical or empirical form, I think its shortcomings could be overshadowed by its positive effect on the RL community.

limitations_and_societal_impact: I think the authors could stress more the limitation of their analysis and that it's just a sketch mostly caused by the current state of theory in DL.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use input features concatenated with learned Fourier feature (LF2) basis to control the degree of fitting various frequencies in the training data. Theoretical analysis under NTK regime with 2 layer network shows that the initial variance of Fourier basis controls this degree. In value-based RL, the updates using target Q-values can result in noisy optimization and LF2 can reduce dependence on noise by prioritizing low-frequency functions.  Empirically, LF2 leads to small sample-efficiency gains on a subset of DM control environments.
main_review: **Originality**: The paper can be considered as a small modification on top of existing ideas. Building on prior work[1] that introduced a fixed fourier feature basis to capture high frequency functions, the paper proposes LF2 which learns the fourier feature basis  as randomly initialized features can be quite limited. Additionally, it utilizes the architecture trick to concatenate the input features with these fourier features. 

Although the related work is cited generally well, the paper doesn’t have an explicit section but could benefit from one. One point which is not discussed much is why the inherent spectral bias of deep networks towards learning low-frequency functions [1] is not sufficient for learning in deep RL. For example, recent work shows that this spectral bias can lead to too low effective rank of the learned features in deep RL [2] with theoretical results in the NTK regime. [3] also confirmed such findings for online RL.
While Figure 2 takes an simulation attempt at it but can possibly be alleviated with deeper networks like ResNets or just using L2-regularization. Also 

**Quality**: The submission seems to be technically sound. I did not go through the proofs but the lemme presented in the paper made sense. The empirical evaluation currently seems to indicate LF2 only results in tiny improvements and could be somewhat stronger. Some concerns I have are listed below:

- The empirical results in Figure 5 indicate that LF2 gains are marginal compared to standard MLPs (and might have just resulted from randomness in evaluation on some tasks). Furthermore, l2-weight decay performs reasonably well on most of the domains (and even outperforms LF2 on certain domains). Same is found to be true for PPO-based agents in the appendix. 

- The results on image-based domains in Figure 6 seem moderate too and are evaluated only in a smaller subset of 4 environments reported in Figure 5 (which may seem to be cherry-picked?).
Furthermore, no other baseline is evaluated on such image-based domains including the l2-reg baseline in Fig 5. If the improvement is coming from learning smoother functions, another way to impose such constraints is to impose spectral normalization which is also reported to work quite well in RL [3] (and even on DM control tasks[4]). 

- The ablation study in Sec 6.2 cripples the standard SAC agent by removing the target network and adding a noise value of “11” and shows that LF2 works well. However, in practice, target networks are almost always used. So, how does LF2 compare to standard SAC when the target network is added back – are target networks sufficient for handling the noise? Also, why was a specific noise value of 11 picked – why not smaller/larger (or better a range of values on log scale)?

- How much of the improvement in LF2 is just coming from concatenation of the input?  It was not clear to me whether the ablation studies answer this question (as there was no baseline which uses concatenated inputs with MLP/conv architecture with the same parameter count).
Specifically, what if I use the same architecture for the baseline as used currently but with the inputs concatenated on the first layer output as done for the LF2 architecture?

**Clarity**: The submission seems to be well written and organized. A minor comment: Instead of reporting the 68% confidence intervals, it might be better to report the standard deviation as 68% CIs may not have a true coverage of 68% as the reported results are only with 5 seeds.

**Significance**: The results seem to indicate that LF2 architecture may be better than standard MLPs on certain hard tasks such as humanoid. However, it is unclear when LF2 should be used or why it should be preferred over other existing methods like using ResNets or spectral normalization. That said, the example pointed in the appendix about learning overly complex Q-functions was quite neat and shows that LF2 might be useful in certain scenarios (but the main paper focuses on the other scenario). Overall, the paper in its current form may not be that exciting to the community.

[1] Rahaman, Nasim, et al. "On the spectral bias of neural networks." International Conference on Machine Learning. PMLR, 2019.
[2]. Kumar, Aviral, et al. "Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning." International Conference on Learning Representations. 2020.
[3] Gogianu, Florin, et al. "Spectral Normalization for Deep Reinforcement Learning: an Optimization Perspective." ICML (2021).
[4] Bjorck, Johan, Carla P. Gomes, and Kilian Q. Weinberger. "Towards Deeper Deep Reinforcement Learning." arXiv preprint arXiv:2106.01151 (2021).
limitations_and_societal_impact: -- Small gains in empirical experiments
-- Lack of evaluation of relevant baselines listed in main review (such as deeper nets / spectral normalization / l2-normalization)
-- Adhoc selection of environments for reporting gains 

====================================================================================================
Update

Based on the author response, I have updated the score to 6 (from initial score of 5) and increased confidence to 4 (from 3) as the new experiments and ablations show the proposed method indeed leads to consistent performance gains even though they may not be large. The authors also provided some empirical experiment on a toy gridworld domain, thus providing some evidence the premise of the paper for noise amplification in RL. Assuming that authors would incorporate the discussion in their camera ready (especially the part related to underfitting and noise amplification), I can recommend publishing this work at NeurIPS.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to augment the input space of deep value functions with a finetuned random projection followed by two periodic activations, both cos and sin. An interesting interpretation of this parameterization, Learned Fourier Features (LF2), is that it is an implicit Fourier decomposition of the input space, and that the variance used to initialize the random projection is monotonically related to the dominant frequencies that these random features capture (i.e. $\sigma^2 \to 0$ captures lower frequencies).

It is hypothesized that deep value networks tend to capture high-frequency features, which lead to poor Bellman updates, and thus that LF2 should help by reducing the capture of these high-frequency noise features. 
In practice, LF2 does improve performance on some control tasks of the DMControl benchmark.
main_review: Although it proposes an interesting method, this work does not really prove its hypothesis.
The central hypotheses of this work are that (A) RL methods are susceptible to target noise and (B) LF2 reduces noise in the Bellman update. (A) is used as a premise here and is never shown to be true, and although it is "common wisdom" in the deep RL community, it is contrary to some existing empirical literature. (B) is also never measured directly, and so never shown to be true. 
While it appears correct to claim that LF2 improves performance in some cases, it is impossible to link that improvement to the noise reduction hypothesis.

The theoretical part of this paper is interesting, and is probably a valuable contribution. Unfortunately, results in this section do not trivially transfer to hypotheses (A) & (B).

While I am confortable with deep RL, I am only superficially familiar with Fourier features literature. Training Fourier features appears to be something that isn't normally done, and while lots of papers have proposed regularizations to value functions, I am unaware of frequency-domain methods. The paper itself is well written and was easy to understand.

Although I find the method interesting and in-line with many intuitions that deep RL researchers have, this paper does not teach us much, and relies too heavily on proxies to "prove" its hypothesis. More work needs to be done for this paper to be impactful, and to really teach us something about deep RL.


Specific comments:
- l.32 Farebrother et al. [1] successfully apply dropout to DQN
- Section 3, Some unusual notation: using \gamma for the frequency mapping, and \tau for the activation function. In RL, \phi is typically used for the former, and in deep learning, \sigma or `a` for the latter.
- l.245 'with the shaded region denoting 68% confidence interval'. 68%? This seems like a rather odd choice. 90 and 95% confidence intervals are more usual. I wonder what justified this choice? You may want to refer to papers such as [2] for guidance.
- l.248 'LF2 does clearly better than MLPs in 6 out of 8 environments'. My _subjective_ reading of Figure 5 does not suggest so, but a statistical test would provide a quantitative assessment of superiority. How statistically significant are these improvements?
- Figure 8, I again doubt that any of these curves are significantly different from each other.
- l.285 'This validates our hypothesis that Bellman updates remain noisy [..] and that LF2 reduces [noise]'. I'm not convinced by this experiment. Not only are the results of Figure 8 probably marginal, noise has not been measured. It would be much more convincing to actually measure this noise, if it exists at all (current literature suggests instead that deep value networks _underfit_ severely, thus what is badly propagated is not noisy memorized estimates but underfit overgeneralized value estimates, see [3]; [6] may be relevant as well, and touches on the correlation between Bellman noise and agent performance in Atari). 
- I find the choice of using SAC as a baseline to improve over quite odd. Presumably, the proposed method stabilizes the value function, but SAC is a policy gradient method -- one can only expect marginal improvements from improving the value function. Why not compare to a value-based method?
- Baselines: A number of functional regularizations for deep RL have been proposed recently, but none appear here. For references, see e.g. [3,4] and Section 5 of [5].


[1] Generalization and Regularization in DQN, Jesse Farebrother, Marlos C. Machado, Michael Bowling, 2018  
[2] Deep Reinforcement Learning that Matters, Henderson et al. 2017  
[3] Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning, Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine, 2020  
[4] Temporal Regularization in Markov Decision Process, Pierre Thodoroff, Audrey Durand, Joelle Pineau, Doina Precup, 2018  
[5] Beyond Target Networks: Improving Deep Q-learning with Functional Regularization, Alexandre Piché, Joseph Marino, Gian Maria Marconi, Christopher Pal, Mohammad Emtiyaz Khan, 2021 (Yes, this paper was only recently submitted to arxiv, but its Section 5 might contain good works to refer to)  
[6] Interference and Generalization in Temporal Difference Learning, Bengio et al. 2020  


=======  
Update  
I thank the authors for their answers, updates and additional experiments, I think that these more convincingly explain why the proposed method is advantageous. My main problem with the paper was connecting the hypothesis behind the method with the improvements, but this connection is much better now. I will update my score. To be clear, validating all hypotheses should be a main feature of papers like this one, not just showing improvements. I encourage the authors to feature this prominently when updating this paper.
It does additionally seem like the improvements in the paper are consistent. Their magnitude is limited, but could be valuable in certain environments where target noise can become a significant issue.
limitations_and_societal_impact: The authors on superficially address the limitations of their method. More could be done, for example, experiments could show the kinds of environments/observation spaces where their method fails (a failure setup similar to the one in Figure 2 should be easy to create).

There are no mentions of potential negative societal impacts, but this is fairly abstract work. As an exercise, the authors may want to situate the impact of RL as a field (to which they contribute to) on society.
review_title: null

Meta-review: This paper considered the learned Fourier features in RL as an alternative parametrization to the vanilla MLP for Q network. The author argued that the benefits of the Fourier features can **i)** reduces high-frequency noise; **ii)** the noise is the major performance killer in RL. Then, the authors showed the better performances of the proposed parametrized Q in off-policy RL tasks on DeepMind Control Suite from both state and image. 

Besides the novelty of using Fourier features raised by several reviewers (jV7a, aHKm,  Susc),  the major issue of this paper is that the claimed benefits of Fourier feature layer are indeed the main reason leading to better performances (N6iX and Susc). Specifically, the empirical study does not provide convincing support of claim, while the theoretical justification does not offer strong insight (Reviewer N6ix) and actually may not able to explain the empirical results. 

 - As reviewer Susc mentioned, the motivation is clearly justified: **i)** reduces noise; **ii)** the noise is the major performance killer in RL.


 - The difference between vanilla MLP and Fourier feature is in fact different parametrization families. With square loss in Eq. 4, the optimal solution actually is $E[y|x]$ if the parametrization is realizable. All parametrization is actually able to reduce the noise. How can the author guarantee the superior comes from uncertainty canceling, not **approximation error** reduction is never discussed. 


 - In the theoretical analysis part, the author calculate NTK, which relies on the assumption in the **limit case** and the neural network weights are not far away from initialization. In the experiment, only around 1000 random feature is used. As many recent papers [1, 2 and to name a few] suggest the situation is different in finite case. With this gap in mind, it is not clear to me the phenomena from kernel can be straightforwardly extend to random feature. To support the claim, the author may first compare the NTK vs. random feature. 


- The authors also provide the contraction proof for kernel feature. **i)** this proof is not clear to me whether the relu layer is used; **ii)** in fact, how to characterize the benefits of the contractition vs. noise cancellation should be discussed.

Let me quote the suggestion from Reviewer Susc "validating all hypotheses should be a main feature of papers like this one, not just showing improvements." As a scientific paper, we should make careful claim with supportive evidence. The paper indeed reveals some interesting phenomena, but with some overclaims without convincing support. 

Minor:

The term "functional regularization" might not be appropriate and not aligned with its vanilla meaning. In this paper, it is in fact parametrization family selection. 

Since most of the reviewers agreed to accept this paper, I will still recommend acceptance for this submission. However, I would suggest the authors can revise the paper,  taking the comments in reviews and meta-review into account. Specifically, reduce the over-simplified justification and the misleading claim, just make the claims which the empirical evidence can support to avoid the potential misleading of the whole community. 

[1] Fort, Stanislav, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel." arXiv preprint arXiv:2010.15110 (2020).

[2] Allen-Zhu, Zeyuan, and Yuanzhi Li. "What can ResNet learn efficiently, going beyond kernels?." arXiv preprint arXiv:1905.10337 (2019).
Decision: Accept (Poster)