Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes Random Fourier Networks (RFNs), MLPs with random Fourier featurization within the first layer. RFNs are supposed to learn high-frequency components of the value function more quickly than standard MLPs. The authors present an initialization scheme for the Fourier layer that is better-suited for high-dimensional inputs. RFNs are compared to non-Fourier-feature baselines on high-dimensional environments.
main_review: Discussion period update: After reading the other reviews (note: no author response), my overall score has not changed.

-This work has limited novelty. The core architecture is basically pulled from existing work. One difference is a change in weight initialization, but the initialization schemes are not directly compared within the experiments. Additionally, if I understand correctly, both schemes still require a hyperparameter (with appropriate choice of hyperparameter, the schemes are the same). The other difference is that this work focuses on DRL.

-The baselines include a decent set of non-Fourier baselines. However, this work does not compare to architectures from Sitzmann 2020 and Tancik 2020 (though these works are cited). As a minor comment, including "dense+deep" and "wide" would be better, as would including more discussion on the number of parameters vs depth of different baselines.

-This work includes claims about "evading the spectral bias" and, in some sections, uses performance to "show" the efficacy of the proposed method. However, improved performance does not necessarily mean the bias issue has been reduced.

Other comments:

-The MLP example (paragraph 2) in the introduction is presented in an unclear way. Insufficient information is provided to show that the training process was not the reason for poor performance.

-The third paragraph of the introduction is better placed in a later section.

-For Figure 2 and Figure 7, showing the learned function does not convey the difference between the true and learned functions. Showing the absolute difference between the two would be more informative.

-Much of Section 2 ("Preliminaries") is not relevant to this work's contributions. This section should be substantially condensed.

-Reorganizing Section 3.1 would improve clarity. Additionally, the first paragraph largely repeats content from earlier in the paper.

-The Mountain Car experiment is better placed elsewhere in the text.

-If I understand correctly, when varying bandwidth, the number of parameters in the network also changes. This should be considered when performing the ablation studies. Additionally, the SIREN-style selection of width should be explicitly compared to the proposed method.

-Related Work is best placed earlier in the paper.

This work would benefit from another editing pass:

-"Given that a more gradient updates" -> "Given that more gradient updates"

-"as pointed in Fu et al." -> "as pointed out in Fu et al."

-"is called the spectral norm" -> "is the spectral norm"

-"in figure 4" -> "in Figure 4"

-"Xiavier-like" -> "Xavier-like"

-"deepr reinforcement learning" -> "deep reinforcement learning"

-"In figure 7e" -> "In Figure 7"

-"its performance vary a lot" -> "its performance varies a lot"
summary_of_the_review: This work overstates its novelty. Using sinusoidal activations on RL tasks is of limited novelty, and no comparisons are made to related methods / few changes are made that are specific to RL.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel parameterization for MLPs, Random Fourier Features (RFF), which amount to using a sinusoidal activation, and initializing parameters to capture different frequencies (by initializing $w_{ij}$ with a variance proportional to a bandwidth hyperparameter $b$) and phases ($b_j \sim U(-\pi,\pi)$).
This particular parameterization is motivated by the need, in RL, to learn value functions which have been found to potentially require high frequency components (whereas vanilla MLPs have a low-frequency/simplicity bias). The paper offers a very detailed exposition of why this is the case and of why RFFs can help.
RFFs are then used on 3 standard control problems, showing some improvement when using DDPG with (vs without) RFFs. The choice of bandwidth as well as the number of features required are tested.
main_review: Strengths:
- The paper tackles an important problem and has an exemplary exposition of the problem and the proposed solution.
- Its results suggest that much more could be done in RL to take into account the peculiarities of the functions being estimated to find good architectures (that may depart from their supervised learning counterparts).

Weaknesses:
- This particular parameterization is fairly close to prior work
- The empirical evaluation on "large scale" problems is limited

On the empirical evaluation, I think this is an important point since, as the authors point out, there is no study in this paper of the generalization properties of RFFs. This is particularly important since we know, e.g. from the SIREN paper, that sinusoidal features are particularly good at memorization.
This may be a fundamental limitation of generalization/function approximation. How much can we get out of generalization when high-frequency features are involved? Results on this question would be potentially very impactful--see for example Hooker et al. (this paper is very much orthogonal to the present work, no need to cite it, but is relevant to the general question of high-frequency things).

Characterising Bias in Compressed Models, Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, Emily Denton, 2020.


Comments:
- Be careful when using citations, some are missing parentheses around them or just the year. Inline citations where the citation is an object in the sentence (e.g. "Newton (1687) showed that X") should use \citet, whereas parenthesized citations should use \citep (e.g. "It has been shown that X (Newton 1687)".
- The text legends in Figures 8-11 could be larger
- An "ablation" is when something is removed. It doesn't seem like anything is removed in the first part section 4.3. Rather, different hyperparameters are tested to attempt to understand their impact.

summary_of_the_review: The pedagogical aspect of this paper is certainly an appreciable contribution. This paper also contributes to understanding how certain biases of DNNs can be dealt with in deep RL, but only within a limited scope. While it certainly refines our intuition and could help the field focus its research in the right areas, the novelty of the concepts introduced in this paper is fairly limited.

I think that, as it is, this paper meets the bar for acceptance, but its potential seems much higher.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discussed the problem of spectral bias in MLPs (i.e., dependence on the higher-frequency part of input feature is under-fitted) within RL domain. The authors first described the problem with toy examples, which clearly explained this problem. Then it was suggested to use random Fourier feature network (RFN) to transform the input to frequency domain and then feed to the network, to resolve this issue. 

Experiments demonstrated effectiveness of RFN to learn a complicated Q-function in low-dimensional state space. And it was also shown that RFN contribute to performance gain with DDPG on high-dimensional robotic tasks. The authors also did an “ablation study” (I would rather call it “sensitivity analysis” instead) to empirically show how the performance depends on the choice of hyper-parameters.

main_review: Pros:

The motivation is clear and sound. The problem discussed is well-defined and the introduction to the problem makes sense.

The proposed method RFN was shown to be effective on both toy tasks and high-dimensional robotic control tasks with DDPG. In particular, RFN +DDPG show a significant performance gain in the challenging humanoid control task. Also, RFN can theoretically be applied to any neural network for function approximation.

Concerns / Questions:

It is impressive to see DDPG+RFN works on Humanoid in contrary with the original DDPG’s terrible performance. However, we also know that more advance algorithms like SAC or TD3 can also get much better performance than DDPG (similar to RFAC) without resolving the spectral bias issue. Does this mean that spectral bias is not critical to performance if other aspects of the RL algorithm are good? 

While the empirical results are good, the novelty is a bit weak. My understanding is that the authors borrowed the RFN from recent studies and made an investigation of the hyper-parameters so as to fit in RL. Please correct me if I have any misunderstanding.

For the RFF layer, it was written that “We find this initialization scheme work well across all domains, with minimum requirement on tuning, even when stacked as learnable units.”. I am a bit confused here. Are the weights and bias of RFF layer learnable or not in your implementations?

The format of references is not professional nor consistent. Some of them has hyperlinks and some not. Also, Please carefully confirm the published venue of each citation. For example, the Adam paper was published on ICLR, please do not use the arxiv version. 

For the MuJoCo experiments, I would like to see RFAC’s performance on other tasks such as hopper, walker2D, etc., if possible. These results may give people more understanding of RFAC’s effectiveness on (relatively) simpler and lower-dimensional control tasks.  

For the performance curves (figure 8-12). How many random seeds were tested? What was the shaded area (STD of S.E.M.?) In figure 9, the authors claimed that “its performance varies on Humanoid with different values of b”. But from the plot I cannot see clear statistically significant difference.


Smaller problems:
1.	The abbreviation TD needs to be explained at the first usage.
2.	"Note that DDPGtakes 1 gradient step" -> DDPG takes
3.	It is a bit weird to put Related Works in the end. How about putting it before the Discussion?
4.	Comma and period should be properly used in the equations.

summary_of_the_review: The paper discussed an important problem in RL that was previously under-investigated. Overall, I think the paper successfully addressed it by suggesting the usage of a random Fourier feature network. However, the novelty is limited since the main contribution borrows the methodology in existing papers, together with the aforementioned concerns, I am not convinced that the paper meets the acceptance criteria.

### Post-rebuttal
I really appreciate the authors' work for providing new results on DMC tasks. The empirical results are much more convincing and comprehensive now. I would like to raised up my score because I think the current results have successfully supported the motivations. However, now the only and the largest concern is that necessity of a new round of review for such a major revision (unfortunately this is not possible now). After all, I set my recommendations to weak accept since I cannot (and should not) access the newest revised version
review_title: null

Meta-review: The paper points out an interesting and, to me unexpected, problem when learning Q-functions to do with spectral bias. Figures 1 and 2 are quite striking. The diagnosis and proposed solution elegantly combines ideas from NTKs and NeRFs. The proposed random Fourier actor-critic performs well in practice. The main problem reviewers had in the end is that the authors added substantial new empirical results too late to review thoroughly.
Decision: Accept (Poster)