Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper presents a method to detect out-of-distribution or anomalous data points. They argue that Fourier networks have lower confidence and thus better estimates of uncertainty in areas far away from training data. They also argue for using “large” initializations in the first layers and sin(x) as the activation function for the final hidden layer.

The paper does not seem to have any significant logical reasoning on why their specific architecture works, but "describes" what they did. It is not clear what the novelty is, besides that they found an architecture that seems to work. Additionally while Fourier networks have lower confidence, that does not necessarily mean they are more accurate estimates of uncertainty. However the reviewer does acknowledge that the estimates are mostly likely better than ReLU networks that are well known for having terrible estimates of uncertainty.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
I have read the reviews and the comments. Overall I am still positive about the paper and I have confirmed the rating.

======================

This paper proposes a method for the uncertainty estimates for Neural Network classifiers, specifically out-of-distribution detection. Previous methods use an ensemble of independently trained networks and average the softmax outputs. The authors investigate this method (ensembles of ReLU networks) and observe three fundamental limitations:
“Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classification (CONSTANT FUNCTIONS ON THE TRAINING MANIFOLD).

To mitigate these problems the authors proposed the following:

- Changing the activation function of the last hidden layer to the sin(x) function, and they claimed that this is going to guard against overgeneralization.
- Use larger than usual initialization to increase the chances of obtaining more diverse networks for an ensemble.
- They claimed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets.

The paper addresses an important problem, out of distribution detection, by proposing a Fourier network which is somewhere between a ReLU network (small initialization) and a nearest neighbor classifier (large initialization). The authors claimed that this leads to an out-of-distribution detection which is better than either of them.

The paper is well written and easy to follow. The authors did an interesting and precise investigation in how to force the confidence score to decay like a Gauss function by proposing to use the Fourier transform of such a Gauss function. By doing so they get the advantage of ReLU (ability to generalize) and prevent the network to become arbitrarily certain of its classification for all points. However, the authors claimed that when they switch the activation function to sin(x) the increase of |x| will usually stop at the first maximum or minimum of sin which is (around \pi/2). However, the authors did not explain how they get this result (i.e the value \pi/2). It would be interesting if the authors could show results for the case greater than or less than (\pi/2) to show the difference.
In addition, Figure 3 shows that the ensemble of ReLU networks is overconfident in most of the area, whereas the ensemble of Fourier networks is only confident close to the input, and in the discussion of constant function of their training manifold the authors discuss some example.

I would like to ask the authors: 

Did the Fourier networks learn the input distribution?
How are defined: usual initialization, small initialization and large initialization?

The experiment section is adequate. However, it would strengthen the paper if the authors compared against other approaches such as: 
- Predictive uncertainty estimation via prior networks, NeurIPS 2018.
- Generative probabilistic novelty detection with adversarial autoencoders, NeurIPS 2018.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: ** Updates after rebuttal **

I thank the authors for the response, though I am still skeptical about the evaluation of the method, which might be a result of heavy tuning and overfit to the chosen test sets. The proposed approach also requires more theoretical justification.

------------------------------------------------

I'm not an expert in this area but I do find this paper interesting. Though the name "Fourier networks" is a bit arbitrary because the proposed approach also applies to multi-layer networks where only the last hidden layer has the proposed change.

The extrapolation problem of ReLU networks is an interesting point. I don't know previous works that point out this for out-of-distribution detection but it's worth figuring out if this observation has been made in the adversarial robustness community.

I do have several concerns, summarized below:
* On page 3 the fourier transform is motivated by that RBF networks "do not generalize as well as ReLU networks". I doubt if there is any evidence for this argument.
* I have some issue understanding proposition 1: what is w_i'? Only w_i is mentioned before.
* The "Fourier network" is not defined explicitly in the paper, which makes it hard to understand the architecture/algorithm details. If I understand it correctly, it is only about changing the activation function of the last hidden layer and large initialization, with everything else the same as the ReLU networks?
* How does the magic number "\sigma_1 = 0.75" and "\sigma_2 = 0.0002" come from? Did you search it by looking at the test performance? Is the performance sensitive w.r.t. the two parameters?

I'm willing to increase my score if the authors addressed my concerns.


review_title: Official Blind Review #3

Meta-review: This paper presents a new method for detecting out-of-distribution (OOD) samples.

A reviewer pointed out that the paper discovers an interesting finding and the addressed problem is important. On the other hand, other reviewers pointed out theoretical/empirical justifications are limited. 

In particular, I think that experimental supports why the proposed method is superior beyond the existing ones are limited. I encourages the authors to consider more scenarios of OOD detection (e.g., datasets and architectures) and more baselines as the problem of measuring the confidence of neural networks or detecting outliers have rich literature. This would guide more comprehensive understandings on the proposed method.

Hence, I recommend rejection.


Decision: Reject