Fourier networks for uncertainty estimates and out-of-distribution detection
Authors: Hartmut Maennel
Alexandru Țifrea
Venue:  (2020)
Keywords: Fourier network
out-of-distribution detection
large initialization
uncertainty
ensembles

Abstract: A simple method for obtaining uncertainty estimates for Neural Network classifiers (e.g. for out-of-distribution detection) is to use an ensemble of independently trained networks and average the softmax outputs. While this method works, its results are still very far from human performance on standard data sets. We investigate how this method works and observe three fundamental limitations: "Unreasonable" extrapolation, "unreasonable" agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out-of-distribution inputs, but do not contribute to the classification. To mitigate these problems we suggest "large" initializations in the first layers and changing the activation function to sin(x) in the last hidden layer. We show that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-
distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).

Under review as a conference paper at ICLR 2020
FOURIER NETWORKS FOR UNCERTAINTY ESTIMATES AND OUT-OF-DISTRIBUTION DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
A simple method for obtaining uncertainty estimates for Neural Network clas- sifiers (e.g. for out-of-distribution detection) is to use an ensemble of indepen- dently trained networks and average the softmax outputs. While this method works, its results are still very far from human performance on standard data sets. We investigate how this method works and observe three fundamental lim- itations: "Unreasonable" extrapolation, "unreasonable" agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some to the classification. To mitigate these problems we suggest "large" initializa- tions in the first layers and changing the activation function sin(x) in the last hidden layer. We show that this combines the out-of-distributie nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of- distribution detection on standard data sets MMNISTfishionNNSSnoMMMM SVHN/CIFARIO).
inputs, but do not contribute
behavior from
INTRODUCTION
When a neural network classifies inputs, we often need to have some measure the uncertainty involved in the prediction. In particular, in safety critical applications we would like to know when the classifier received an input that is different from the inputs it has been trained on in such cases it could be dangerous to just use the "best guess". instead may want to choose some safe action or involve a human.
One solution for this "out-of-distri" distriibiutiin detection" problem could be to add label "unknown" and augment the training set with inputs that do not belong to any of the labels of the classifier and which get the new label "unknown". However, the danger is that the classifier learns the particular type of "known unknowns" was trained on and does not generalize to the "unknown unknowns".. To avoid this, we treat -ut-of-distribution detection as "one class classification" problem, i.e. we want to learn the input distribution without help of out of distribution inputs.
The simplest approach is use the softmax output of neural net as confidence measure. It is well known that the softmax output tends to be "overconfident", so we cannot interpret it directly as a probability for the chosen class. However, it still tends to be more confident for samples from the correct distribution than for outliers, so setting a threshold for the softmax output can to some extent distinguish between "in distribution" and "outtof-distriiuttio"" samples. This is the "baseline method for outlier detection suggested in (Hendrycks & Gimpel| 2017).
In (Lakshminarryyaaa et al. 2017) this baseline improved by using an ensemble of classifiers (and adversarial training): Often each random initialization of neural network gives overconfi- dent classifier, but different classifiers disagree averaging the softmax output over an ensemble of classifiers then gives improved signal.
Other, more computationally demanding approaches are Bayesian Neural Networks Neal 1996). (Barber& & Bishop 1998), (Blundell et al. 2015)). nearest neighbor methods (e.g. (Mandelbaum & Weinshall|2017 Jiing et al. 2018). (Papernot McDaniel] 2018), (Frosst et al. 2019).. For some further (less closely) related works, see appendix|A

---

Under review as a conference paper at ICLR 2020
We investigate limitations of the "ensemble averaged softmax output" method, this seems to hit the sweet spot of being easy and fast, but still giving good results. To overcome the observed limitations and further improve this method, we propose two changes:
Changing the activation function of the last hidden layer to the sin(x) function.
(We will see in the next section how this can guard against overgeneralization)
Use larger than usual initialization, to increase the chances obtaining more diverse net- works for ensemble.
In the next three sections we go through three observed problems with ensembles of ReLU net- works and explain how these changes mitigate them, and in the following sections we evaluate the suggested changes standard data sets (MNIST, CIFAR 10, etc.).
We evaluate this uncertainty mainly with view to antor--distribution detection: we have meaning- ful uncertainties, the classifier should in particular "know when it does not know the answer". This means should display uncertainty for inputs that do not "belong to the same type of input" as the training examples. By selecting different thresholds for the uncertainty, by recording for each threshold the fraction of samples below the threshold for "in distribution" (x) and for "out-of "out-of-distriburs of-distribution" (y) inputs. In particular, figure I gives this evaluation for method against the standard ensemble methods on two standard data sets.
plot the ROC curve
1..
1.-
cISCTIIE recworoa
cuammbl relwerka)
stancard
standadd
00
m0
8.00
3.6
8.66
0.60
a
00
0.5
0.1
0.11
0.1
Figure I: ROC curves for networks ensembles trained on MNIST, evaluated notMNIST (left) and fashionMNIST (right). w-axis: MNIST samples recognized as not belonging to MNIST, y-axis: notMNIST samples recognized as belonging to MNIST.
2 PROBLEM: "UNREASONABLE" EXTRAPOLATION
We first look at the simplest possible example of classification: The training set consists only of one point zo IRd which has label "1" (we assume there is another label, for which we do not have sample). Training standard ReLU networks with one hidden layer just one point 2g = 2 R get something like figure 1: Gradient descent finds a simple functions describing the data, which will (with very high probability) be an increasing function of z. So for larger values of z the network will become arbitrarily certain that the label of x should also be "1". (See appendix Cf more details.)
Figure 2: Logits and probabilities from 10 randomly initialized ReL.Unetworks (left) and Fourier- networks (right). æ-axiis: Input x, y-axis: Estimated logits/probabilitss for input x.
has been observed in (ein et al. 2018 that this true in general: Given any ReLU network trained some data for classification problem in Rd it will describe piecewise linear function. we look at the values of this function along a ray to infinity, this will be a linear function when we far enough from the origin, and in general this means the network will become arbitrarily certain
2

---

Under review as a conference paper at ICLR 2020
of its classification for all points the ray far enough from the origin. This is clearly not desirable, we would rather get a low confidence when we are far from the training data. (Here "far from the training data" is a convenient aspect that can easily be checked mathematically. Since the data are often bounded, it is actually more relevant in practice that this behavior starts as we leave the range of the training data. As illustration, figures and
Going back to our simplest possible special case, what should our confidence be that other points belong the same label? Of course, there is no unique "correct" answer, but a reasonable model could be that our confidence decays like a Gauss function around this point To, i.e. is proportional to
Eo /20"
(1)
for some that defines the scale on which our confidence decays.
One way to force such decay would be to use directly Gauss functions (equation I activation functions, which leads to RBF networks which we could consider as form of nearest neighbor methods. While they recognize reliably points that are not close to one of the mgg in (equation I they do not generalize as well as ReLU networks (see appendix G.. so we will use more indirect approach.
Using that the Fourier transform of such a Gauss function is again of this shape, we can write equation 1 also as an expectation:
1a-50 2/20" E. ((. cos (ui (I {o)]]
where E is the diagonal covariance matrix o --... When we approximate the expectation by a finite sum over N samples Wi from (0, Σ) we get an expression of the form
f(x) = Σ ui cos(ut; I #+ bi)
where u, = 1/N and the b; are chosen such that the cos( To + bb) = 1. This function can be interpreted as the output of neural net with one hidden layer of N neurons with activation function cos(r) (we will later shift the phase b, and use sin(x)). (The output is one dimensional since we only care about the logits for the one label that appears our training set.) This is the main idea in the proof of the following result (see appendix C for details and proof):
Proposition Assume the weight vectors V, are sampled from a normal distribution (0, Σ) and then fixed, the numbers b; and u; are sampled independently from some distribution (not identically zero) with finite second moments (e.g. a normal distribution) and then trained on one data point zo with label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network's 's output converge to function that is approximately proportional to - 22/2 This approximation becomes exact when the number of neurons approaches infinity, or when we take the expected value.
We also have for general training sets (see appendix C for proof):
Proposition 2: Under the same idcalized conditions (infinite number of neurons, weights Wi; frozen), but arbitrary finite training set, and assuming the training converged, the function goes for [ズ] + 00.
To get an exact formula, we froze the weights W. In practice we would train them as well, but at least experimentally we do not see a difference when the Wi; are large: They do not get changed by large amount, and the result still looks the same. An example is given in the two plots of the right side of figure] 2 When we train on training set of a few points, we see experimentally that (under the same idealized conditions: Large number of hidden neurons, frozen wT,) we get an
approximation to weighted sum of Gaussian functions at these points (see appendix E So it may seem we constructed a complicated approximation to nearest neighbor classifier.
We now switch the activation function to sin(r), this makes no difference for the function if we compensate by shifting the b, by x x/2, but the region of "small initialization" |,ū,], b, P 0 is now similar to linear network: The input to the hidden neurons i will be w. I 2s O. and sin (wū, .f) se w; H. As training progresses, |x] usually increases. For ReLU activation, this increase is not bounded (if we the side ui; H > 0), but for sin the increase will usually stop at the first
3

---

Under review as a conference paper at ICLR 2020
maximum minimum of sin, i.c. around ~ 2 (or also different minimum/maxxmmmm if start from higher initialization, see appendix P for experimental examples). This means that on the training features (i.e. the output of the sin(æ) neurons in the last hidden layer) get large value by tuning the input to around x = + which is difficult to achieve "accidentally" for input that not in the training By contrast, for ReLU (or tanh) activation, the inputs only have achieve any high value.
We will call networks in which the last hidden layer uses the sin(2) activation function "Fourier networks".
So the Fourier network somewhere between ReLU network (small initialization) and nearest neighbor classifier (large initialization). In particular for intermediate initializations it generalizes better than nearest neighbor classifier, but still detects more outliers than a ReLU network, which leads to an out-of -iitiii-ittiittoonnn detection which is better than either of them, as we will see later (see in particular appendix G for comparison nearest neighbor classifiers).
PROBLEM: "UNREASONABLE" SIMILARITY OF NETWORKS IN THE ENSEMBLE
In section 2 we considered only network, and saw that ReLU networks become arbitrarily certain far from the training set. However, averaging the softmax output over an ensemble of networks could fix that: Even if each network is very certain about its prediction, the ensemble could give low confidence the different networks disagree. This indeed happens, but less often than one might expect. To illustrate this, we generate the input in figure which has 20 clusters of points with labels "red", "green", or "blue".
Insemble Networks
Ersemble of Networics
Figure 3: A 2-dim classification problem. Left image: Input, color indicates label. Following images: Average output an ensemble of networks.
We take an ensemble of 50 networks that were independently trained on the same data set, and use the average "probabilities" for the 3 possible labels as RGB values for the corresponding pixels. In the second image (ReLU networks) the ensemble is still overconfident in most of the area, whereas on the right the ensemble of Fourier networks is only confident close to the input.
The reason for the overconfidence of the ReLU ensemble is that the individual networks tend to agree even far from the sample points. On the other hand, the Fourier networks find different sparse Fourier interpolations to describe the input, so their output only agrees close to the inputs, see figure For another 2-dimensional example, see also appendix
As higher dimensional illustration, we train 50 ReLU networks with standard initializations on MNIST images with labels from 0 to 4. Evaluated on MNIST images for labels to 9, for quarter of all images all 50 classifiers agrec the same label (which is of course wrong, since it is of known labels 0 - 4). If we use smaller initialization, all 50 classifiers agree even for the majority of inputs.
We can understand why this similarity of independently initialized networks is particularly prevalent for "small" initializations: We expect that Gradient Descent finds local optima "close to" the initial point. If we start with high random initializations, we may have better chance that they are far apart and converge different solutions, but for small initialization the networks likely converge
4

---

Under review as a conference paper at ICLR 2020
Figure 4: Predictions of 4 randomly initialized networks. Upper row: ReLU networks, lower row: Fourier networks.
the same closest solution. This can be made precise for ReLU networks with one hidden layer and "infinitesimal" initialization, see appendix H
As method between RBF networks and Fourier networks, we can also using Gauss function as an activation function. While this does show some of the benefits of Fourier networks, extrapolating with Gauss functions predictably reverts to zero, whereas the sum of periodic functions become essentially random far from the training distribution, resulting in far more diverse ensembles, see appendix [ figure 10
PROBLEM: CONSTANT FUNCTIONS ON THE TRAINING MANIFOLD
The third problem we see may be general problem for using networks trained as classifiers also for -iiit-idiittitttinn detection: If we have a feature (function) that is constant on the training distribution, but takes other values out-of -fit-ii-iiiittiinn we use to detect out-of -istrribiitiiiiiionn inputs. However, when we train for distinguishing between different parts of the input distribution, such feature is useless and may be dropped during training.
How do such features arise? One simple possibility is that some part of the input is constant. For example, for MNIST images, the corner pixels are always white. But in notMNIST fashionM- NIST this is not always true (see e.g. the "E" in figure 17). and thus this gives an easy way to detect images MNIST.
Another possibility is that there may be different ways to come to the conclusion that an input gets certain label L. For example, for MNIST we usually can already guess the input has label "0" when we only see the top or the bottom half of the image. Let us say features f and get value their clues to detect label L is present in the image, and are 0 otherwise. Then f f- g is always on the training set (as they are if and only if the image is of label L), but it may take on different values out-oo-distribution. (For some preliminary experiments with forcing such different features into separate networks, see appendix D).
Why would such features be "dropped"? we start from small initialization, only weights that are useful for distinguishing between labels will be increased. So c.g. if the corners always white, there is no incentive to change the weight that connects a corner pixel to the next layers, and if we start from small initialization, these pixels will never contribute to the end result.
So an obvious mitigation is to use larger initialization: Although the weights will still not be changed, the pixel will likely have an influence on the end result, and in different networks of the ensemble this influence will be different. So if the pixel is not white, this may lead to the different classifiers disagreeing about the label.
5

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS: TWO HIDDEN LAYERS, FULLY CONNECTED
We follow the example given in [Laksmmnnaraynnan et al. 2017) and look at fully connected net- works with two hidden layers of 200 neurons, which we train MNIST. For the baseline network, we use the model presented in Laaksminarayaaaan et al. 2017), namely ensembles of networks with ReLU activations and the standard initialization of the weights (a = /2/n with n = 784 for the first layer, n 200 for the second layer). For the Fourier network we change the activation function of the deeper layer to sin and use 0 = 0.75 and Ø2 = 0.0002.
We train the networks the training set of MNIST (60000 images), and evaluate them both the test set of MNIST and on other data sets, here we use noMMNIST (see (Bulatov and MNSST (see (Xiao al., (017)) as the "outlier" data sets. For other data sets see appendix The variation of these
K The ROC curves that are obtained this way are presented in figure | curves captured by figure 15 in appendix
While using an ensemble of ReLU nets gives a very substantial improvement over using one ReLU net, the result is still not close what humans would achieve: The classifiers used here have some 98% accuracy, if we allow not recognizing an image as MNIST digit in 2% of the cases, we might expect to recognize that the printed letters of the alphabet are no handwritten digits in most cases, but in the above graph 2% error rate for MNIST corresponds to "only" 80% success rate on notMNIST. However, the ensembles of Fourier networks recognizes almost all notMNIST images if we allow 2% error rate MNIST.
In appendix M figure 22 shows the corresponding results for training on the more challenging data set fashionMNIST, the results are qualitatively the same.
In the literature, the performance of such "outtof-diittriuttio" detection is often measured by the area under the ROC curve, in Table D are numbers for MNIST vs. notMNIST.
Paper
Method
ID set
OOD
AUROC
(Hendrycks & Gimpel| 2017)
baseline
MNIST
nOtMNIST
93.2%
Liang et al. 2018
outlier exposure
MNIST
notMNIST
98.2%
ours
Fourier networks
MNIST
notMNIST
99.9%
Malinin Gales
2018)
outlier exposure
MNIST
FashionMNISS 5-9
96.5%
ours
Fourier networks
MNIST
FashionMNIST 5-9
99.7% %
Table 1: Area under ROC curve for entropy as the OOD detection score.
Some of the approaches in Table I use outliers for training/tuning the model, so they are dependant on how similar the outliers used for training are those used at test time. For more details about the baselines we refer the reader to Appendix R
6 EXPERIMENTS: EFFECT OF INITIALIZATION VS. ACTIVATION
we use only one hidden layer, we can only vary the for weights between the input and the hidden layer (we keep the initialization of the output layer at the standard = 22/n). This makes it easy to plot how much of the effect is due to "high initialization" vs. the choice of activation function.
We plot the area under the ROC curve in dependence on initialization: On the r-axis we plot log(ar), on the y-axis the negative logarithm of the "error" := area above the ROC curve.
The error bars correspond to two standard deviations (we has used randomly sampled ensembles of 10 networks out of 50 pre-computed networks).
We see in the first three images of figure 5 that notMNIST images are the easiest to detect, and flipped digits from MNIST are the most difficult, but each case the Fourier networks clearly outperform the ReLU networks. One reason why notMNIST is particularly easy probably is that its images often can be recognized by just looking at pixels close to the corner:: MNIST these pixels are always white, but in notMNIST often not. To eliminate this shortcut, also consider "masked" versions of notMNIST and fashionMNIST in which these pixels are set to white (see appendix K for details). The rightmost image in figure 5 shows the corresponding graphs for the

---

Under review as a conference paper at ICLR 2020
Figure 5: Influence of initialization and activation function
a--axis: log(a), y-axis: - log( (Area above ROC curve), higher is better Left to right: notMNIST, fashion MNIST, flipped MNIST, masked notMNIST
masked version of notMNIST, figures 20 and 21 in appendix L for masked fashionMNIST and images in which each pixel is independently sampled from MNIST images, so each pixel has the same value distribution in MNIST. Even with the "looking at the corners shorteut" disabled, the Fourier networks still perform well.
The vertical line corresponds for all test sets. This corresponds to standard deviation of around 1.4 for the input to the sin(x) activation function (for our encoding of the MNIST pixels, which we scale into the interval [- 11,1)) 1).
0.2, which seems to be close optimal for Fourier networks
We also see that "larger than usual" initialization alone already helps for ReLU networks, but using the sin(x) activation function gives a significant additional benefit. (See also appendix N for more general initialization schemes.)
7 EXPERIMENTS, CONVOLUTIONAL NETWORKS ON SVHN VS CIFARIO
The same phenomena described in the previous section for fully-connected networks apply for con- volutional networks well, thus allowing outoof-disttibutiinn detection on more complex image datasets. To illustrate this, we train ensembles of classifiers on the Street View House Numbers dataset (SVHN) (Netzer et al. 2011] and use CI- FARI0 (Krizhevsky et al. 2009) as out-of-distribttion samples. Once again, we compare against baseline of ReLU networks, first described as performing well on this task in (Lakshminarayanan
reap the benefits of using Fourier networks for
et al. 2017).
Three different convolutional architectures have been considered, with increasing complexity: i) 2-layer convolutional block followed by a fully-connected layer with ReI.U activations and one with Sine activations dubbed SimpleCNN, ii) network containing the first 4 layers from the classic VGG16 architecture (Simonyan & Zisserman 2015) followed by a sine fully-connected layer ferred to as Shallow VGG, and iii) the full-fledged VGG16 network. Appendix contains the exact hyperparameter values used for each network.
Table 2 presents the best results obtained using this setup for both the ReLU and Fourier networks. disttr--isttitution samples detected using approaches, by selecting threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble (presented in appendix Q or for the entropy of the ensemble's prediction (presented in Table 2 For evaluation, we used the area under the ROC curve (AUROCT) as well as the false positive rate 80% true positive rate (FPR80L). The arrows indicate how the metrics change as the model gets better (i.e. t if larger value is better and J if a smaller value is better). Additional results are presented in [
One other way in which one can assess qualitatively the performance of an out-of-distribution detection approach is by inspecting the distribution of the entropy the softmax outputs over in-distribution and out-of-diitrribttinn samples. Ideally, the entropy for in-distribution samples stays close to (meaning that the model is certain about its predictions) while the one for out- of-distribution samples pushed as far from 0 possible (the maximum entropy that 10-class classifier can achieve is log 10 == 2.3).
Figure 6 shows how Fourier networks compare regular ReLU networks for the Shallow VGG architectures (with = 0.01 used as the standard deviation of the initialization distribution of the last layer). The Fourier networks make lower certainty predictions about distribbtioonoo samples.

---

in- Under review as a conference paper at ICLR 2020
Network
Activation
AUROCT
FPR801
ReLU
93.2%
10.3%
Simple CNN
sin(x)
96.6%
5.17%
ReLU
93.5%
9.97%
Shallow VGG
sin(x)
95.8%
6.71 %
ReLUU
95.7% 95.9 %
6.73%
VGG16
sin(x:
6.51 %
Table 2: Out-of-distribttion performance on CIFARIO, for ensembles of models trained on SVHN.
The histograms for the ReLU networks are similar to the ones reported by (Lakshminarayanan al., 2017) the same task using ensembles of networks, with without adversarial training. The curves correspond to ensembles I, 5, and models.
Figure 6: Entropy histograms for SVHN as in-distribution set (top row) and CIFARIO as out-of- distribution set (bottom row). Comparison between using sin(z) (right-hand side) or ReLU activa- tions (left-hand side) on the last layer of Shallow VGG network.
CONCLUSION
We suggested using the sin(x) activation function in the last layer, and larger than usual weight initializations to mitigate three problems we saw in using ensembles of ReLU networks for out-of- distribution detection:
"Uneeasonalle" extrapolation:
While ReLU networks tend to get more confident away from the training set, Fourier net- works get decreasing confidence in the average.
"Unreasonable" agreement between the networks in ensemble:
Larger initialization make the networks more diverse, and the mixture of sin(x) func- tions with different frequencies makes the networks more random away from the training set.
Filtering out of features that distinguish the training distribution from some out-of- distribution inputs, but not contribute to the classification:
Larger initialization can preserve such features, they do not influence the in distribution classification, but create diversity in the ensemble when applied to put.
We showed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of distrribubtio detection standard data sets MMNIST/aahiaMMNNITTooNNNNNS SVHN/CIFAR10),
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder hased anomaly detection using reconstruc- tion probability. In Technical Report. SNU Data Mining Center, 2015.
D. Barber and Christopher Bishop. Ensemble learning in bayesian neural networks. In Generaliza- fion in Neural Networks and Machine Learning, pp. 215-237. Springer Verlag, January 1998.
C. M. Bishop. Novelty detection and neural network validation. In IEEE Proceedings-Visiiee Image and Signal processing, 141(4), pp. 217-222, 1994.
Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
Charles Blundell, Julien Corebiie,, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In https//urri..or//uu///555...444 205p://///////110000044...4 2015.
Yaroslav Bulatov. notmnist dataset. yaroslawvh.boosspoo.com. 2011.
URL http://
waroslavvb. blogspot. com/ 201// cttpc////1///////cccaaaataaaaaae
Emmanuel Candes, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal
reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theor, 52(2): 489-509, February 2006.
Hyunsun Choi, Eric Jang, and Alexander A. Alemi Waic, but why? generative ensembles for robust anomaly detection. In hutps://rziin.rg/a//1111..0010.0
2018.
Terrance DeVries and Graham W. Taylor. Learning confidence for dii-trrddittiitoo detection in neural networks. In http:://arsisoor//////0000.00000 0.00://.../.8020.055.. 2018.
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In https://arzii.or/aa//////0000000
2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
A. C. Gilbert, S. Guhay, P. Indykz, S. Muthukrishnan, and M. Strauss. Near-optimal sparse fourier representations via sampling. In STOC'02, May 19-21, 2002, Montreal, Quebec, Canada, 2002.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Matthias Hein, Maksym Adnriushchenko, and Julian Bitterwolf. Why ReLU networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In arxiv, 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and distr--distriuution examples. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust not trust classifier. In Advances in Neural Information Processing Systems, pp. 5546-5557, 2018. 2018,
Mark Kliger and Shachar Fleishman. https//tazissorg/aa///1122. 10560, 2018
Novelty detection
with gan. In
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). http://wwww.c..orrmmo.. edu/ kriz/cifar.hmml 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estinfdeeation using deep ensembles. In 31 Conference on Neural Information Processing Systems (NIPS), 2017.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidennce-allibratedd classifiers for detecting out-rf-distributiinn samples. In http://arxi.orr/dbb//11...222 1.0922. 2018.

---

Under review as a conference paper at ICLR 2020
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Machine Learning (ICML), 2018.
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. In https://arriivsor//a///// 100.08030. 2018.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Pro- ceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7047-7055 USA, 2018. Curran Associates Inc. URL http::/ttp/l...... acm. org/ citations
3327808
Amit Mandelbaum and Daphna Weinshall. Distance-based confidence score for neural network classifiers. arXiv preprint arXiv: 2017.
Marc Masana, Idoia Ruiz, Joan Serrat, and Antonio M. Lopez Joost van de Weijer. Metric learning for novelty and anomaly detection. In https://arivv.rg/cc//1888.. http://bb/1800054922,, 05492, 2018
Boris S. Mityagin. The zero set of a real analytic function. In hutp::/auzzioorraa////112777 2015.
E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don't 't know? In arXiv: 0rXhv:1810.09136. 2018.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer, 1996.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NISS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL
hoosernumbers//ppp20111 www.shuuuueuesseoee. http:////..pp...ooooouurrreeeee
Nicolas Papernot and Patrick McDanicl. Deep k-nearest neighbors: Towards confident,ineerpetabble and robust deep learning. In hatp::/arraioorr/abb///...... 2018.
Giambattist Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: sine as activation function in deep neural networks. In http::/ppeeenreeeeeeeeee//ddddds 2017.
Nick Pawlowski, Andrew Brock, Matthew C.H. Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. In hupps:/arzivvrr//u////11...1 2018
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for catrof-distribution detection. In https://arxiv.or//bb/// 02845, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: central 2018.
limit theorem. In
J. M. Sopena, E. Romero, and R. Alquezar. Neural networks with periodic and monotonic activation functions: a comparative study in classification problems. In 1999 Ninth International Conference pp. 323-328 vol.1,
Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume
1999.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms. CoRR, htts//708.07744 2017.
Y. Bengio LeCun, L. Bottou and P. Haffner. Gradient- learning applied to document recog- nition. In Proceedings of the IEEE, 86(11), pp. 2278-2324, 1998.
Pourya Habib Zadeh and Reshad Hosseini. Deep-rbf networkds revisited: Robust classification with rejection. In https:/arriioorggaa/////22/.111 2018
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX A: MORE RELATED WORKS
Related to (Lakshminarayanan et al. 2017) is the suggestion of "MC dropout" (Gal & Ghahramani 2016), here the ensemble of networks is replaced by one network, from which different predictions are produced by Dropout at prediction time. This is faster in training and evaluation, but in general not as effective as training the networks independently.
Another method that is also suggested in [Lakshminarayanaa et al. 2017) is adversarial training, but its effect seems to be small compared to the effect of using ensembles instead of single networks.
Using RBF layer on top of a normal network was first suggested in Y. LeCun & Haffner 1998
and is further investigated in (Zadeh & Hosseini 2018).
While we are avoiding the use of "outliers" as training input, it can be helpful. A recent work in this direction is (Hendrycks et al. (2019). "Interesting outliers" can also be generated by a GAN, see (Lee et al. 2018 and (Kliger& & Fleishman, 2018).
We focused on the use of uncertainties for -iistiibiititttoonn detection, and for this purpose it does not matter whether the "probabilities" are really correct (as long they higher for in distribbution than for out-of- distribution). However, if one wants "correctly scaled" probabilities, one can try to learn the right scale from the training data, see (Pawlowski et al. 2018).
We used the classifying networks and their "probability" output also for aaa--clas--clasffcaatiii as this is the computationally cheapest solution. But another intuitive approach is use generative model to produce a probability that an input belongs to the training distribution (Bishop) 1994). However, (Choi et al. 2018)) and (Nalisnick produce higher probabilities for out of distribution inputs. An interesting method that fixes this problem was recently described in (Ren al. 2019).
al.) 2018)) showed how this can fail and actually
For more works related to (x) as activation function, see the next section.
B ACCURACY AND TRAINING WITH SIN AS ACTIVATION FUNCTION
We did not investigate in detail the general behavior of the (in(s))nnuneeee as an activation function with respect to the usual accuracy and training behavior, but in our experiments, did not see any significant difference to (x)) networks inside the training distribution.
For example, the table below shows the classification accuracy of ensembles of 5 ReLU networks and Fourier networks on the SVHN test set. For each case, the accuracy is reported for the ensemble with the best diitrii-iitiiitoonnnn detection performance (the same that are presented in Table
Network Simple CNN
ReLU network
ReLU ensemble
Fourier network
Fourier network ensemble
89.38%
91.9%
89.50%
91.9%
Shallow VGG
89.10%
90.7%
89.60%
90.8%
VGG16
86.21%
90.6%
86.47%
90.5%
Table 3: Classification accuracy of individual networks (averaged over 5 instances) and ensembles of 5 models on SVHN.
These observations are consistent with the remarks in chapter 6.3.3. "Other Hidden Units" in (Good fellow al. 2016).
is usually said that Re works much better than other activation functions for deep neural nets, but that is less relevant for since we only use sin(2) in the last layer.
Nevertheless, there may be special cases in which sin(z) behaves better or worse than ReLU, as has been observed in e.g. (Sopena et al. 1999) and (Parascandolo et al. 2017).
Another way to formulate what a sin(2) (x) layer does, is that it computes a sparse Fourier approxima- tion. Sparse Fourier approximation has been extensively studied (e.g. (Gilbert et al. 2002). (Candes) ct al. 2006)), but in a quite different context: Usually the aim is to fully reconstruct a function from small amount of input/ output data. This is possible when the function is known to have sparse
11

---

Under review as a conference paper at ICLR 2020
Fourier representation. In context it is unlikely that our functions have sparse Fourier repre- sentation, and instead of trying to reconstruct the function the aim is rather to use different sparse Fourier interpolations to show the uncertainty we have in areas where do not have data.
C
PROOFS OF PROPOSITION I AND 2
We study the Gradient Descent dynamic neural net with
input Rd
one hidden layer with N neurons with activation function cos, connected to the input layer by fixed weights Vi E R2, 1, 2..,..,
one output (()) E R, connected to the hidden layer with weights uy IR.
The output of this neural net at the point ह is given by
N
f(r) = Σ u, cos(w;; E b;)
(We are using only one output dimension since for Proposition we will be training it only on one data point, so the outputs for other labels do not matter for now.)
We use the cross entropy loss, which is for a data set of one point To with the correct label given by
L = log
+ (1+c-1(80)
The derivative of L for some parameter 0 (i.e. one of the b; or us)
0
θ
00 (T)).
00
1 + eft)
With d, == wi To the last derivative is
0
[(o)
sin(d: b.)
0b.
f()) cos(d; + b:)
Du:"
We will assume "infinitesimal" learning rate, compensated by "infinitely many" steps, get differential equation for the parameters, and a learning "velocity" (t)). Then we get as differential equation
a
o<())
Øt (t)
(+ee((a) u4 sin(d; (.()))
0
a(1)
Ø1 (t) 1 1 ef(o) cos(d, + (t)))
We assume we can train as long as we want, and we are not interested in how fast this procedure is proceeding. This means we are only interested in the direction of the vectors in (equation 2 and can well look at the re-scaled vectors giving the differrnniall equation
Øt bx(()
- sin(d: + by (())
01. (t) = cos(d, + bt)))
(3)
These equations are independent for each neuron i, we we can fix a neuron i and drop the subscript. Introducing the function
c(t) cos(d + (()))
12

---

d + n Under review as a conference paper at ICLR 2020
2
-2
-8
d + 0
di + 2n
b
Figure 7: Vector fields for differential equations (equation
(equation
and using the dot notation for the time derivative we get the simple differential equation
u (1 -1-c7
(4)
(See figure for a plot of the corresponding vector fields.)
Lemma For JLahessue-Jalmost all (co, un) E [-1,1] x the solution curve of (equation 4 for &
starting at ( at (co: uo) will go either to +00) to
Proof of Lemma 1:
Since have symmetry (c, + (-c, u)) is enough to consider up 2 0.
For co E {+1} we obviously have e constant and ul)) = uo + + Co + +oo0
So will assume |c| < 1. For 0 co 1,0 uo the c(t) is increasing, hence must have u(t) 2 un + t..o. cp > +oc. So we can assume also 44 2 1. Then c(t) must be above the solution to the differential equation
j=
which has the solutions
22
((1) 2 +
so we also must have 2 c(t) 2 f(t) + hence c(t) + 1.
For co 0, 0 ug 0, co 0 move for small into the domain 0, 0,
know that for them as well we have e(t) 1, utt) + toc. By symmetry, for tgg 0, Co < 0 we will have c(t)
u(t)
For co = 0, dig = 0 the functions c, a are constant 0, so this a point we have to exclude.
So the remaining part is -1 0, ug- From (equation 4 see that in that case > 0 and
O. So a solution curve starting in the quadrant- - < co < 0, 2n has to leave this quadrant by either
13

---

Under review as a conference paper at ICLR 2020
1. Going through point 0, u > 0, (light blue in figure 7 or
2. Going through point c << 0, t 0, (light red in figure 7.
3. approaching 0, (red line in figure 7.
We have already seen that the first two possibilities lead to c(t), u(t) - +00 and c(t), u(t) - -1,-00 respectively, it remains show that the points that will approach
0, = are a null set. In fact, we show that for each -1 < Co < 0 there can be at most one un with that property: From (equation 4)
that the tangent to a solution curve through point (cn, ug) has slope u c/(1 - c). If we look at the vertical distance between two solution curves in our quadrant, this means that with increasing their distance also increases. Therefore, two curves with positive distance at cannot converge both to (0, 0), and for each given there can be at most one uo in our quadrant such that (co, uo) belongs to the third above.
With Lemma 1 we can now prove:
Proposition 1: Assume the weight vectors vē; sampled from normal distribution (O, Σ) and then fixed, the numbers b, and u, are sampled independently from some distribution (not identically zero) with finite second moments (c.g. a normal distribution) and then trained on one data point žo with label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network's 's output converge to function that is approximately proportional to 53//2/ This approximation becomes exact when the number of neurons approaches infinity, or when we take the expected value.
Proof:
We assume that we have trained equation for some time T long enough such that we are in the situation of Lemma by symmetry (changing u, H -2, b, b; if necessary) we can assume that we have always un >> 0 and cos(d; + bi) ~ 1. Setting Z = >: ui, we have
1
z (()
R
U cos (wi; (Ø -oo)
(5)
Z
(E cos (wi (H Ko)
(6)
(For the second approximation we that have sampled the पł, independent from the u..) This is then the Fourier transform of the Gauss function and we get
f(t) = 22/2
The approximation (equation becomes exact when we train infinitely long, and (equation 6) when we have infinitely many neurons (or take the expected value).
Proposition 2: Under the same idcalized conditions (infinite number of neurons, weights wi, frozen), but arbitrary finite training set, and assuming the training converged, the function goes to for [ + 00.
Proof: We assume trained on M points 21, ZM (with the correct label) and use shallow Fourier net with N x oo neurons. We also assume that the training "converged" in the sense that the by converged (the uj keep growing). Analog the above case of one training point, we have
OL 0b,
M
1 1+ef(E)) Uj sin(w, Ft + by).
=
k=
With
M
E(w) = Σ
ck e i
Ck = 1 (
Am
we can write this using the imaginary part 3:
ØL
= - uy @6 F(1T,))
14

---

Under review as a conference paper at ICLR 2020
If E(w;,) = 0, the value of L does not depend on by; we first show that this only happens for (Lebesque- Nuuss set of w.
The real part EE) is real analytic function, and its value at 0 does not vanish since
E(0)
ck O.
Therefore the zero set Ry E has Lebesgue measure zero (an exercise in Calculus II, according to, and solved in (Mityagin 2015)). This implies that the even smaller zero set Z == {wi Rd E(w)= 0} also has Lebesgue measure zero.
Using the arg function
arg (0) - R/2222, r. r +
we can define the continuous function
B Rdd
\ 2 + R/R//10 22, + arg(E(())
which makes
B Rdd 2 c v0 + B((
B(の)
a well defined continuous function.
Now for ii IRd 2 and uj +
0L bb
3 (c E(wi))
depends on by E R2/2T, and L is minimal if this derivative crosses from negative positive values around by, which happens exactly when
- B - ((T,) +
" 0
bj
U <0
So for almost all J the weight vi determines the phase by in a way that depends continuously on T... With this our function becomes
Σ |uyl cos (w - B(w,))
and after rescaling by dividing by Z = Σ, |ui this converges the limit of infinitely many neurons to
f(3) := E., cos(ii . x B(N)))
With the Gauss function
ΣE
(w)) = ((2)d (e)
giving the probability density function for (10, Σ) this
[ Φv(wT) cos(u v B(ū)) duū
f(क)
= R xx(w))
au e
BB() du
Rd
B
BB()
u du
which is up multiplicative constant the real part of the Fourier transform of the function
g := dx (w))
BB))
Since Je (B) = 1 and Φs E L (RR), this function is also in L (RR), so the Riemann -Lbuesuee Lemma gives that f()) - 0 for [ claimed.
15

---

Under review as a conference paper at ICLR 2020
D RELU NETWORKS IN THE SIMPLEST 1-D PROBLEM
After randomly initializing the weight Wy ~ NNO, 0) and bias b; (O, a) for neuron i, the location of the resulting kink of ReLU(wr
b;) is bb/W;, so it is Cauchy-diste Sauchy-distriuuted with center 0 and part of them will fall on the right hand side of To. and another part on the left hand side. Training the parameters tries increase the value zo, but will not move the kink across zi. So we will with high probability (probability for infinitely wide networks) see the sum of functions of which some increase for large positive z. others increase for very negative X, so in general the expected picture is indeed curving upward on both sides, like the experimental result in the first image of figure 2
E
FEW POINTS
We construct as small set of points around -2,
$ = {-5.3, -5., -5.1, -2.1, -2,, -1.88, 3.2,3.1, 8.95, 9.1,9.22
The Fourier approximation gives figure
44
Figure 8: Fourier approximation for set S of four small clusters in ID
F GAUSS NETWORKS
We can define "Gauss networks" for which we take as activation functions Gauss functions
((x)= = /2
0.2
Figure Example function g(0.4 x - 1)
Using this function instead of ReLU(x) or sin(x) gives in the 2-dim toy example the figure (compare with figure
10
G COMPARISON TO NEAREST NEIGHBOR METHODS
The "nearest neighbor" classifier assigns label L to a test input if the closest training input had label L. We also the "nearest neighbor" method for out of distribution detection: For
16

---

Under review as a conference paper at ICLR 2020
test
Ensemble of KeLl Networia
Ensemble of Gauss Netvoris
Encemble of Fourer Nenvercs
Figure 10: Upper row: Input, Average output for ensemble of 50 ReLU / Gauss/ Fourier networks.
Lower row: Predictions of 4 individual randomly initialized Gauss networks.
threshold r we call an input "in distribution" if and only if its nearest neighbor in the training set has distance S r. Varying the then gives a ROC curve, analogous to varying the threshold 0 for the softmax output of network ensemble.
There are many variations of this simple method ne-nearest neighbor, RBF networks, adjusting the metric to data), but for the general principles we want to discuss here the differences not matter and we will use the basic implementation.
In (Bishop 1995). chapter 5.9, pp. 183-186 the author writes about the "major potential difficulties with radial basis function networks" stemming from the "localized nature" of the hidden units (in our implementation of the nearest neighbor method the hidden units would correspond to the training points).
The problem is that nearest neighbor methods "just memorize" training points, and only generalize to nearby points this requires that we have training point "near" any possible input point of our training distribution. On the other hand, "normal" networks learn "rules", which can generalize the "important parts" and ignore the "noise" that does not contribute the classification.
This can be a problem for nearest neighbor methods mainly in high dimensions, but as first il- lustration reproduce the two-dimensional toy problem from the book: Assume we have three classes (red, green, blue) that can be distinguished by their zo-coordenate, but we also have ran- dom cooroiidea with high variance (see figure
If the red / green / blue stripes have length ó along the zi-axis, a nearest neighbor classifier or RBF network may pick circles of radius around 8/2 characterize the points that belong the same stripe. However, for that we need to have enough points also in direction, if there is a gap of much more than 8, we will have points stripe that are not recognized as belonging to the training distribution (or to the corresponding label).
On the other hand, ReLU network will learn some regions and (over)confiddenlly classify all points in these regions red / green / blue.
We can demonstrate in this toy example how the Fourier network "interpolates" between "ReLU behavior" and "Nearest Neighbor behavior" when we adjust the standard deviation used in the weights' initialization of Fourier network: For small initializations, we are close to the "generaliz- ing" ReLU behavior, large initializations give something close to "memorizing" Nearest Neighbor behavior. We show in figure 12 how an ensemble of ReLU networks and ensembles of Fourier Net- works generalize to the 2D plane. The upper gives the classification around the input, the lower row zooms out, in particular showing that for Fourier networks the ensembles are not extending their confidence region infinity (as the ReL.U networks do).
17

---

Under review as a conference paper at ICLR 2020
Input
10
-5
-10
-10
-5
5
Figure 11: Input the points on the left have label "red", in the middle "green", and on the right "blue".
Emsnbi of RoUl Neteoto
Emorbic of hocee Networts
Eruembicof Faunko hidkerta
Exsemble of Nurto Rdwatta
Etienble of Nouner lidwats
Beill
Fasrebleed of liu fomanki
Ratarbble Ra Nerwnks
Exumibe Tourer
Figure 12: From left to right: Output of ReLU network, and Fourier networks with 0.15, 0.55, 1, 2. Areas with confidence < 0.75 grayed out.
Upper row: Range [-10, 10]; Lower row: Zoomed out to range [-50, 50)
To illustrate the resulting effect in higher dimensions, we add some "random background" to the MNIST and fashionMNIST images (see figure (3).
The "random background" plays the role of the 22 coordinate above: For a new input, the nearest neighbor classifier now has find training image that close the new one both in the digit and in the background. This is of course much harder (even with 60000 training points), and corre- spondingly classification accuracy the nearest neighbor classifier drops from 96.9% to 76.2% (see table 4. For ReLU (and Fourier) nets, also a drop, but only to 91.3%, the classifier just learns to "ignore" the background. (The classification accuracies reported here are the accuracy of the averaged softmax output of the networks in ensemble.)
For out of distribution detection, we see the same effect for the nearest neighbors method the area under the ROC curve drops from 98.9% to 77%, but for ReLU nets only from 91.8% to 91.1% 1%. The Fourier networks are supposed to combine the best both approaches, and indeed they give better cut-of-distributioo detection than either method in both cases.
18

---

Under review as a conference paper at ICLR 2020
0--
15
15.
0-1
10
10
15
-20
20
35
25
25
0-1
5-
15
15
20
20
30
Figure 13: Embedding MNIST fashionMNIST into random background
Metric
Area under ROC
Classifier accuracy
Model
Original
With background
Original
With background
ReLU nets
91.8%
91.1%
98.2%
91.3%
Fourier nets
99.7%
95.8%
97.5%
91.2%
Nearest Neighbor
98.9%
77.0%
96.9%
76.2%
Table Comparison between ensemble one-layer ReLU networks, an ensemble of one-layer Fourier networks and just using the distance to the nearest neighbor in feature space (which in this case coincides with the pixel space) as score for OOD detection. The in-distribution dataset is MNIST while the out-of-distribution data is coming from FashionMNIST.
As second example, we produce 60000 wider images by randomly selecting 4 MNIST digits and combining them to one new image. As the label of these images we take the label of the first digit. For out-of-sample images we replace the first digit by a fashionMNIST image, see figure 14
Again, this makes it much harder for nearest neighbor method now it has to find images in which all digits are similar (which still possible with 60000 training images, but harder), whereas the ReLU classifier can focus the pixels in the first quarter of the image and learn to ignore the rest, see 5able for the results.
An interesting philosophical question arises when produce images by exchanging the last digit by a fashionMNIST image: Are the resulting images "outliers" or not? In any case, the results in table 5 show that the ReLU classifiers now do not flag many "outliers", which is in agreement with our expectation that they focus on the left quarter of the image and mostly ignore the rest.
Metric Model
Area under ROC
Classifier accuracy
First digit fashion
(last digit fashion)
Original
ReLU nets
85.4%
(59.0%)
94%
Fourier nets
95.2%
(79.1%) 1%)
92%
Nearest Neighbor
79.4%
(79.7%)
57%
Table 5: As table but for "4 digits" data set.
19

---

Under review as a conference paper at ICLR 2020
0-
5-
32
381
8
20
40
60
80
100
0
1
5-
12
3
4
10
15-
20
40
60
80
100
8
10
6
15
20
25
20
40
60
100
Figure 14: 4 digits
H RELU NETWORKS WITH INFINITESIMAL INITIALIZATION
For given training data and infinitesimal initialization, there are only finitely many possibilities to which networks can converge (independent of network size), see (Maennel et al., 2018). The argu- ments given in that paper show also that with increasing number of weights we should with high probability end up at the same possibility.
1 MORE RESULTS ON MNIST
I.1 VARIATION OF ROC CURVES FOR MNIST VS NOTMNIST AND FASHIONMNIST
The ROC curves obtained when considering MNIST as in-distribution and notMNIST samples as out-of-distributtin and using two-layer classifiers for training were shown in figure 1 To visualize the variation of these curves, compute 50 networks, and apply this method 10 times ensembles of 10 randomly selected networks, this gives the left hand graph in figure 15] (The right hand graph in figure 15 shows the corresponding results for testing for "outliers" from fashionMNIST. We also get similar figures when training on fashionMNIST, see appendix M
02
stancurd crucmbic retental OF retuork stancord rctwork
02
atandand uncmmbi (18 mzzoerfoa
netware
zondoro network
0.08
003
Figure 15: 10 random samples of ROC curves for networks ensembles trained on MNIST, evaluated on notMNIST (left) and fashionMNIST (right). z-axis: MNIST samples not recognized as belonging to MNIST, y-axis: notMNIST samples recognized as not belonging to MNIST.
20

---

Under review as a conference paper at ICLR 2020
1.2 ENTROPY HISTOGRAMS
To compare our method also to the evaluation given in (Lakshminarayaaann al., 2017), we compute for each image the entropy of the softmax output and plot histogram over the MNIST and the notMNIST data set for the ensemble sizes 1,5,10. The first three columns of figure [6 from figure 3 in (Lakshminarayanan et al., 2017) show the resulting entropy of the softmax output for ReLU networks, first using the ensemble as is, then with additional adversarial training, and in the third column the corresponding results for "dropout at test time".
Ensemble
Ensemble
dropout
14
34
12
10
10
10
10
10
-0.5 0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0,0 0.5 1.0 1.5
2.0
entropy values
entropy values
entropy values
Ensemble
Ensemble + AT
MC dropout 0.1
14
12
12
10
10
10
10
10
0.0 0.5 1.0 2.0 entropy values
0.5 0.5 1.0
2.0
0.5 1.5 2.0 entropy values
entropy values
Figure 16: Histogram of the predictive entropy from figure 3 in (Lakshminarayanan et al. 2017) (first three columns) compared to the corresponding Fourier networks (last column). Top row: Known classes (MNIST), bottom row: Unknown classes (notMNIST).
For input from the MNIST test set (first row, blue curves) the entropy is almost always close to O, i.e. the classifier puts almost all probability on one label. For input from the notMNIST data set the classifier ensembles distribute the probability over several labels, resulting in much higher entropy. Use of adversarial training (second column) increases the entropy even a bit further for the notMNIST examples.
The fourth column shows the results for the corresponding Fourier networks. On the MNIST test set the behavior is basically the same the usual network, but notMNIST even one network often is unsure about the label, and an ensemble of 10 networks gives most of the time an entropy around 2 (the maximum would be (10) R 2.3 for the uniform distribution across all 10 labels).
1.3 CONVOLUTIONAL NETWORKS ON MNIST
Since MNIST vs. notMNIST too easy to profit from better classifiers, we use a more difficult problem: Train classes 0-4, whether detect classes 5-9 as "out of distribution".
For MNIST we use two convolutional layers with 7 x 7 windows and max pooling in 3 x 3 windows with stride 2, and add two fully connected layers with 200 / 100 neurons. We also use data augmen- tation by randomly cropping to 24 x 24 pixels. For the Fourier network we change the activation function of the last layer to sin(2r)
21

---

Under review as a conference paper at ICLR 2020
First experiments seem to indicate that a good way to choose the initialization for the different layers of the Fourier network to start with small initializations for the first layers and the last layer, and then choose the second last initialization large possible (see appendix J.3 for details).
With this setup we get for the area under the ROC curve:
Network
standard ReLU
ReLJU, large init sin(xx)
convolutional
95.9%
98.1%
99.1%
fully connected
95.2%
96.2
96.3%
Table 6: AUROC on MNIST 0-4 MNIST 5-9.
J DETAILED SPECIFICATIONS OF THE NEURAL NETWORKS AND THEIR PARAMETERS
J.1 ONE DIMENSIONAL TOY EXAMPLE
Input data:
Only one point at ærp = with label "1".
Network:
One hidden layer with 500 neurons, weights initialized with a = 0.0632 & 2/5500.
Learning:
ReLU: 500 steps with learning rate 0.001
sin: 5000 steps with learning rate 0.001
J.2
TWO DIMENSIONAL TOY EXAMPLE
Input data:
20 centers randomly chosen in -10 < x < 10, -10 < S 10. To each center one of
the labels "Red", "Green", "Blue" was randomly chosen. For each center (x, there were 50000 points randomly drawn from a normal distribution (n,, E), where Σ R D for randomly drawn rotation matrix R and D the diagonal matrix with entries σ 0.2 and 0.5.
Network:
One hidden layer with 500 neurons, activation function sin or ReLU, Weights (including bias terms) initialized with 12.
Learning:
26 epochs, batch size 256, learning rate 0.02.
J.3 784 DIM INPUT, ONE HIDDEN LAYER (FULLY CONNECTED). TRAINED MNIST
Input: 764 pixels, normalized to be in [-1,
Hidden layers: One hidden layer of 100 neurons
Initialization: E (0.002, 0.01, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2, 0.22, 0.25, 0.275, 0.3, 0.35, 0.4, 0.5, 1.0, 2.0, 4.0, 6.0)
Batch size: 256
Learning rate: 200 epochs with 0.02, 200 epochs with 0.005.
J.4 784 DIM INPUT, TWO HIDDEN LAYERS (FULLY CONNECTED), TRAINED ON MNIST
Input: 764 pixels, normalized be in [-1,1]
22

---

Under review as a conference paper at ICLR 2020
Hidden layers:
Two hidden layer of 200 neurons, first layer: ReLU, second layer: ReLU sin.
Initialization:
For ReI.U network: 01 = 0.07, 02 0.14.
For Sine network: 01 0.75, Ø2 0.0002
Parameters searched: For ReLU this is the standard initialization o - 2/ V(n).
For Sine networks, initial tests gave the result that for given 02 the on should be large as possible such that the training still converges. Then I tried 02 {(0.0002, 0.001, 0.01] and 0r E {0.25, 0.5, 0.75, 1.0). Learning rate was 0.01.
Best results: (01, 02 area under ROC curve for MNIST vs. notMNIST):
0.5, 0.0002, 0.99898
0.5, 0.001, 0.99908
0.5, 0,01, 0.99946
0.75, 0.01, 0.99974
0.75, 0.001, 0.99977
0.75, 0.0002, 0.99977
J.5 784 DIM INPUT, CONVOLUTIONAL NETWORKS, TRAINED ON MNIST
Input:
764 pixels, normalized to be in [-1,
Preprocessing: 4 times random cropping from 28 28 to 24 24.
Hidden layers:
Convolution, x 7 windows, 1 16 channels (to size 24 x 24 x 16)
Max pooling in x 3 windows with stride 2 (to size 12 x 12 x 16)
Convolution, 5 x 5 windows, 16 + 32 channels (to size 12 x 12 32)
Max pooling in 3 x 3 windows with stride 2 (to size 6 x 6 x 32)
Fully connected, 200 neurons, ReLU activation function
Fully connected, 100 neurons, ReLU or sin(2) activation function
Batch size: 40
Learning rate: 25 epochs 0.002, 20 epochs 0.001, epochs 0.0005
Initialization: Given by o (Ø1, (2, 03, 7.).
For Fourier networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
(0.001, 0.001, d3 0.0001), best result: 99.1% for d3 20
0 (0.002, 0.001, 03, 0.5), best result: 99.1% for σ3 1.5
(0.1, 0.1, 0.001), best result: 98.8% for σ3 1.75.
For ReLU networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
23

---

Under review as a conference paper at ICLR 2020
(0.01, 0.01, da. 1.0), best result: 97.8% for d3 0.3
a (0.01, 0.010 03. 0.6), best result: 97.4% for d3 0.3
o (1, best result: 97.7% for Ø1 = 0.9
(1) best result: 98. % for Φ1 = 0.55
J.6 SIMPLE CNN, TRAINED ON SVHN
Input:
32x32x3-shapar images, with the pixels normalized to be in [0, 1].
Hidden layers:
Convolution, x 7 windows, 1 16 channels (to size 32 x 32 x 16)
Max pooling in x 3 windows with stride 2 (to size 16 x 16 16)
Convolution, 5 windows, 16 channels (to size 16 16 32)
Max pooling x 3 windows with stride (to size 8 x 8 x 32)
Fully connected, 200 neurons, ReLU activation function
Fully connected, 200 100000 neurons, ReLU sin(x: activation function
Batch size: 128
Learning rate: 0.01 (for 200 neurons on the last layer) and 0.001 (for 100000 neurons). Halved every 40 epochs.
Best a for initializing the last layer: 0.75.
J.7 SHALLOW VGG, TRAINED ON SVHN
Input: 32x32x3 images, with the pixels normalized to be in [0, 1],
Hidden layers:
the first 4 convolutional blocks the VGG16 architecture (including the corresponding max pooling layers) (to size 8 8 x 128)
Fully connected, 200 or 100000 neurons, ReLU or sin(x) activation function
Batch size: 128
Learning rate: 0.001. Halved every 40 epochs.
Best a for initializing the last layer: 0.01.
J.8 VGG16, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0, 1].
Hidden layers:
all the convolutional blocks of the original VGG16 architecture (to size 1 x 1 512)
Fully connected, 200 neurons, ReLU activation function
Fully connected, 100 or 100000 neurons, ReLU or sin(r) activation function
Batch size: 128
Learning rate: 0.001. Halved every 40 epochs.
Best a for initializing the last layer: 0.25 and 0.1 for sin(x) and ReLU respectively, for 100 neurons on the last layer and 0.002 for 100000 neurons.
24

---

Under review as a conference paper at ICLR 2020
K DEFINITION OF THE DATA SETS
As input distributions that are "clearly different" from MNIST we can use the fashionMNIST ( (Xiao
al. 2017) and notMnist (Bulltoov 2011 data sets, figure 17
6
B E
Figure 17: FashionMNIST (top) and notMNIST (bottom) data sets.
We also some constructed data sets:
"Masked fashionMNISTT":
In MNIST some pixels are always or almost always white, figure 19
Out of fashion-
MNIST we derive this data set by setting all pixels to (white) that have an average gray value of below 0.01.
"Masked notMNIST"
This derived from notMNIST in the same way.
Circles and lines:
Two lines and two circles in random positions.
Flipped MNIST:
Derived from MNIST by mirroring the digits 3,4,6,7,9 horizontally and the digits 4,6,7,9 vertically.
IID noise: Uniform iid pixel values from [0.256].
Independent pixels:
Each pixel is randomly sampled from pixels at the same position at MNIST training images, independent of the other pixel values.
PseudoMNIST:
Here try to mimic not only the value distribution for each pixel, but also local correlations. We start with an image in which each pixel is independently sampled like in the previous distribution. Then we perform 1000 (PseudoMNISTI) doMNIST2) updates of the following form: We randomly select 3 x window, and find the closest match in 1000 randomly sampled images from MNIST, where "closest" is with respect to the L2 distance of the outer pixels of the x 3 window. Then we replace the middle pixel our image with the value of the middle pixel in this "closest match" from MNIST.
1500 (Pseu-
We will also use a "masked version" with a mask from MNIST: see figure 19 for the origin of the mask, and figure [8]or examples.
L TRAINING ON MNIST, USING OTHER OUTLIER SETS
With these data sets we can again compute the ROC curves that show how well we can distinguish the training set from the outlier set by setting threshold the "confidence". As confidence take again the averaged softmax output of the predicted label over an ensemble of 10 classifiers, in which the weights randomly sampled from normal distribution with mean O and standard
25

---

Under review as a conference paper at ICLR 2020
A
Figure 18: Constructed data sets.
Top row: Masked fashionMNIST, masked notMNIST, circles and lines, flipped MNIST.
Bottom row: IID noise, independent pixels, PseudoMNISTI, PseudoMNIST2.
5-
10
15
20
10
15
Figure Average pixel value < ttreeshlds 0, 0.0001, 0.001, 0.01, 0.1 on MNIST
deviation ø. We consider the area above the ROC curve "error" and plot - log(error) against the initialization used for the networks. The resulting graphs are in figure 20
Figure 20: Influence of initialization
rraxiis: log(a), y-axis: - log(error). higher is better Test sets: Circles and lines, pseudo mnist, IID noise.
In some cases, it easy to distinguish MNIST samples from outliers by just looking at pixels close to the corners: In MNIST these pixels are always white. but in notMNIST often not. To climinate this shortcut, we also consider "masked" versions of notMNIST and fashionMNIST in which these pixels are for the masked versions, and for images which each pixel is independently sampled from MNIST images, each pixel has the same value distribution in MNIST. Even with the "looking at the corners shortcut" disabled, the Fourier networks still perform well.
white (see appendix K for details). In figure 21 we show the corresponding graphs
26

---

Under review as a conference paper at ICLR 2020
Figure 21: Influence of initialization, axes as in figure 5
Left to right: masked notMnist fashion MNIST, pixels sampled independently from MNIST
M TRAINING ON FASHION MNIST
We use the same two hidden layer network architecture, initializations, and learning rates as in figure M but train fashion MNIST, the result is in figure (ggure22beelow below. Since this is more challenging data set, the same architecture gives not as good results as for MNIST (note the æ-axis extends now to 0.2), but the difference between ReLU and sin(x) networks is comparable, resulting in graphs (figure 22) that look very similar to figure
10.
12.
Vaaced andi steck)))
800 0.35
als
a50
0.50
8/n
810
615
0.32
Figure 22: 10 random samples of ROC curves for networks ensembles trained on fashion MNIST, evaluated notMNIST (left) and MNIST (right). w-axis: MNIST samples not recognized as belonging to MNIST, y-axis: notMNIST samples recognized as belonging to MNIST.
We can also plot the area under the ROC curve for different initializations of network with one hidden layer- this gives figure Again this qualitatively the same result as figure or in 20gure20 above.
20
Sn
Figure 23: Influence of initialization, networks trained on fashion MNIST r-axii: log(o), y-axis: log(er)or), higher better Test sets: Circles and lines, MNIST, notMnist.
27

---

Under review as a conference paper at ICLR 2020
N INFLUENCE OF THE DISTRIBUTION FROM WHICH WEIGHTS ARE SAMPLED
Our method for getting variety of networks relies different randomly sampled initializations giving different networks. According to (Sirignano & Spiliopoulos, 2018) this actually no longer happens if the number of neurons is very large: Then the networks converge idealized net- work in which there are infinitely many neurons that have their weights distributed according to the probability distribution from which we sample.
To illustrate this potential problem visually, we look at a very simple 2-dimensional toy example, in which we can again plot how the classifiers generalize from the training distribution other points, like we did in section 3
As input we use only line, with labeled training samples given as blue dots in the middle and red dots at the end (figure 24
Figure 24:
Input for the 2-dim example.
We will use ReLU network with one hidden layer of 100 neurons to classify all points in the surrounding area of the plane. Colors encode the "probabilities" that the softmax computes from the output layer, with blue red signifying "probability 0/1 of being blue". The result is in figure 25
Figure 25: ReLU networks, extrapolating from line 2d.
Left: Output of three example networks, right: average softmax output over 50 networks
As in section 3 we see that most points belong with high confidence to on of the labels (here they are either blue or red), and there is only limited variation for classifiers initialized from different randomly chosen weights.
If change the activation function of the hidden layer from ReLU sin(x), we obtain different results outside of the training distribution (figure 26 While individual points are still mostly red or blue, ie. the individual classifiers are still "overconfident" the different classifiers now tend to disagree and averaging the softmax outputs over several of them we get much more realistic confi- dences for the Fourier networks than for the ReI.U networks: Now the ensemble only confident about the label for points in the vicinity of the training inputs.
Figure 26: Fourier networks, extrapolating from a line to 2d.
Left: Output of three example networks, right: average softmax output over 50 networks
28

---

Under review as a conference paper at ICLR 2020
To demonstrate the potential problem with very wide networks, 20000 neurons (instead of the 100 we used in the above pictures). Now the results different initializations become more similar to each other, figure 27
Figure 27: Different Fourier networks with weights initialized by samples from same distribution, 20000 hidden sin-neurons
So in this case, to change the distribution itself. For higher dimensional input, this effect may be more theoretical because the number of neurons required to get into that regime is longer realistic.
longer rely on different sampling from the same distribution, but need
However, even for a moderate number of neurons it may make sense to vary the distribution of initial weights from which we sample to obtain greater diversity. Here an example in our case of 2-dimensional input, 100 neurons, ReLU activation:
The weights in the 2-dimensional case have coordinates: Two for the direction and one for the bias. We can now sample the "direction" weights according to Normal Distribution with one standard deviation dı, and the "bias" weight according to a Normal Distribution with different standard deviation 02. Different pairs (0),02) describe different distributions from which we initialize the weights.
If we vary weight initialization by varying (Ø1, 02), the resulting networks seem to converge to different classifiers, see figure
Figure 28: Ensemble average, 100 hidden ReL.Uneuroos Left to right: σ1, 02 = (3, (0.01, 0.01), (5,5) (0.01,
In particular, the networks initialized by σı, 02 = (5, 5) could theory contain the same initializa-
tions the networks initialized by dı, 02 = (0.01, 1), but the emphasis shifts to different networks: The latter give blue cones extending downwards most of the time, whereas such a result is not among the 50 samples I obtained for o,, 02 (5. 5).
So even for not extremely large networks it would make sense to restrict the distribution of weights from which sample in different ways obtain more diverse set of networks.
29

---

Under review as a conference paper at ICLR 2020
O SEPARATING DIFFERENT FEATURES FOR THE SAME LABEL
We try restricting the weights in a neural network classifier to certain subspaces not only during initialization, but throughout the training. The hope would again be that classifiers with different restrictions develop different features which generalize different ways to unseen examples, giving more variety for out-of-distriution samples.
In the MNIST case, we evaluate two strategies restricting the weights of the first layer:
Only allow access half of the pixels.
If each classifier sees only half of the image, they usually still can figure out which digit is:
2
7
Figure 29: Restricting MNIST input to half of the image
Only allow access "line detectors" with certain directions.
We implement this by convolution with a 7 x 7-matrix which corresponds to a stripe of certain direction. We allow 4 (of 10) directions per location, and reduce the dimension of the resulting weight space we subsample the locations to a grid, see figure 33) This again gives vector space of dimension which is roughly half of the original dimension.
Figure 30: Convolution with stripes in some directions, centered at some locations
This seems to give significant increase of the area under the ROC curve in particular for the difficult case "trained 0-4, evaluate 5-9 outliers":
Area under ROC curve
"Outliers" MNIST 5-9
sin
sin with half input
sin with stripes
combined
96.5%
97.3%
97.5%
97.6%
fNSHionMNIST notMNIST 99.9%
99.3%
99.5%
99.6%
99.7%
99.9%
99.9%
99.9%
Same test with restriction to half images on fNshionMNIST:
The classifier is trained fashionMNIST 0)-4.
30

---

Under review as a conference paper at ICLR 2020
"Outliers"
relu, large init
relu, restr.
sin
sin, restr.
MNSST 5-9
91.3%
92.5%
93.5%
94.3%
MNIST 94.2%
95.1%
98.9%
99.2%
notMNIST 93.4%
95.2%
99.6%
99.6%
We have also applied the same approach when training convolutional networks on SVHN. The table below shows the area under ROC curve obtained with a Simple CNN and Shallow VGG. By applying a square mask on one of the corners of the image (top left, top right, bottom left, bottom right) we obtain an ensemble of constrained classifiers that is slightly more robust out-of- distribution samples.
Model
Oniginal imput
Masked input
Simple CNN
96.6%
96.5%
Shallow VGG
95.8%
96.6%
P INPUT TO ACTIVATION FUNCTION
In this network the first hidden layer has ReLU neurons, and the second hidden layer has sin- Neurons. We plot the activation observed as input to the second layer, for the neurons which have the largest weight going to the output of "0" (left), and the for the 5 neurons with the smallest weight going to the output of "0" (right) for all images that are indeed a "0". We observe that the peaks are separated by 2n, the left at the maxima of sin, the right at the minima of sin. A similar pattern arises for other imaoes from the trainino set.
130
14
43
28
Figure 31: Activation function input for images of "0", left: largest, right: smallest weight.
000
400
200
Figure 32: notMnist activation
However, when we plot the input to the sin activation for images from notMnist, we no longer see the peaks maxima / minima of sin, suggesting that the output for these images will be "random":
31

---

Under review as a conference paper at ICLR 2020
Q MORE EXPERIMENTAL RESULTS FOR SVHN VS CIFARIO
Figure 33 shows which are the regions of optimal initialization for the final fully-connected layer for both ReLU and Fourier networks. Note that the initialization is too large, gradient descent on the Simple VGG Fourier network will sometimes fail to converge.
Figure 33: Influence of initialization on AUROC for SVHN, axes like in figure 5
The distribution of the entropy of the softmax outputs samples for a Simple CNN network is shown in Figure 34 Results are presented for both regular ReLU and sin(x) activations for the Simple CNN architectures (with = 0.75 used as the standard deviation of the initialization distribution of the last layer). The curves correspond to ensembles of
in-distribution and out-of-distriiuttio
1, 5, and 10 models.
Figure 34: Entropy histograms for SVHN as in-distribution set (top row) and CIFARIO as out- of-distribution set (bottom row). Comparison between using sin(x) (right-hand side) or ReLU activations (left-hand side) on the last layer of a Simple CNN network.
Table 7 presents the best results obtained using convolutional networks trained on SVHN for both the ReLU and Fourier setup. Out-of-distribution samples are detected using two approaches, by selecting threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble or for the entropy of the ensemble's prediction. For evaluation, we used the following metrics:
the area under the ROC curve (AUROCT)
the false positive rate 80% positive rate (FPR80L)
the negative log-likelihood score (NLL))
the Brier score (Brier()
In order to be closer to the ideal case, which is the premise for theoretical results proved above, also report numbers for the same network architectures with significantly more neurons on the final fully-connected layer (i.e. 100,000). For each configuration, the table contains the numbers for the
32

---

Under review as a conference paper at ICLR 2020
model with the best value for the initialization σ. The dependency between the under the ROC curve and the initialization strength is presented in figure 33
Network
Last
Activation
Last Layer Units
Last Layer Init
AUROCT using max-p
PPRSOL using max-p
AUROCT using entropy
FPR80J using entropy
NLL Score]
Brier Score]
Simple CNN
ReLU
100000
91.97%
12.89%
92.18%
11 11.82%
0.3303
0.0136
Simple CNN
sinc
100000
0.75
95.25%
7.80 %
96.29%
5,55%
0.3324
0.0141
Simple CNN
ReLU
200
0.75
92.89%
11.55%
93.23%
10.31%
0.3058
0.0123
Simple CNN
sinc
200
0.75
95.62%
7.38%
96.63%
5.18%
0.2895
0.0123
Shallow VOG
ReLU
100000
0.25
94.76%
8.41%
95.851
6.29%
0.3512
0.0149
Shallow VGG
sine
100000
0.1
95.05%
8.34%
96.40%
5.67%
0.3094
0.0133
Shallow VGG
ReL.
200
0.01
92.65%
12.21%
93.51%
9.97%
0.3261
0.0138
Shallow VGG
sine
200
0.01
94.40%
9.56%
95.80%
6.72%
0.3184
0.0137
VGG
KcT.D
T00000
0.01
95.36%
7.50%
96.43%
5.31%
0,2452
0,001
VGG
sine
100000
0.002
95.69%
7.20%
96.49%
5.05%
0.2728
0.0090
VGG
KcL.0
100
0.1
9.77%%
95.77%
6.73% 6.52%
0,3348
0,0144
VGG
sine
100
0.25
94.63%
9.07%
95.88 %
0.3286
0.0143
Table 7: Extensive experimental results for ensembles of models trained on SVHN and evaluated on CIFARIO as out-of-distribution dataset.
R COMPARISON WITH OTHER OOD DETECTION METHODS
Paper
Method
In-distrib. set
OOD set
AUROC
(eendrycks Gimpel 2017)
baseline
MNIST
notMNIST
93.2%
Liang et al. 2018
outlier exposure
MNIST
notMNIST
98.2%
ours
Fourier networks
MNIST
notMNIST
99.9%
Malinin & Gales 2018)
outlier exposure
MNIST
MashionMNIST 5-9
96.5%
ours
Fourier networks
MNIST
FashionMNIST 5-9
99.7%
Table 8: Area under ROC curve for entropy the OOD detection score.
(Hendrycks & Gimpel 2017) use slightly larger network and GELU activation functions, (Liang et al. 2018)) train also on images that do not belong to MNIST, but also not on the "notMNIST images that used for evaluation. Similarly. (Malinin & Gales 2018 use samples from set of outliers to tune the model; the more similar these are to the outhiers used for testing, the better the performance of the model is. Table shows the results obtained using the Dirichlet Prior Network model (DPN) for various settings. In all instances we used the default hyperparameter values (sug- gested either in the paper or in the code released by the authors setting on which the Prior Network model performs best half of the out-of-distribution test samples are drawn from the outlier distribution that was used for fine-tuning the model (namely the distribu- tion of MaSHI samples from classes 0-4, since we use the whole test set of FashionMNIST as OOD samples at test time). Moreover, for the best performing DPN results we tweaked the orig- inal setup and allowed the model to use the FNSHionMNIST training samples belonging to classes 0-4, instead of the (fewer) test samples, as prescribed by the authors of the paper. However, even on this significantly simplified setup, the DPN baseline falls behind the Fourier network ensemble even though the latter never sees any outliers during training.
important to note that for the
Model
In-distrib.
OOD
OOD test
AUROC
AUROC
training
(entropy)
(max-prob)
DPN DPN ReLU nets
MNIST
Omniglot
IMNIST 5-9
92.60%
93.00%
MNIST
IMNIST 0-4
fMNIST 5-9
96.50%
96.60%
MNIST
N/A
IMNIST 5-9
91.78%
91.89% 99.51 %
Fourier nets
MNIST
N/A
fMNIST 5-9
99.69%
Table 9: Comparison between ReLU / Fourier network ensembles and various settings of the Dirich-
let Prior Network model.
33


Reviews:
[{"review_id": "H1xbytdCFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nI have read the reviews and the comments. Overall I am still positive about the paper and I have confirmed the rating.\n\n======================\n\nThis paper proposes a method for the uncertainty estimates for Neural Network classifiers, specifically out-of-distribution detection. Previous methods use an ensemble of independently trained networks and average the softmax outputs. The authors investigate this method (ensembles of ReLU networks) and observe three fundamental limitations:\n“Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classification (CONSTANT FUNCTIONS ON THE TRAINING MANIFOLD).\n\nTo mitigate these problems the authors proposed the following:\n\n- Changing the activation function of the last hidden layer to the sin(x) function, and they claimed that this is going to guard against overgeneralization.\n- Use larger than usual initialization to increase the chances of obtaining more diverse networks for an ensemble.\n- They claimed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets.\n\nThe paper addresses an important problem, out of distribution detection, by proposing a Fourier network which is somewhere between a ReLU network (small initialization) and a nearest neighbor classifier (large initialization). The authors claimed that this leads to an out-of-distribution detection which is better than either of them.\n\nThe paper is well written and easy to follow. The authors did an interesting and precise investigation in how to force the confidence score to decay like a Gauss function by proposing to use the Fourier transform of such a Gauss function. By doing so they get the advantage of ReLU (ability to generalize) and prevent the network to become arbitrarily certain of its classification for all points. However, the authors claimed that when they switch the activation function to sin(x) the increase of |x| will usually stop at the first maximum or minimum of sin which is (around \\pi/2). However, the authors did not explain how they get this result (i.e the value \\pi/2). It would be interesting if the authors could show results for the case greater than or less than (\\pi/2) to show the difference.\nIn addition, Figure 3 shows that the ensemble of ReLU networks is overconfident in most of the area, whereas the ensemble of Fourier networks is only confident close to the input, and in the discussion of constant function of their training manifold the authors discuss some example.\n\nI would like to ask the authors: \n\nDid the Fourier networks learn the input distribution?\nHow are defined: usual initialization, small initialization and large initialization?\n\nThe experiment section is adequate. However, it would strengthen the paper if the authors compared against other approaches such as: \n- Predictive uncertainty estimation via prior networks, NeurIPS 2018.\n- Generative probabilistic novelty detection with adversarial autoencoders, NeurIPS 2018.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g8Iq1Z5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method to detect out-of-distribution or anomalous data points. They argue that Fourier networks have lower confidence and thus better estimates of uncertainty in areas far away from training data. They also argue for using “large” initializations in the first layers and sin(x) as the activation function for the final hidden layer.\n\nThe paper does not seem to have any significant logical reasoning on why their specific architecture works, but \"describes\" what they did. It is not clear what the novelty is, besides that they found an architecture that seems to work. Additionally while Fourier networks have lower confidence, that does not necessarily mean they are more accurate estimates of uncertainty. However the reviewer does acknowledge that the estimates are mostly likely better than ReLU networks that are well known for having terrible estimates of uncertainty.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxqXb7TFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Updates after rebuttal **\n\nI thank the authors for the response, though I am still skeptical about the evaluation of the method, which might be a result of heavy tuning and overfit to the chosen test sets. The proposed approach also requires more theoretical justification.\n\n------------------------------------------------\n\nI'm not an expert in this area but I do find this paper interesting. Though the name \"Fourier networks\" is a bit arbitrary because the proposed approach also applies to multi-layer networks where only the last hidden layer has the proposed change.\n\nThe extrapolation problem of ReLU networks is an interesting point. I don't know previous works that point out this for out-of-distribution detection but it's worth figuring out if this observation has been made in the adversarial robustness community.\n\nI do have several concerns, summarized below:\n* On page 3 the fourier transform is motivated by that RBF networks \"do not generalize as well as ReLU networks\". I doubt if there is any evidence for this argument.\n* I have some issue understanding proposition 1: what is w_i'? Only w_i is mentioned before.\n* The \"Fourier network\" is not defined explicitly in the paper, which makes it hard to understand the architecture/algorithm details. If I understand it correctly, it is only about changing the activation function of the last hidden layer and large initialization, with everything else the same as the ReLU networks?\n* How does the magic number \"\\sigma_1 = 0.75\" and \"\\sigma_2 = 0.0002\" come from? Did you search it by looking at the test performance? Is the performance sensitive w.r.t. the two parameters?\n\nI'm willing to increase my score if the authors addressed my concerns.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for detecting out-of-distribution (OOD) samples.

A reviewer pointed out that the paper discovers an interesting finding and the addressed problem is important. On the other hand, other reviewers pointed out theoretical/empirical justifications are limited. 

In particular, I think that experimental supports why the proposed method is superior beyond the existing ones are limited. I encourages the authors to consider more scenarios of OOD detection (e.g., datasets and architectures) and more baselines as the problem of measuring the confidence of neural networks or detecting outliers have rich literature. This would guide more comprehensive understandings on the proposed method.

Hence, I recommend rejection.


Decision: Reject