Year: 2019

Abstract:
A simple method for obtaining uncertainty estimates for Neural Network classifiers (e.g. for out-of-distribution detection) is to use an ensemble of independently trained networks and average the softmax outputs. While this method works, its results are still very far from human performance on standard data sets. We investigate how this method works and observe three fundamental limitations: “Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classification. To mitigate these problems we suggest “large” initializations in the first layers and changing the activation function to sin(x) in the last hidden layer. We show that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-ofdistribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).

text: A simple method for obtaining uncertainty estimates for Neural Network classifiers (e.g. for out-of-distribution detection) is to use an ensemble of independently trained networks and average the softmax outputs. While this method works, its results are still very far from human performance on standard data sets. We investigate how this method works and observe three fundamental limitations: “Unreasonable” extrapolation, “unreasonable” agreement between the networks in an ensemble, and the filtering out of features that distinguish the training distribution from some out–of–distribution inputs, but do not contribute to the classification. To mitigate these problems we suggest “large” initializations in the first layers and changing the activation function to sin(x) in the last hidden layer. We show that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of- distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).

heading: 1 INTRODUCTION
text: When a neural network classifies inputs, we often need to have some measure of the uncertainty involved in the prediction. In particular, in safety critical applications we would like to know when the classifier received an input that is different from the inputs it has been trained on - in such cases it could be dangerous to just use the “best guess”, instead we may want to choose some safe action or involve a human.
One solution for this “out-of-distribution detection” problem could be to add a label “unknown” and augment the training set with inputs that do not belong to any of the labels of the classifier and which get the new label “unknown”. However, the danger is that the classifier learns the particular type of “known unknowns” it was trained on and does not generalize to the “unknown unknowns”. To avoid this, we treat out-of-distribution detection as a “one class classification” problem, i.e. we want to learn the input distribution without help of out of distribution inputs.
The simplest approach is to use the softmax output of a neural net as a confidence measure. It is well known that the softmax output tends to be “overconfident”, so we cannot interpret it directly as a probability for the chosen class. However, it still tends to be more confident for samples from the correct distribution than for outliers, so setting a threshold for the softmax output can to some extent distinguish between “in distribution” and “out-of-distribution” samples. This is the “baseline method” for outlier detection suggested in (Hendrycks & Gimpel, 2017).
In (Lakshminarayanan et al., 2017) this baseline is improved by using an ensemble of classifiers (and adversarial training): Often each random initialization of a neural network gives an overconfident classifier, but different classifiers disagree - averaging the softmax output over an ensemble of classifiers then gives an improved signal.
Other, more computationally demanding approaches are Bayesian Neural Networks ((Neal, 1996), (Barber & Bishop, 1998), (Blundell et al., 2015)), nearest neighbor methods (e.g. (Mandelbaum & Weinshall, 2017), (Jiang et al., 2018), (Papernot & McDaniel, 2018), (Frosst et al., 2019)). For some further (less closely) related works, see appendix A.
We investigate limitations of the “ensemble averaged softmax output” method, as this seems to hit the sweet spot of being easy and fast, but still giving good results. To overcome the observed limitations and further improve this method, we propose two changes:
• Changing the activation function of the last hidden layer to the sin(x) function. (We will see in the next section how this can guard against overgeneralization.)
• Use larger than usual initialization, to increase the chances of obtaining more diverse networks for an ensemble.
In the next three sections we go through three observed problems with ensembles of ReLU networks and explain how these changes mitigate them, and in the following sections we evaluate the suggested changes on standard data sets (MNIST, CIFAR10, etc.).
We evaluate this uncertainty mainly with view to out-of-distribution detection: If we have meaningful uncertainties, the classifier should in particular “know when it does not know the answer”. This means it should display uncertainty for inputs that do not “belong to the same type of input” as the training examples. By selecting different thresholds for the uncertainty, we can plot the ROC curve by recording for each threshold the fraction of samples below the threshold for “in distribution” (x) and for “out-of-distribution” (y) inputs. In particular, figure 1 gives this evaluation for our method against the standard ensemble methods on two standard data sets.
x-axis: MNIST samples recognized as not belonging to MNIST, y-axis: notMNIST samples recognized as not belonging to MNIST.

heading: 2 PROBLEM: “UNREASONABLE” EXTRAPOLATION
text: We first look at the simplest possible example of classification: The training set consists only of one point x0 ∈ Rd which has label “1” (we assume there is another label, for which we do not have a sample). Training standard ReLU networks with one hidden layer on just one point x0 = 2 ∈ R we get something like figure 1: Gradient descent finds a simple functions describing the data, which will (with very high probability) be an increasing function of x. So for larger values of x the network will become arbitrarily certain that the label of x should also be “1”. (See appendix C for more details.)
It has been observed in (Hein et al., 2018) that this is true in general: Given any ReLU network trained on some data for a classification problem in Rd, it will describe a piecewise linear function. If we look at the values of this function along a ray to infinity, this will be a linear function when we are far enough from the origin, and in general this means the network will become arbitrarily certain
of its classification for all points on the ray far enough from the origin. This is clearly not desirable, we would rather get a low confidence when we are far from the training data. (Here “far from the training data” is a convenient aspect that can easily be checked mathematically. Since the data are often bounded, it is actually more relevant in practice that this behavior starts as we leave the range of the training data. As an illustration, see figures 3 and 4.)
Going back to our simplest possible special case, what should our confidence be that other points belong to the same label? Of course, there is no unique “correct” answer, but a reasonable model could be that our confidence decays like a Gauss function around this point ~x0, i.e. is proportional to
e−|~x−~x0| 2/2σ2 (1)
for some σ that defines the scale on which our confidence decays.
One way to force such a decay would be to use directly Gauss functions (equation 1) as activation functions, which leads to RBF networks which we could consider as a form of nearest neighbor methods. While they recognize reliably points that are not close to one of the ~x0 in (equation 1), they do not generalize as well as ReLU networks (see appendix G), so we will use a more indirect approach.
Using that the Fourier transform of such a Gauss function is again of this shape, we can write equation 1 also as an expectation:
e−|~x−~x0| 2/2σ2 = E~w∼N (0,Σ) [ cos ( ~w · (~x− ~x0) )] ,
where Σ is the diagonal covariance matrix σ−2 · I . When we approximate the expectation by a finite sum over N samples ~wi from N (0,Σ) we get an expression of the form
f(x) = N∑ i=1 ui · cos(~wi · ~x+ bi)
where ui = 1/N and the bi are chosen such that the cos(~wi · ~x0 + bi) = 1. This function can be interpreted as the output of a neural net with one hidden layer of N neurons with activation function cos(x) (we will later shift the phase bi and use sin(x)). (The output is one dimensional since we only care about the logits for the one label that appears in our training set.) This is the main idea in the proof of the following result (see appendix C for details and proof):
Proposition 1: Assume the weight vectors ~wi are sampled from a normal distribution N (0,Σ) and then fixed, the numbers bi and ui are sampled independently from some distribution (not identically zero) with finite second moments (e.g. a normal distribution) and then trained on one data point ~x0 with label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network’s output converge to a function that is approximately proportional to e−~x
T Σ~x/2. This approximation becomes exact when the number of neurons approaches infinity, or when we take the expected value.
We also have for general training sets (see appendix C for a proof):
Proposition 2: Under the same idealized conditions (infinite number of neurons, weights ~wi frozen), but arbitrary finite training set, and assuming the training converged, the function goes to 0 for |~x| → ∞. To get an exact formula, we froze the weights ~wi. In practice we would train them as well, but at least experimentally we do not see a difference when the ~wi are large: They do not get changed by a large amount, and the result still looks the same. An example is given in the two plots of the right side of figure 2. When we train on a training set of a few points, we see experimentally that (under the same idealized conditions: Large number of hidden neurons, frozen ~wi) we get an approximation to a weighted sum of Gaussian functions at these points (see appendix E). So it may seem we constructed a complicated approximation to a nearest neighbor classifier.
We now switch the activation function to sin(x), this makes no difference for the function if we compensate by shifting the bi by π/2, but the region of “small initialization” |~wi|, bi ≈ 0 is now similar to a linear network: The input to the hidden neurons i will be ~wi · ~x ≈ 0, and sin(~wi · ~x) ≈ ~wi · ~x. As training progresses, |x| usually increases. For ReLU activation, this increase is not bounded (if we are on the side ~wi · ~x > 0), but for sin the increase will usually stop at the first
maximum or minimum of sin, i.e. around ≈ ±π/2 (or also at a different minimum/maximum if we start from higher initialization, see appendix P for experimental examples). This means that on the training set, features (i.e. the output of the sin(x) neurons in the last hidden layer) get a large value by tuning the input to around x ≈ ±π/2, which is difficult to achieve “accidentally” for input that is not in the training set. By contrast, for ReLU (or tanh) activation, the inputs only have to achieve any high value.
We will call networks in which the last hidden layer uses the sin(x) activation function “Fourier networks”.
So the Fourier network is somewhere between a ReLU network (small initialization) and a nearest neighbor classifier (large initialization). In particular for intermediate initializations it generalizes better than a nearest neighbor classifier, but still detects more outliers than a ReLU network, which leads to an out-of-distribution detection which is better than either of them, as we will see later (see in particular appendix G for a comparison to nearest neighbor classifiers).

heading: 3 PROBLEM: “UNREASONABLE” SIMILARITY OF NETWORKS IN THE ENSEMBLE
text: In section 2 we considered only one network, and saw that ReLU networks become arbitrarily certain far from the training set. However, averaging the softmax output over an ensemble of networks could fix that: Even if each network is very certain about its prediction, the ensemble could give low confidence if the different networks disagree. This indeed happens, but less often than one might expect. To illustrate this, we generate the input in figure 3, which has 20 clusters of points with labels “red”, “green”, or “blue”.
We take an ensemble of 50 networks that were independently trained on the same data set, and use the average “probabilities” for the 3 possible labels as RGB values for the corresponding pixels. In the second image (ReLU networks) the ensemble is still overconfident in most of the area, whereas on the right the ensemble of Fourier networks is only confident close to the input.
The reason for the overconfidence of the ReLU ensemble is that the individual networks tend to agree even far from the sample points. On the other hand, the Fourier networks find different sparse Fourier interpolations to describe the input, so their output only agrees close to the inputs, see figure 4. For another 2-dimensional example, see also appendix N.
As a higher dimensional illustration, we train 50 ReLU networks with standard initializations on MNIST images with labels from 0 to 4. Evaluated on MNIST images for labels 5 to 9, for a quarter of all images all 50 classifiers agree on the same label (which is of course wrong, since it is one of known labels 0 – 4). If we use smaller initialization, all 50 classifiers agree even for the majority of inputs.
We can understand why this similarity of independently initialized networks is particularly prevalent for “small” initializations: We expect that Gradient Descent finds local optima “close to” the initial point. If we start with high random initializations, we may have a better chance that they are far apart and converge to different solutions, but for small initialization the networks are likely to converge to
the same closest solution. This can be made precise for ReLU networks with one hidden layer and “infinitesimal” initialization, see appendix H.
As a method between RBF networks and Fourier networks, we can also try using a Gauss function as an activation function. While this does show some of the benefits of Fourier networks, extrapolating with Gauss functions predictably reverts to zero, whereas the sum of periodic functions become essentially random far from the training distribution, resulting in far more diverse ensembles, see appendix F, figure 10.

heading: 4 PROBLEM: CONSTANT FUNCTIONS ON THE TRAINING MANIFOLD
text: The third problem we see may be a general problem for using networks trained as classifiers also for out–of–distribution detection: If we have a feature (function) that is constant on the training distribution, but takes on other values out–of–distribution, we can use it to detect out–of–distribution inputs. However, when we train for distinguishing between different parts of the input distribution, such a feature is useless and may be dropped during training.
How do such features arise? One simple possibility is that some part of the input is constant. For example, for MNIST images, the corner pixels are always white. But in notMNIST or fashionMNIST this is not always true (see e.g. the ”E” in figure 17), and thus this gives an easy way to detect images not in MNIST.
Another possibility is that there may be different ways to come to the conclusion that an input gets a certain label L. For example, for MNIST we usually can already guess the input has label ”0” when we only see the top or the bottom half of the image. Let us say features f and g get value 1 if their clues to detect label L is present in the image, and are 0 otherwise. Then f − g is always 0 on the training set (as they are 1 if and only if the image is of label L), but it may take on different values out–of–distribution. (For some preliminary experiments with forcing such different features into separate networks, see appendix O).
Why would such features be “dropped”? If we start from small initialization, only weights that are useful for distinguishing between labels will be increased. So e.g. if the corners are always white, there is no incentive to change the weight that connects a corner pixel to the next layers, and if we start from small initialization, these pixels will never contribute to the end result.
So an obvious mitigation is to use larger initialization: Although the weights will still not be changed, the pixel will likely have an influence on the end result, and in different networks of the ensemble this influence will be different. So if the pixel is not white, this may lead to the different classifiers disagreeing about the label.

heading: 5 EXPERIMENTS: TWO HIDDEN LAYERS, FULLY CONNECTED
text: We follow the example given in (Lakshminarayanan et al., 2017) and look at fully connected networks with two hidden layers of 200 neurons, which we train on MNIST. For the baseline network, we use the model presented in (Lakshminarayanan et al., 2017), namely ensembles of networks with ReLU activations and the standard initialization of the weights (σ = √ 2/n with n = 784 for the first layer, n = 200 for the second layer). For the Fourier network we change the activation function of the deeper layer to sin and use σ1 = 0.75 and σ2 = 0.0002.
We train the networks on the training set of MNIST (60000 images), and evaluate them both on the test set of MNIST and on other data sets, here we use notMNIST (see (Bulatov, 2011)) and fashionMNIST (see (Xiao et al., 2017)) as the “outlier” data sets. For other data sets see appendix K). The ROC curves that are obtained this way are presented in figure 1. The variation of these curves is captured by figure 15 in appendix I.
While using an ensemble of ReLU nets gives a very substantial improvement over using one ReLU net, the result is still not close to what humans would achieve: The classifiers used here have some 98% accuracy, if we allow not recognizing an image as MNIST digit in 2% of the cases, we might expect to recognize that the printed letters of the alphabet are no handwritten digits in most cases, but in the above graph 2% error rate for MNIST corresponds to “only” 80% success rate on notMNIST. However, the ensembles of Fourier networks recognizes almost all of notMNIST images if we allow 2% error rate on MNIST.
In appendix M, figure 22 shows the corresponding results for training on the more challenging data set fashionMNIST, the results are qualitatively the same.
In the literature, the performance of such “out-of-distribution” detection is often measured by the area under the ROC curve, in Table 1 are numbers for MNIST vs. notMNIST.
Some of the approaches in Table 1 use outliers for training/tuning the model, so they are dependant on how similar the outliers used for training are to those used at test time. For more details about the baselines we refer the reader to Appendix R.

heading: 6 EXPERIMENTS: EFFECT OF INITIALIZATION VS. ACTIVATION
text: If we use only one hidden layer, we can only vary the σ for weights between the input and the hidden layer (we keep the initialization of the output layer at the standard σ = √ 2/n). This makes it easy to plot how much of the effect is due to “high initialization” vs. the choice of activation function.
We plot the area under the ROC curve in dependence on initialization: On the x-axis we plot log(σ), on the y-axis the negative logarithm of the “error” := area above the ROC curve.
The error bars correspond to two standard deviations (we has used randomly sampled ensembles of 10 networks out of 50 pre-computed networks).
We see in the first three images of figure 5 that notMNIST images are the easiest to detect, and flipped digits from MNIST are the most difficult, but in each case the Fourier networks clearly outperform the ReLU networks. One reason why notMNIST is particularly easy probably is that its images often can be recognized by just looking at pixels close to the corners: In MNIST these pixels are always white, but in notMNIST often not. To eliminate this shortcut, we also consider “masked” versions of notMNIST and fashionMNIST in which these pixels are set to white (see appendix K for details). The rightmost image in figure 5 shows the corresponding graphs for the
masked version of notMNIST, see figures 20 and 21 in appendix L for masked fashionMNIST and images in which each pixel is independently sampled from MNIST images, so each pixel has the same value distribution as in MNIST. Even with the “looking at the corners shortcut” disabled, the Fourier networks still perform well.
The vertical line corresponds to σ = 0.2, which seems to be close to optimal for Fourier networks for all test sets. This corresponds to a standard deviation of around 1.4 for the input to the sin(x) activation function (for our encoding of the MNIST pixels, which we scale into the interval [−1, 1]). We also see that “larger than usual” initialization alone already helps for ReLU networks, but using the sin(x) activation function gives a significant additional benefit. (See also appendix N for more general initialization schemes.)

heading: 7 EXPERIMENTS: CONVOLUTIONAL NETWORKS ON SVHN VS CIFAR10
text: The same phenomena described in the previous section for fully-connected networks apply for convolutional networks as well, thus allowing one to reap the benefits of using Fourier networks for out-of-distribution detection on more complex image datasets. To illustrate this, we train ensembles of classifiers on the Street View House Numbers dataset (SVHN) (Netzer et al., 2011) and use CIFAR10 (Krizhevsky et al., 2009) as out-of-distribution samples. Once again, we compare against a baseline of ReLU networks, first described as performing well on this task in (Lakshminarayanan et al., 2017).
Three different convolutional architectures have been considered, with increasing complexity: i) a 2-layer convolutional block followed by a fully-connected layer with ReLU activations and one with Sine activations dubbed SimpleCNN, ii) a network containing the first 4 layers from the classic VGG16 architecture (Simonyan & Zisserman, 2015) followed by a sine fully-connected layer referred to as ShallowVGG, and iii) the full-fledged VGG16 network. Appendix J contains the exact hyperparameter values used for each network.
Table 2 presents the best results obtained using this setup for both the ReLU and Fourier networks. Out-of-distribution samples are detected using two approaches, by selecting a threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble (presented in appendix Q) or for the entropy of the ensemble’s prediction (presented in Table 2). For evaluation, we used the area under the ROC curve (AUROC↑) as well as the false positive rate at 80% true positive rate (FPR80↓). The arrows indicate how the metrics change as the model gets better (i.e. ↑ if a larger value is better and ↓ if a smaller value is better). Additional results are presented in appendix Q. One other way in which one can assess qualitatively the performance of an out-of-distribution detection approach is by inspecting the distribution of the entropy of the softmax outputs over in-distribution and out-of-distribution samples. Ideally, the entropy for in-distribution samples stays close to 0 (meaning that the model is certain about its predictions) while the one for outof-distribution samples is pushed as far from 0 as possible (the maximum entropy that a 10-class classifier can achieve is log 10 ≈ 2.3). Figure 6 shows how Fourier networks compare to regular ReLU networks for the Shallow VGG architectures (with σ = 0.01 used as the standard deviation of the initialization distribution of the last layer). The Fourier networks make lower certainty predictions about out-of-distribution samples.
The histograms for the ReLU networks are similar to the ones reported by (Lakshminarayanan et al., 2017) on the same task using ensembles of networks, with or without adversarial training. The curves correspond to ensembles of 1, 5, and 10 models.

heading: 8 CONCLUSION
text: We suggested using the sin(x) activation function in the last layer, and larger than usual weight initializations to mitigate three problems we saw in using ensembles of ReLU networks for out–of– distribution detection:
• “Unreasonable” extrapolation: While ReLU networks tend to get more confident away from the training set, Fourier networks get decreasing confidence in the average.
• “Unreasonable” agreement between the networks in an ensemble: Larger initialization can make the networks more diverse, and the mixture of sin(x) functions with different frequencies makes the networks more random away from the training set.
• Filtering out of features that distinguish the training distribution from some out–of– distribution inputs, but do not contribute to the classification: Larger initialization can preserve such features, they do not influence the in–distribution classification, but create diversity in the ensemble when applied to out–of–distribution input.
We showed that this combines the out-of-distribution behavior from nearest neighbor methods with the generalization capabilities of neural networks, and achieves greatly improved out-of-distribution detection on standard data sets (MNIST/fashionMNIST/notMNIST, SVHN/CIFAR10).

heading: A APPENDIX A: MORE RELATED WORKS
text: Related to (Lakshminarayanan et al., 2017) is the suggestion of “MC dropout” (Gal & Ghahramani, 2016), here the ensemble of networks is replaced by one network, from which different predictions are produced by Dropout at prediction time. This is faster in training and evaluation, but in general not as effective as training the networks independently.
Another method that is also suggested in (Lakshminarayanan et al., 2017) is adversarial training, but its effect seems to be small compared to the effect of using ensembles instead of single networks.
Using a RBF layer on top of a normal network was first suggested in (Y. LeCun & Haffner, 1998), and is further investigated in (Zadeh & Hosseini, 2018).
While we are avoiding the use of “outliers” as training input, it can be helpful. A recent work in this direction is (Hendrycks et al., 2019). “Interesting outliers” can also be generated by a GAN, see (Lee et al., 2018) and (Kliger & Fleishman, 2018).
We focused on the use of uncertainties for out–of–distribution detection, and for this purpose it does not matter whether the “probabilities” are really correct (as long as they are higher for in–distribution than for out–of–distribution). However, if one wants “correctly scaled” probabilities, one can try to learn the right scale from the training data, see (Pawlowski et al., 2018).
We used the classifying networks and their “probability” output also for one–class–classification, as this is the computationally cheapest solution. But another intuitive approach is to use a generative model to produce a probability that an input belongs to the training distribution (Bishop, 1994). However, (Choi et al., 2018) and (Nalisnick et al., 2018) showed how this can fail and actually produce higher probabilities for out of distribution inputs. An interesting method that fixes this problem was recently described in (Ren et al., 2019).
For more works related to sin(x) as activation function, see the next section.

heading: B ACCURACY AND TRAINING WITH SIN AS ACTIVATION FUNCTION
text: We did not investigate in detail the general behavior of the sin(x)–function as an activation function with respect to the usual accuracy and training behavior, but in our experiments, we did not see any significant difference to ReLU(x) networks inside the training distribution.
For example, the table below shows the classification accuracy of ensembles of 5 ReLU networks and Fourier networks on the SVHN test set. For each case, the accuracy is reported for the ensemble with the best out-of-distribution detection performance (the same that are presented in Table 2).
These observations are consistent with the remarks in chapter 6.3.3. “Other Hidden Units” in (Goodfellow et al., 2016).
It is usually said that ReLU works much better than other activation functions for deep neural nets, but that is less relevant for us since we only use sin(x) in the last layer.
Nevertheless, there may be special cases in which sin(x) behaves better or worse than ReLU, as has been observed in e.g. (Sopena et al., 1999) and (Parascandolo et al., 2017).
Another way to formulate what a sin(x)–layer does, is that it computes a sparse Fourier approximation. Sparse Fourier approximation has been extensively studied (e.g. (Gilbert et al., 2002), (Candes et al., 2006)), but in a quite different context: Usually the aim is to fully reconstruct a function from a small amount of input / output data. This is possible when the function is known to have a sparse
Fourier representation. In our context it is unlikely that our functions have a sparse Fourier representation, and instead of trying to reconstruct the function the aim is rather to use different sparse Fourier interpolations to show the uncertainty we have in areas where we do not have data.

heading: C PROOFS OF PROPOSITION 1 AND 2
text: We study the Gradient Descent dynamic of a neural net with
• input ~x ∈ Rd
• one hidden layer with N neurons with activation function cos, connected to the input layer by fixed weights ~wi ∈ Rd, i = 1, 2, ..., N
• one output f(~x) ∈ R, connected to the hidden layer with weights ui ∈ R.
The output of this neural net at the point ~x is given by
f(~x) = N∑ i=1 ui · cos(~wi · ~x+ bi)
(We are using only one output dimension since for Proposition 1 we will be training it only on one data point, so the outputs for other labels do not matter for now.)
We use the cross entropy loss, which is for a data set of one point ~x0 with the correct label given by L = log ( 1 + e−f(~x0) ) .
The derivative of L for some parameter θ (i.e. one of the bi or ui) is
∂ ∂θ L = − 1 1 + ef(~x0) · ∂ ∂θ f(~x0).
With di := ~wi · ~x0 the last derivative is ∂
∂bi f(~x0) = −ui · sin(di + bi)
∂
∂ui f(~x0) = cos(di + bi)
We will assume “infinitesimal” learning rate, compensated by “infinitely many” steps, so we get a differential equation for the parameters, and a learning “velocity” α(t). Then we get as differential equation
∂ ∂t bi(t) = − α(t) 1 + ef(~x0) · ui · sin(di + bi(t))
∂ ∂t ui(t) =
α(t)
1 + ef(~x0) · cos(di + bi(t)) (2)
We assume we can train as long as we want, and we are not interested in how fast this procedure is proceeding. This means we are only interested in the direction of the vectors in (equation 2) and can as well look at the re-scaled vectors giving the differential equation
∂ ∂t bi(t) = −ui · sin(di + bi(t))
∂ ∂t ui(t) = cos(di + bi(t)) (3)
These equations are independent for each neuron i, we we can fix a neuron i and drop the subscript. Introducing the function
c(t) := cos(d+ b(t))
and using the dot notation for the time derivative we get the simple differential equation
ċ = u · (1− c2) u̇ = c (4)
(See figure 7 for a plot of the corresponding vector fields.)
Lemma 1: For (Lebesgue–)almost all (c0, u0) ∈ [−1, 1] × R the solution curve of (equation 4) starting at t = 0 at (c0, u0) will go either to (1,+∞) or to (−1,−∞) for t→∞. Proof of Lemma 1:
Since we have a symmetry (c, u) 7→ (−c,−u), it is enough to consider u0 ≥ 0. For c0 ∈ {±1} we obviously have c constant and u(t) = u0 + t · c0 → ±∞. So we will assume |c| < 1. For 0 < c0 < 1, 0 < u0 the c(t) is increasing, hence we must have u(t) ≥ u0 + t · c0 → +∞. So we can assume also ut ≥ 1. Then c(t) must be above the solution to the differential equation ḟ = 1− f2, which has the solutions
f(t) = e2t − a e2t + a ,
so we also must have 1 ≥ c(t) ≥ f(t)→ 1, hence c(t)→ 1. For c0 = 0, u0 > 0 or u0 = 0, c0 > 0 we move for small t > 0 into the domain c > 0, u > 0, so we know that for them as well we have c(t) → 1, u(t) → +∞. By symmetry, for u0 = 0, c0 < 0 we will have c(t)→ −1, u(t)→ −∞. For c0 = 0, u0 = 0 the functions c, u are constant 0, so this is a point we have to exclude.
So the remaining part is −1 < c0 < 0, 0 < u0. From (equation 4) we see that in that case ċ > 0 and u̇ < 0. So a solution curve starting in the quadrant −1 < c0 < 0, 0 < u0 has to leave this quadrant by either
1. Going through a point c = 0, u > 0, (light blue in figure 7), or
2. Going through a point c < 0, u = 0, (light red in figure 7), or
3. approaching c = 0, u = 0 (red line in figure 7).
We have already seen that the first two possibilities lead to c(t), u(t) → 1,+∞ and c(t), u(t) → −1,−∞ respectively, so it remains to show that the points that will approach c = 0, u = 0 are a null set. In fact, we show that for each −1 < c0 < 0 there can be at most one u0 with that property: From (equation 4) we see that the tangent to a solution curve through a point (c0, u0) has slope u · c/(1− c). If we look at the vertical distance between two solution curves in our quadrant, this means that with increasing c their distance also increases. Therefore, two curves with positive distance at c0 cannot converge both to (0, 0), and for each given c0 there can be at most one u0 in our quadrant such that (c0, u0) belongs to the third set above.
With Lemma 1 we can now prove:
Proposition 1: Assume the weight vectors ~wi are sampled from a normal distribution N (0,Σ) and then fixed, the numbers bi and ui are sampled independently from some distribution (not identically zero) with finite second moments (e.g. a normal distribution) and then trained on one data point ~x0 with label 1, then Gradient Descent (for the usual cross-entropy loss) will make the network’s output converge to a function that is approximately proportional to e−~x
T Σ~x/2. This approximation becomes exact when the number of neurons approaches infinity, or when we take the expected value.
Proof:
We assume that we have trained equation 2 for some time T long enough such that we are in the situation of Lemma 1, by symmetry (changing ui 7→ −ui, bi 7→ bi + π if necessary) we can assume that we have always ui 0 and cos(di + bi) ≈ 1. Setting Z := ∑ i ui, we have
1 Z f(~x) ≈ 1 Z N∑ i=1 ui · cos ( ~wi · (~x− ~x0) ) (5)
≈ E~w∼N (0,Σ) [ cos ( ~w · (~x− ~x0) )] (6)
(For the second approximation we use that we have sampled the ~wi independent from the ui.) This is then the Fourier transform of the Gauss function and we get
1 Z f(~x) ≈ e−~x T Σ~x/2.
The approximation (equation 5) becomes exact when we train infinitely long, and (equation 6) when we have infinitely many neurons (or take the expected value).
Proposition 2: Under the same idealized conditions (infinite number of neurons, weights ~wi frozen), but arbitrary finite training set, and assuming the training converged, the function goes to 0 for |~x| → ∞. Proof: We assume we trained on M points x1, ..., xM (with the correct label) and use a shallow Fourier net with N ≈ ∞ neurons. We also assume that the training “converged” in the sense that the bj converged (the uj keep growing). Analog to the above case of one training point, we have
∂L ∂bj = − M∑ k=1
1
1 + ef(~xk) · uj · sin(~wj · ~xk + bj).
With
ek := 1
1 + ef(~xk) , E(~w) := M∑ k=1 ek · ei ~w·~xk
we can write this using the imaginary part =:
∂L ∂bj = −uj=
( eibjE(~wj) ) .
If E(~wj) = 0, the value of L does not depend on bj ; we first show that this only happens for a (Lebesque–)Null set of ~w.
The real part <(E) is a real analytic function, and its value at 0 does not vanish since
E(0) = M∑ k=1 ek > 0.
Therefore the zero set of <(E) has Lebesgue measure zero (an exercise in Calculus II, according to, and solved in (Mityagin, 2015)). This implies that the even smaller zero setZ := {~w ∈ Rd |E(~w) = 0} also has Lebesgue measure zero. Using the arg function arg : C \ {0} → R/2πZ, r · eiφ 7→ φ we can define the continuous function
B : Rd \ Z → R/2πZ, ~w 7→ arg(E(~w))
which makes eiB : Rd \ Z → C, ~w 7→ ei·B(~w)
a well defined continuous function.
Now for ~w ∈ Rd \ Z and uj 6= 0
∂L ∂bj = −uj=
( eibjE(~wj) ) depends on bj ∈ R/2π, and L is minimal if this derivative crosses from negative to positive values around bj , which happens exactly when
bj = { −B(~wj) uj > 0 −B(~wj) + π uj < 0
So for almost all ~wj the weight ~wj determines the phase bj in a way that depends continuously on ~wj . With this our function becomes∑
j
|uj | · cos ( ~w · ~x−B(~wj) ) and after rescaling by dividing byZ := ∑ i |ui| this converges in the limit of infinitely many neurons to f(~x) := E~w∼N (0,Σ) [ cos(~w · ~x−B(~w)) ] .
With the Gauss function
φΣ(~w) := e−~x T Σ~x√ (2π)d · det(Σ)
giving the probability density function for N (0,Σ) this is
f(~x) = ∫ Rd φΣ(~w) · cos(~w · ~x−B(~w)) d~w
= < (∫
Rd φΣ(~w) · ei ~w·~x · e−iB(~w) d~w ) = <
(∫ Rd φΣ(~w) · e−iB(~w) · ei ~w·~x d~w ) ,
which is up to a multiplicative constant the real part of the Fourier transform of the function
g(~w) := φΣ(~w) · e−iB(~w).
Since |e−iB(~w)| = 1 and φΣ ∈ L1(Rd), this function is also in L1(Rd), so the Riemann–Lebesgue Lemma gives that f(~x)→ 0 for |~x| → ∞ as claimed.

heading: D RELU NETWORKS IN THE SIMPLEST 1-D PROBLEM
text: After randomly initializing the weight wi ∼ N (0, σ) and bias bi ∼ N (0, σ) for neuron i, the location of the resulting kink of ReLU(wi · x + bi) is at −bi/wi, so it is Cauchy-distributed with center 0 and part of them will fall on the right hand side of x0, and another part on the left hand side. Training the parameters tries to increase the value at x0, but it will not move the kink across x0. So we will with high probability (probability 1 for infinitely wide networks) see the sum of functions of which some increase for large positive x, others increase for very negative X , so in general the expected picture is indeed curving upward on both sides, like the experimental result in the first image of figure 2.

heading: E FEW POINTS
text: We construct as small set of points around -5, -2, 3, 9:
S = {−5.3,−5.,−5.1, −2.1,−2.,−1.88, 3.2, 3.1, 8.95, 9.1, 9.22} The Fourier approximation gives figure 8.

heading: F GAUSS NETWORKS
text: We can define “Gauss networks” for which we take as activation functions Gauss functions
g(x) := e−x 2/2
Using this function instead of ReLU(x) or sin(x) gives in the 2-dim toy example the figure 10 (compare with figure 4).

heading: G COMPARISON TO NEAREST NEIGHBOR METHODS
text: The “nearest neighbor” classifier assigns a label L to a test input if the closest training input had label L. We can also use the “nearest neighbor” method for out of distribution detection: For a
threshold r we call an input “in distribution” if and only if its nearest neighbor in the training set has distance ≤ r. Varying the r then gives a ROC curve, analogous to varying the threshold θ for the softmax output of a network ensemble.
There are many variations of this simple method (k–nearest neighbor, RBF networks, adjusting the metric to the data), but for the general principles we want to discuss here the differences do not matter and we will use the basic implementation.
In (Bishop, 1995), chapter 5.9, pp. 183–186 the author writes about the “major potential difficulties with radial basis function networks” stemming from the “localized nature” of the hidden units (in our implementation of the nearest neighbor method the hidden units would correspond to the training points).
The problem is that nearest neighbor methods “just memorize” training points, and only generalize to nearby points – this requires that we have a training point “near” any possible input point of our training distribution. On the other hand, “normal” networks learn “rules”, which can generalize the “important parts” and ignore the “noise” that does not contribute to the classification.
This can be a problem for nearest neighbor methods mainly in high dimensions, but as a first illustration we reproduce the two–dimensional toy problem from the book: Assume we have three classes (red, green, blue) that can be distinguished by their x1–coordinate, but we also have a random x2–coordinate with high variance (see figure 11).
If the red / green / blue stripes have length δ along the x1–axis, a nearest neighbor classifier or RBF network may pick circles of radius around δ/2 to characterize the points that belong to the same stripe. However, for that we need to have enough points also in x2 direction, if there is a gap of much more than δ, we will have points in a stripe that are not recognized as belonging to the training distribution (or to the corresponding label).
On the other hand, a ReLU network will learn some regions and (over)confidently classify all points in these regions as red / green / blue.
We can demonstrate in this toy example how the Fourier network ”interpolates” between a “ReLU behavior” and a “Nearest Neighbor behavior” when we adjust the standard deviation used in the weights’ initialization of a Fourier network: For small initializations, we are close to the “generalizing” ReLU behavior, large initializations give something close to a “memorizing” Nearest Neighbor behavior. We show in figure 12 how an ensemble of ReLU networks and ensembles of Fourier Networks generalize to the 2D plane. The upper row gives the classification around the input, the lower row zooms out, in particular showing that for Fourier networks the ensembles are not extending their confidence region to infinity (as the ReLU networks do).
To illustrate the resulting effect in higher dimensions, we add some “random background” to the MNIST and fashionMNIST images (see figure 13).
The “random background” plays the role of the x2 coordinate above: For a new input, the nearest neighbor classifier now has to find a training image that is close to the new one both in the digit and in the background. This is of course much harder (even with 60000 training points), and correspondingly the classification accuracy of the nearest neighbor classifier drops from 96.9% to 76.2% (see table 4). For ReLU (and Fourier) nets, we also see a drop, but only to 91.3%, the classifier just learns to “ignore” the background. (The classification accuracies reported here are the accuracy of the averaged softmax output of the networks in an ensemble.)
For out of distribution detection, we see the same effect – for the nearest neighbors method the area under the ROC curve drops from 98.9% to 77%, but for ReLU nets only from 91.8% to 91.1%. The Fourier networks are supposed to combine the best of both approaches, and indeed they give better out–of–distribution detection than either method in both cases.
As a second example, we produce 60000 wider images by randomly selecting 4 MNIST digits and combining them to one new image. As the label of these images we take the label of the first digit. For out–of–sample images we replace the first digit by a fashionMNIST image, see figure 14.
Again, this makes it much harder for nearest neighbor method — now it has to find images in which all 4 digits are similar (which is still possible with 60000 training images, but harder), whereas the ReLU classifier can focus on the pixels in the first quarter of the image and learn to ignore the rest, see table 5 for the results.
An interesting philosophical question arises when we produce images by exchanging the last digit by a fashionMNIST image: Are the resulting images “outliers” or not? In any case, the results in table 5 show that the ReLU classifiers now do not flag many “outliers”, which is in agreement with our expectation that they focus on the left quarter of the image and mostly ignore the rest.

heading: H RELU NETWORKS WITH INFINITESIMAL INITIALIZATION
text: For given training data and infinitesimal initialization, there are only finitely many possibilities to which networks can converge (independent of network size), see (Maennel et al., 2018). The arguments given in that paper show also that with increasing number of weights we should with high probability end up at the same possibility.

heading: I MORE RESULTS ON MNIST
text: I.1 VARIATION OF ROC CURVES FOR MNIST VS NOTMNIST AND FASHIONMNIST
The ROC curves obtained when considering MNIST as in-distribution and notMNIST samples as out-of-distribution and using two-layer classifiers for training were shown in figure 1. To visualize the variation of these curves, we compute 50 networks, and apply this method 10 times on ensembles of 10 randomly selected networks, this gives the left hand graph in figure 15. (The right hand graph in figure 15 shows the corresponding results for testing for “outliers” from fashionMNIST. We also get similar figures when training on fashionMNIST, see appendix M.)
x-axis: MNIST samples not recognized as belonging to MNIST, y-axis: notMNIST samples recognized as not belonging to MNIST.
I.2 ENTROPY HISTOGRAMS
To compare our method also to the evaluation given in (Lakshminarayanan et al., 2017), we compute for each image the entropy of the softmax output and plot a histogram over the MNIST and the notMNIST data set for the ensemble sizes 1,5,10. The first three columns of figure 16 (copied from figure 3 in (Lakshminarayanan et al., 2017)) show the resulting entropy of the softmax output for ReLU networks, first using the ensemble as is, then with additional adversarial training, and in the third column the corresponding results for “dropout at test time”.
For input from the MNIST test set (first row, blue curves) the entropy is almost always close to 0, i.e. the classifier puts almost all probability on one label. For input from the notMNIST data set the classifier ensembles distribute the probability over several labels, resulting in much higher entropy. Use of adversarial training (second column) increases the entropy even a bit further for the notMNIST examples.
The fourth column shows the results for the corresponding Fourier networks. On the MNIST test set the behavior is basically the same as the usual network, but on notMNIST even one network often is unsure about the label, and an ensemble of 10 networks gives most of the time an entropy around 2 (the maximum would be log(10) ≈ 2.3 for the uniform distribution across all 10 labels).
I.3 CONVOLUTIONAL NETWORKS ON MNIST
Since MNIST vs. notMNIST is too easy to profit from better classifiers, we use a more difficult problem: Train on classes 0-4, see whether we detect classes 5-9 as “out of distribution”.
For MNIST we use two convolutional layers with 7×7 windows and max pooling in 3×3 windows with stride 2, and add two fully connected layers with 200 / 100 neurons. We also use data augmentation by randomly cropping to 24 × 24 pixels. For the Fourier network we change the activation function of the last layer to sin(x).
First experiments seem to indicate that a good way to choose the initialization for the different layers of the Fourier network is to start with small initializations for the first layers and the last layer, and then choose the second last initialization as large as possible (see appendix J.5 for details).
With this setup we get for the area under the ROC curve:

heading: J DETAILED SPECIFICATIONS OF THE NEURAL NETWORKS AND THEIR PARAMETERS
text: J.1 ONE DIMENSIONAL TOY EXAMPLE
• Input data: Only one point at x0 = 2 with label ”1”.
• Network: One hidden layer with 500 neurons, weights initialized with σ = 0.0632 ≈ √ 2/500.
• Learning: – ReLU: 500 steps with learning rate 0.001 – sin: 5000 steps with learning rate 0.001
J.2 TWO DIMENSIONAL TOY EXAMPLE
• Input data: 20 centers randomly chosen in −10 ≤ x ≤ 10, −10 ≤ y ≤ 10. To each center one of the labels ”Red”, ”Green”, ”Blue” was randomly chosen. For each center (x, y) there were 50000 points randomly drawn from a normal distribution N (µ,Σ), where Σ = R ·D2 for a randomly drawn rotation matrix R and D the diagonal matrix with entries σ1 = 0.2 and σ2 = 0.5.
• Network: One hidden layer with 500 neurons, activation function sin or ReLU. Weights (including bias terms) initialized with σ = 12.
• Learning: 26 epochs, batch size 256, learning rate 0.02.
J.3 784 DIM INPUT, ONE HIDDEN LAYER (FULLY CONNECTED), TRAINED ON MNIST
Input: 764 pixels, normalized to be in [−1, 1] Hidden layers: One hidden layer of 100 neurons
Initialization: σ ∈ {0.002, 0.01, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2, 0.22, 0.25, 0.275, 0.3, 0.35, 0.4, 0.5, 1.0, 2.0, 4.0, 6.0}
Batch size: 256
Learning rate: 200 epochs with 0.02, 200 epochs with 0.005.
J.4 784 DIM INPUT, TWO HIDDEN LAYERS (FULLY CONNECTED), TRAINED ON MNIST
Input: 764 pixels, normalized to be in [−1, 1]
Hidden layers: Two hidden layer of 200 neurons, first layer: ReLU, second layer: ReLU / sin.
Initialization:
• For ReLU network: σ1 = 0.07, σ2 = 0.14. • For Sine network: σ1 = 0.75, σ2 = 0.0002
Parameters searched: For ReLU this is the standard initialization σ = 2/ √ (n).
For Sine networks, initial tests gave the result that for a given σ2 the σ1 should be as large as possible such that the training still converges. Then I tried σ2 ∈ {0.0002, 0.001, 0.01} and σ1 ∈ {0.25, 0.5, 0.75, 1.0}. Learning rate was 0.01.
Best results: (σ1, σ2, area under ROC curve for MNIST vs. notMNIST):
0.5, 0.0002, 0.99898
0.5, 0.001, 0.99908
0.5, 0.01, 0.99946
0.75, 0.01, 0.99974
0.75, 0.001, 0.99977
0.75, 0.0002, 0.99977
J.5 784 DIM INPUT, CONVOLUTIONAL NETWORKS, TRAINED ON MNIST
Input: 764 pixels, normalized to be in [−1, 1], Preprocessing: 4 times random cropping from 28× 28 to 24× 24. Hidden layers:
• Convolution, 7× 7 windows, 1→ 16 channels (to size 24× 24× 16) • Max pooling in 3× 3 windows with stride 2 (to size 12× 12× 16) • Convolution, 5× 5 windows, 16→ 32 channels (to size 12× 12× 32) • Max pooling in 3× 3 windows with stride 2 (to size 6× 6× 32) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 100 neurons, ReLU or sin(x) activation function
Batch size: 40
Learning rate: 25 epochs 0.002, 20 epochs 0.001, 20 epochs 0.0005
Initialization: Given by σ = (σ1, σ2, σ3, σ4).
For Fourier networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
• σ = (0.001, 0.001, σ3, 0.0001), best result: 99.1% for σ3 = 20 • σ = (0.002, 0.001, σ3, 0.5), best result: 99.1% for σ3 = 1.5 • σ = (0.1, 0.1, σ3, 0.001), best result: 98.8% for σ3 = 1.75.
For ReLU networks:
After initial attempts systematic evaluation of area under ROC curve for MNIST 0-4 vs. MNIST 5-9 for
• σ = (0.01, 0.01, σ3, 1.0), best result: 97.8% for σ3 = 0.3 • σ = (0.01, 0.01, σ3, 0.6), best result: 97.4% for σ3 = 0.3 • σ = (σ1, σ1, 0.1, 0.1), best result: 97.7% for σ1 = 0.9 • σ = (σ1, σ1, σ1, σ1), best result: 98.1% for σ1 = 0.55
J.6 SIMPLE CNN, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0, 1],
Hidden layers:
• Convolution, 7× 7 windows, 1→ 16 channels (to size 32× 32× 16) • Max pooling in 3× 3 windows with stride 2 (to size 16× 16× 16) • Convolution, 5× 5 windows, 16→ 32 channels (to size 16× 16× 32) • Max pooling in 3× 3 windows with stride 2 (to size 8× 8× 32) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 200 or 100000 neurons, ReLU or sin(x) activation function
Batch size: 128
Learning rate: 0.01 (for 200 neurons on the last layer) and 0.001 (for 100000 neurons). Halved every 40 epochs.
Best σ for initializing the last layer: 0.75.
J.7 SHALLOW VGG, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0, 1],
Hidden layers:
• the first 4 convolutional blocks of the VGG16 architecture (including the corresponding max pooling layers) (to size 8× 8× 128)
• Fully connected, 200 or 100000 neurons, ReLU or sin(x) activation function
Batch size: 128
Learning rate: 0.001. Halved every 40 epochs.
Best σ for initializing the last layer: 0.01.
J.8 VGG16, TRAINED ON SVHN
Input: 32x32x3-shaped images, with the pixels normalized to be in [0, 1],
Hidden layers:
• all the convolutional blocks of the original VGG16 architecture (to size 1× 1× 512) • Fully connected, 200 neurons, ReLU activation function • Fully connected, 100 or 100000 neurons, ReLU or sin(x) activation function
Batch size: 128
Learning rate: 0.001. Halved every 40 epochs.
Best σ for initializing the last layer: 0.25 and 0.1 for sin(x) and ReLU respectively, for 100 neurons on the last layer and 0.002 for 100000 neurons.

heading: K DEFINITION OF THE DATA SETS
text: As input distributions that are “clearly different” from MNIST we can use the fashionMNIST ((Xiao et al., 2017)) and notMnist ((Bulatov, 2011)) data sets, see figure 17.
We also use some constructed data sets:
• “Masked fashionMNIST”: In MNIST some pixels are always or almost always white, see figure 19. Out of fashionMNIST we derive this data set by setting all pixels to 0 (white) that have an average gray value of below 0.01.
• “Masked notMNIST”: This is derived from notMNIST in the same way.
• Circles and lines: Two lines and two circles in random positions.
• Flipped MNIST: Derived from MNIST by mirroring the digits 3,4,6,7,9 horizontally and the digits 4,6,7,9 vertically.
• IID noise: Uniform iid pixel values from [0.256].
• Independent pixels: Each pixel is randomly sampled from pixels at the same position at MNIST training images, independent of the other pixel values.
• PseudoMNIST: Here we try to mimic not only the value distribution for each pixel, but also local correlations. We start with an image in which each pixel is independently sampled like in the previous distribution. Then we perform 1000 (PseudoMNIST1) or 1500 (PseudoMNIST2) updates of the following form: We randomly select a 3× 3 window, and find the closest match in 1000 randomly sampled images from MNIST, where “closest” is with respect to the L2 distance of the 8 outer pixels of the 3 × 3 window. Then we replace the middle pixel in our image with the value of the middle pixel in this “closest match” from MNIST.
We will also use a “masked version” with a mask from MNIST: see figure 19 for the origin of the mask, and figure 18 for examples.

heading: L TRAINING ON MNIST, USING OTHER OUTLIER SETS
text: With these data sets we can again compute the ROC curves that show how well we can distinguish the training set from the outlier set by setting a threshold on the “confidence”. As confidence we take again the averaged softmax output of the predicted label over an ensemble of 10 classifiers, in which the weights are randomly sampled from a normal distribution with mean 0 and standard
deviation σ. We consider the area above the ROC curve as “error” and plot− log(error) against the initialization σ used for the networks. The resulting graphs are in figure 20.
In some cases, it is easy to distinguish MNIST samples from outliers by just looking at pixels close to the corners: In MNIST these pixels are always white, but in notMNIST often not. To eliminate this shortcut, we also consider “masked” versions of notMNIST and fashionMNIST in which these pixels are set to white (see appendix K for details). In figure 21 we show the corresponding graphs for the masked versions, and for images in which each pixel is independently sampled from MNIST images, so each pixel has the same value distribution as in MNIST. Even with the “looking at the corners shortcut” disabled, the Fourier networks still perform well.

heading: M TRAINING ON FASHION MNIST
text: We use the same two hidden layer network architecture, initializations, and learning rates as in figure 1, but train on fashion MNIST, the result is in figure 22 below. Since this is a more challenging data set, the same architecture gives not as good results as for MNIST (note the x-axis extends now to 0.2), but the difference between ReLU and sin(x) networks is comparable, resulting in graphs (figure 22) that look very similar to figure 1.
x-axis: MNIST samples not recognized as belonging to MNIST, y-axis: notMNIST samples recognized as not belonging to MNIST.
We can also plot the area under the ROC curve for different initializations of a network with one hidden layer - this gives figure 23. Again this is qualitatively the same result as in figure 5 or in figure20 above.
N INFLUENCE OF THE DISTRIBUTION FROM WHICH WEIGHTS ARE SAMPLED
Our method for getting a variety of networks relies on different randomly sampled initializations giving different networks. According to (Sirignano & Spiliopoulos, 2018) this actually no longer happens if the number of neurons is very large: Then the networks converge to an idealized network in which there are infinitely many neurons that have their weights distributed according to the probability distribution from which we sample.
To illustrate this potential problem visually, we look at a very simple 2-dimensional toy example, in which we can again plot how the classifiers generalize from the training distribution to other points, like we did in section 3.
As input we use only a line, with labeled training samples given as blue dots in the middle and red dots at the end (figure 24).
We will use a ReLU network with one hidden layer of 100 neurons to classify all points in the surrounding area of the plane. Colors encode the “probabilities” that the softmax computes from the output layer, with blue / red signifying “probability 0/1 of being blue”. The result is in figure 25.
If we change the activation function of the hidden layer from to ReLU to sin(x), we obtain different results outside of the training distribution (figure 26). While individual points are still mostly red or blue, i.e. the individual classifiers are still “overconfident”, the different classifiers now tend to disagree and averaging the softmax outputs over several of them we get much more realistic confidences for the Fourier networks than for the ReLU networks: Now the ensemble is only confident about the label for points in the vicinity of the training inputs.
To demonstrate the potential problem with very wide networks, we now use 20000 neurons (instead of the 100 we used in the above pictures). Now the results of different initializations become more similar to each other, see figure 27.
So in this case, we can no longer rely on different sampling from the same distribution, but need to change the distribution itself. For higher dimensional input, this effect may be more theoretical because the number of neurons required to get into that regime is no longer realistic.
However, even for a moderate number of neurons it may make sense to vary the distribution of initial weights from which we sample to obtain a greater diversity. Here is an example in our case of 2-dimensional input, 100 neurons, ReLU activation:
The weights in the 2-dimensional case have 3 coordinates: Two for the direction and one for the bias. We can now sample the “direction” weights according to a Normal Distribution with one standard deviation σ1, and the “bias” weight according to a Normal Distribution with a different standard deviation σ2. Different pairs (σ1, σ2) describe different distributions from which we initialize the weights.
If we vary now our weight initialization by varying (σ1, σ2), the resulting networks seem to converge to different classifiers, see figure 28.
In particular, the networks initialized by σ1, σ2 = (5, 5) could in theory contain the same initializations as the networks initialized by σ1, σ2 = (0.01, 1), but the emphasis shifts to different networks: The latter give blue cones extending downwards most of the time, whereas such a result is not among the 50 samples I obtained for σ1, σ2 = (5, 5).
So even for not extremely large networks it would make sense to restrict the distribution of weights from which we sample in different ways to obtain a more diverse set of networks.

heading: O SEPARATING DIFFERENT FEATURES FOR THE SAME LABEL
text: We try restricting the weights in a neural network classifier to certain subspaces – not only during initialization, but throughout the training. The hope would again be that classifiers with different restrictions develop different features which generalize in different ways to unseen examples, giving more variety for out-of-distribution samples.
In the MNIST case, we evaluate two strategies of restricting the weights of the first layer:
• Only allow access to half of the pixels. If each classifier sees only half of the image, they usually still can figure out which digit it is:
• Only allow access to “line detectors” with certain directions. We implement this by convolution with a 7 × 7–matrix which corresponds to a stripe of a certain direction. We allow 4 (of 10) directions per location, and to reduce the dimension of the resulting weight space we subsample the locations to a grid, see figure 30. This again gives a vector space of dimension which is roughly half of the original dimension.
This seems to give a significant increase of the area under the ROC curve in particular for the difficult case “trained on 0-4, evaluate 5-9 as outliers”:
Area under ROC curve
“Outliers” sin sin with half input sin with stripes combined MNIST 5-9 96.5% 97.3% 97.5% 97.6% fashionMNIST 99.3% 99.5% 99.6% 99.7% notMNIST 99.9% 99.9% 99.9% 99.9%
Same test with restriction to half images on fashionMNIST:
The classifier is trained on fashionMNIST 0-4.
“Outliers” relu, large init relu, restr. sin sin, restr. fashionMNIST 5-9 91.3% 92.5% 93.5% 94.3% MNIST 94.2% 95.1% 98.9% 99.2% notMNIST 93.4% 95.2% 99.6% 99.6%
We have also applied the same approach when training convolutional networks on SVHN. The table below shows the area under ROC curve obtained with a Simple CNN and a Shallow VGG. By applying a square mask on one of the corners of the image (top left, top right, bottom left, bottom right) we obtain an ensemble of constrained classifiers that is slightly more robust to out-ofdistribution samples.
Model Original input Masked input Simple CNN 96.6% 96.5% Shallow VGG 95.8% 96.6%
P INPUT TO ACTIVATION FUNCTION
In this network the first hidden layer has ReLU neurons, and the second hidden layer has sinNeurons. We plot the activation observed as input to the second layer, for the 5 neurons which have the largest weight going to the output of “0” (left), and the for the 5 neurons with the smallest weight going to the output of “0” (right) for all images that are indeed a “0”. We observe that the peaks are separated by 2π, the left at the maxima of sin, the right at the minima of sin. A similar pattern arises for other images from the training set.
However, when we plot the input to the sin activation for images from notMnist, we no longer see the peaks at maxima / minima of sin, suggesting that the output for these images will be “random”:

heading: Q MORE EXPERIMENTAL RESULTS FOR SVHN VS CIFAR10
text: Figure 33 shows which are the regions of optimal initialization for the final fully-connected layer for both ReLU and Fourier networks. Note that if the initialization is too large, gradient descent on the Simple VGG Fourier network will sometimes fail to converge.
The distribution of the entropy of the softmax outputs over in-distribution and out-of-distribution samples for a Simple CNN network is shown in Figure 34. Results are presented for both regular ReLU and sin(x) activations for the Simple CNN architectures (with σ = 0.75 used as the standard deviation of the initialization distribution of the last layer). The curves correspond to ensembles of 1, 5, and 10 models.
Table 7 presents the best results obtained using convolutional networks trained on SVHN for both the ReLU and Fourier setup. Out-of-distribution samples are detected using two approaches, by selecting a threshold for either the maximum probability as obtained after aggregating the outputs of the softmax layers in an ensemble or for the entropy of the ensemble’s prediction. For evaluation, we used the following metrics:
• the area under the ROC curve (AUROC↑) • the false positive rate at 80% true positive rate (FPR80↓) • the negative log-likelihood score (NLL↓) • the Brier score (Brier↓)
In order to be closer to the ideal case, which is the premise for theoretical results proved above, we also report numbers for the same network architectures with significantly more neurons on the final fully-connected layer (i.e. 100,000). For each configuration, the table contains the numbers for the
model with the best value for the initialization σ. The dependency between the area under the ROC curve and the initialization strength is presented in figure 33.
Network Last LayerActivation Last Layer Units Last Layer Init σ AUROC↑ using max-p FPR90↓ using max-p AUROC↑ using entropy FPR80↓ using entropy NLL Score↓ Brier Score↓

heading: R COMPARISON WITH OTHER OOD DETECTION METHODS
text: (Hendrycks & Gimpel, 2017) use a slightly larger network and GELU activation functions, (Liang et al., 2018) train also on images that do not belong to MNIST, but also not on the “notMNIST” images that are used for evaluation. Similarly, (Malinin & Gales, 2018) use samples from a set of outliers to tune the model; the more similar these are to the outliers used for testing, the better the performance of the model is. Table 9 shows the results obtained using the Dirichlet Prior Network model (DPN) for various settings. In all instances we used the default hyperparameter values (suggested either in the paper or in the code released by the authors1). It is important to note that for the setting on which the Prior Network model performs best half of the out-of-distribution test samples are drawn from the outlier distribution that was used for fine-tuning the model (namely the distribution of FashionMNIST samples from classes 0-4, since we use the whole test set of FashionMNIST as OOD samples at test time). Moreover, for the best performing DPN results we tweaked the original setup and allowed the model to use the FashionMNIST training samples belonging to classes 0-4, instead of the (fewer) test samples, as prescribed by the authors of the paper. However, even on this significantly simplified setup, the DPN baseline falls behind the Fourier network ensemble even though the latter never sees any outliers during training.
1https://github.com/KaosEngineer/PriorNetworks-OLD

Review summaries:
The paper presents a method to detect out-of-distribution or anomalous data points. They argue that Fourier networks have lower confidence and thus better estimates of uncertainty in areas far away from training data. They also argue for using “large” initializations in the first layers and sin(x) as the activation function for the final hidden layer.