Periodic Activation Functions Induce Stationarity
Authors: Lassi Meronen
Martin Trapp
Arno Solin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian deep learning
Gaussian process
uncertainty quantification

Abstract: Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.

Periodic Activation Functions Induce Stationarity
Lassi Meronen
Martin Trapp Aalto University Espoo, Finland martin. trappraaltppaaa.fff
Arno Solin Aalto University Espoo, Finland arno. www.solllnaa..o.oooooom..
Aalto University / Saab Finland Oy Espoo, Finland
lassi aaasi..eroneenaalto...
Abstract
Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that 'know what they do not know' by introducing inductive biases the function space. We show that periodic activation functions in Bayesian neural networks establish connection between the prior on the network weights and iossllaton-invariinn stationary Gaussian process priors. Furthermore, show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In series of experiments, functions obtain comparable performance for in-domain data and capture sensitivity perturbed inputs in deep neural networks for out-of-domain detection.
show that periodic activation
Introduction
Deep feedforward neural networks [46, 16] are an integral part of contemporary artificial intelligence and machine learning systems for visual and auditory perception. medicine, and general data analysis and decision making. However, when these methods have been adopted into real-world use, concerns related robustness (with respect to data that has been seen during training), fairness (hidden biases in data being reinforced by the model), and interpretability (why the model acts as it does) have taken a central role. The knowledge gathered by contemporary neural networks has even been characterised as never truly reliable [31]. These issues relate to the sensitivity of the trained model to perturbed inputs being fed through or the lack thereof.
This motivates Bayesian deep learning, where the interests are two-fold: encoding prior knowledge into models and performing probabilistic inference under the specified model. We focus on the former. Probabilistic approaches to specifying assumptions about the function space of deep neural networks have gained increasing attention the machine learning community, comprising, among others, work analysing their posterior distribution [57, 1], discussing pathologies arising in uncertainty quantification [9], and calls for better Bayesian priors (e.g., [42, 50, 37, 10]).
In this paper, we focus on stationary models, which act as a proxy for capturing sensitivity. Stationarity indicates pranllaton-invariande meaning that the joint probability distribution does not change when the inputs are shifted. This seemingly naive assumption has strong consequences in the sense that it induces conservative behaviour across the input domain, both in-distribution and outside the observed data. The resulting model is mean-reverting outside the training data (reversion to the prior), directly leading to higher uncertainty for out-of-distribution (OOD) samples (see Fig. I for examples). These features (together with some direct computational benefits) have made stationary models/priors the standard approach kernel methods [4, 20]. spatial statistics [5], and Gaussian process (GP) models [44], where the kernel is often chosen to induce stationarity.
Neural networks parametric models, which typically fall into the class of non-stationary models. Non-stationarity increases flexibility and is often a sought-after sopperty-especinn if the interest is solely in optimizing for accuracy in-domain test data. In fact, all standard neural network
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Non-stationarity
Local statiouarity
Global/full stationarity (this puper)
Finite-widi Infmitt-width
ArcCos-1 kernel /
Matém- J-NN
RBF-NN
Exponential (Cauchy prior)
Marérn-B & (t-distribution prior)
RBF (Gaussian prior)
ReLU activation [36] kerneVactiivtionn [34]
kemellactivation [59]
Figure 1: Posterior predictive densities of single hidden layer Bayesian neural networks (BNNs) with 30 hidden units and their infinite-width corresponding GPs on the banana classification task. Different activation functions induce different prior assumptions. Estimates obtained through HMC sampling [14] for 10k iterations.
activation functions (ReLU, sigmoid, etc.) induce non-stationarity (see [3, 59]). However, non- stationary models can easily result in over-confidence outside the observed data [18], [55] or spurious relationships between input variables (as illustrated in Fig. where darker shades show higher confidence).
Stationarity in neural networks has been tackled before under the restriction of local stationarity, i..,
only local, induced by a Gaussian envelope (as realized by [59] for the RBF kernel/activation). Recently. Meronen et al. [34] expanded this approach and derived activation functions corresponding to the widely used Matérn class of kernels [32, 44], We go one step further and derive activation functions that induce globul stationarity. To do so, we leverage theory from harmonic analysis [58] periodic functions, which helps expand the effective support over the entire input domain. We also realize direct links previous works leveraging harmonic functions in neural networks [13, 2, 61, 47, 56], and Fourier features in other model families [43, 51].
The contributions of this paper are: (i) We show that periodic activation functions establish direct correspondence between the prior on the network weights and the spectral density of the covariance function of the limiting stationary Gaussian process (GP) of single hidden layer Bayesian neural networks (BNNs). (ii) We leverage this correspondence and show that placing Student-t prior on the weights of the hidden layer corresponds to prior on the function space with Matérn covariance. (iii) Finally, we show a range of experiments that periodic activation functions obtain comparable performance for in-domain data, do not result in overconfident predictions, and enable robust out-of- domain detection.
Related Work
We build upon prior work on exploring the covariance function induced by different activation functions, starting with the seminal work by Williams [59], who discussed sigmoidal (ERF) and Gaussian (RBF) activation function, resulting in locally stationary kernel modulated by Gaussian envelope. Cho and Saul [3] introduced non-stationary GP kernels corresponding to the ReLU and step activations, and [54] later extended the approach to the leaky ReI.U and analysed alternative weight prior specifications. More recently, [34] derived activation functions corresponding to kernels from the Matérn family, which are locally stationary modulated by Gaussian envelope. In addition to the work connecting neural networks to GPs at initialisation, [21] created a connection between the neural network training and kernel methods by introducing the Neural Tangent Kernel (NTK).
At the same time, uninformative priors have been widely criticised in Bayesian deep learning [10]. and alternative strategies have been proposed to incorporate prior knowledge. Pearce et al. [42] proposed compositions of BNNs to mimic compositional kernels their limiting case. Sun et al. [50] proposed a variational objective for BNNs acting on the function space rather than the network parameters. Nalisnick al. [37] proposed tractable priors on the functional complexity, obtained through the change of variables formula applied to the KL-divergence from reference model/prior.

---

Highly nue statonury
ArcCas-1 ReL.U
ArcCss-O ~ Step
NN n Signoodd
ArcCos- ~ RaLU
ArdCze-l Step
NX Sigmoid
Locally stationry
XBF-NN
Maténa-3/a NNN
Esp-NN (local)
RBF-NN (loca))
Mat5m-3/2-NN (local)
Exp-NN
Statonary
RBF
Maiém-3/ 3/2
Exponeniall
RDF
Masérn-3/2
Exponential
(a) Gram matrices
(b) Regression results with corresponding prior
Figure 2: Left: Gram/covariance matrices (for k(x, x')) reflecting different prior beliefs on how infinite-widde NN model functions ought to behave outside the observed data. Right: ID regression results corresponding to the model induced by the priors, showing the posterior and predictive 95% intervals. See Fig. appendix for the corresponding finite-width results.
Furthermore, Morales-Alvaree et al. [35] proposed to encode prior assumptions on the activation level rather than the model parameters.
Recent works have shown the potential of periodic activation functions neural networks, including approaches learn implicit neural representations [47], methods for recognition of handwritten digits [61]. and out-of ontoo--distribution detection [30]. Earlier work used periodic activation functions to resemble the Fourier series [13, 56] and showed that such models universal function approximators [2]. In broader terms, [27] recently showed that the use of the Fourier transform, replacing self- attention, in Transformers can drastically improve the runtime. Inspired by the Fourier duality of covariance and spectral density, [43] proposed random Fourier features to approximate GP with an RBF kernel using Monte Carlo integration. [51] later analysed the quality of the approximation, and various works have explored (random deterministic) Fourier features, e.g., in the context of kernel methods (e.g., [24, 19, 48)). [53] investigated the use of Fourier features multilayer perceptrons (MLPs) to help represent high-frequency functions.
To the best of our knowledge, none of the previous works has shown that periodic activation functions in BNNs establish a direct correspondence between the prior on the network weights and the density of the spectral density decomposition, thus, inducing global stationarity and enabling principled prior choices for BNNs.
Encoding Inductive Biases into Infinitely-wide Networks
are concerned with the notions of similarity and bias in input space when performing supervised learning. For this, we consider modelling assumptions present in random (untrained) models that induce the priori (before observing data) characteristics the model. We argue that these assumptions, when combined with data, either reinforce suppress data biases depending the prior assumptions. For analysis, we consider the mean function, (4.)), and covariance (kernel) function, <(,)) -), induced by model functions f(x), representing the input-output mapping of single- (f()))
where x, x' E IRd are two inputs and the expectations are taken over model functions. probabilistic machine learning, rather than inferring the covariance structure from the expectations over model functions, one typically directly encodes assumptions by choosing a suitable parametric kernel family. However, both approaches, building model by choosing form for f choosing kernel family, equivalent.
layer: (xx) = Elf [x)] and <(x, x') =

---

Specification of prior knowledge through the covariance function and doing inference under this model, is at the core of Gaussian process (GP) models [44]. These models admit the form of a prior f(x) ~ (P()
and a likelihood (observation) model y | ~ T P(yi [(x))),
where the data D = {(x; vi) i==1 are input- output pairs and X, E Rd. This non-par non-parametric machine learning paradigm covers regression and classification tasks, under which GPs are known to offer a convenient machinery for learning and inference, while offering meaningful uncertainty estimates.
Neal [38] showed that an untrained single-layer network converges GP in the limit of infinite width. This link is well-understood and generalizes to deep models [7], [26]. By placing an i.i.d. Gaussian prior on the weights and biases of a network, the distribution on the output of an untrained network converges to Gaussian distribution in the limit of infinite width. By application of the multivariate Central Limit Theorem, the joint distribution of outputs for any collection of inputs is multivariate Gaussian, as in GP, and completely characterized by some kernel function k(:;) .). Let (-) be some non-linear (activation) function, such as the ReLU or sigmoid, and w and be the network weights and biases. Then, the associated kernel for the infinite-width network can be formulated terms of [38, 12]:
/ p(()) p(b) o(wTx +
<( x²)
+ +b) dw db,
where priors p(w) and p(b) are chosen suitably. From the formulation in Eq. (1) one can read that the covariance function corresponding to an untrained (random) single-layer neural network is fully characterized by the choice of activation function o-)) and the priors on the network weights and
biases.
Many of the covariance functions corresponding to commonly used activation functions under Gaussian priors on the weights have closed-form representations. The ArcCos kernel [3] covers the ReLU (AreCos-1) and the step (ArcCos-O) activations, the so-called 'neural network kernel' [59, 44] links to sigmoidal activations. Gram matrices K., = x(xr, dfj), evaluated for input pairs x and x' over x, x' E 1, 3]. are visualized in Fig. 2a to illustrate these covariance structures. The models induced by the ReLU, step, and sigmoid activations are by nature local, with the concentration of the non-linearity around the origin. As can be seen in the ID regression results in Fig. 2b, the strong inductive bias in the prior (model) is reflected in the posterior. The ReLU extrapolates by matching the local bias of the data, while the step and sigmoidal saturate outside the data, and priors are clearly sensitive to translations and highly non-stationary
Stationarity For stationary (homogeneous) covariance function the covariance structure of the model functions (x) is the same regardless of the absolute position in the input space, and thus the parametrization can be relaxed to x(x,x') & n(x - x') = <(r). Stationarity implies translation- invariance, i.e. the notion of similarity between inputs x, and x' is only a function of their distance x t-x x'. This implies weak stationarity under stochastic process theory [40], and results in reversion to the prior outside observed data (see Fig. 2b). For stationary covariance functions, the best-known family is the Matérn [32, 49] family of kernels, which features models with sample functions of varying degrees of differentiability (smoothness):
Maat. (x, x')
21
(V22
xx11 - L
|x
K, 22
x x'Il
(2)
I'2))
where v is a smoothness parameter, a characteristic length-scale parameter, (.) the modified Bessel function, and I(-) the gamma function. For the Matérn class, the processes are [v]] 1 times mean-square differentiable, and the family includes the RBF (squared exponential) and the exponential (OrnttennUUhlenbeete kernels as limiting cases as + oo and = 1/2.
Local stationarity Neural networks do not naturally exhibit stationarity. Formally, this stems from the problem of representing non- non-linnasisies over infinite input domain with a finite set of local non-linear mappings. However, to bridge the gap between neural networks and widely-used stationary kernels, the typical approach is to restrict the invariance to be local (see Fig. 2a). This is the approach in the RBF-NN [60, 44] and general Matérn-NN [34] activation functions, where the prior is a composite covariance function with a Gaussian decay envelope (see discussion in [15]), Le..
(3) where 2 = 20 + l (see Fig. 2a for the decay effect of the envelope). One motivation for this approach is to assume Gaussian input density the training/testing inputs, thus restricting our
MM (x, x') x exp(-x x/ x/20"n) KMa. (x, x') exp(-x' Tx 1220

---

M
W
WWW
W
-2
Sinusoidal activation
Sinc-Cosine activation
Triangle wave activation
Periodic ReLU activation
Figure 3: Sketches of types of periodic activations functions a--) visualized over two repeating periods each. The two leftmost infinitely differentiable and the two rightmost piecewise linear. The dashed line overlays the Sinusoidal activation for comparison.
interest only to inputs the vicinity of the training data. The local behaviour is highlighted in Fig. 2b, where the locally stationary models revert to the mean (as expected), but the marginal uncertainty of the model functions drops to zero when far enough from training samples. This motivates us to seek activation functions that induce (globally) stationary behaviour (bottom-row in Fig. 2).
Global Stationarity-inducin Activation Functions
We derive a direct connection between (fully) stationary covariance functions and periodic activation functions. Our main result leverages the spectral duality of the spectral density of stationary covariance functions to establish direct connection between the weight prior and the induced prior on the function space. Furthermore, basis and establishes an equivalence relationship between a Student-t distribution on the weights and prior the function space with Matérn covariance structure.
show that this connection is not restricted to the (sinusoidal) Fourier
3.1 Spectral Duality Under Harmonic Functions
For stationarity-inducinn priors, the covariance function be written equivalently in terms of its spectral density function. This stems from Bochner's theorem (see [6] and 36A in [28] for more general formulation) which states bounded continuous positive definite function x(r) can be represented through the following norm-presse corm-preserving isomorphism
<(r) = = ((T) p(dw),
(4)
where (s a positive measure and r E Rdd the measure (uw)) has density, it is called the spectral density S(w) corresponding the covariance function s(r). With w(r) (2x) (or))=exp(iod (i r). this gives rise to the Fourier duality of covariance and spectral density, which is known as the Wiener-Khindl Khinchin theorem (e.g., [44]):
<(r) (2 / ((w)eew'r de and S(w) - (<(T) -iw'r dr.
(5)
For d > 1, if the covariance function is isotropic (it only depends on the Euclidean norm ||||| such that (()r) A ([[[]]])) then it is invariant all rigid motions of the input space. Moreover, the spectral density will also only depend on the norm of the dual input variable w. In the neural network case, assume the previous layer will take care of scaling the inputs, and thus effectively interested in the isotropic case, which brings us to analysing ID projections.
An important question is whether we are restricted to the (sinusoidal) Fourier basis. Let (z)(2) be any sequence of bounded, qquare-integrabee (over its period) periodic functions, such that these functions define an inner product space (generalising a basis of vector space). In signal processing terms, they define a frame that provides a redundant yet stable way representing a signal. From a generalised harmonic perspective [58], we could resort to any convenient basis, such as Gabor wavelets [1 1]. However, in this work we seek choose (xx) suitably, such that we have uniform convergence to w(r) (cf., Eq. (4)), ensuring that we retain the spectral density S(w).
5

---

3.2 Types of Periodic Activation Functions
We show how various periodic activation functions (see Fig. 4) under the view of Eq. (1) directly link back to the spectral duality. These periodic activation functions are continuous, bounded, and centered zero, while not necessarily differentiable and never monotonic over R (but monotonic within half-periods
Sinusoidal activation By assuming the activation function to be a sinusoid, (si((z) = 22 (a) (see Fig. 4), placing uniform prior the bias term, p(b) UUiffrmm-n; x), and substituting this into Eq. (1), we can obtain an expression for the covariance function <(z,x"):
x(x, x') = p(()) [
sin (we + b) sin(wa' + b) db dw.
(6)
By solving the inner integral and applying Euler's formula (cos(z) = _ee²= = i=))), we obtain:
<(r, x') = ((w)eeetees dw + ʃ (owwe-lu(e-- dwc
which simplifies under the assumption that the prior on the weight is symmetric and has support on the real line, i.e.,
k(I, x') - f (u())il(eree dw.
(8)
By letting r x x', we find that we recover the spectral density decomposition of a stationary process given by the Wiener-Khiin Khinchin theorem in Eq. (5), where p(w) Σम obtain a direct connection between the prior on the weights and the spectral density. A detailed derivation can be found in App. A.I.
3 S(w). Therefore, we
Sine-Cosine activation An alternative way of using sinusoidal activation functions is the use of composite sine-cosine activation, i.e., Jsin cos (z) = sin(x) + cos(r) (Fig. 4 compares the sin-cos to the sinusoidal activation). Using such construction has the benefit of removing the need to integrate over the bias, which can result in reduced variance [51] of the estimates, giving us covariance function in the form of:
k(x,x')= = f p(w) [sin(wz) + cos(wwr)) (x:)] [sin(wx') + cos(wa')) dw = p(w) sin(w(( + x²)) dw + p(w) cos(w(z x²)) dw.
(9)
By application of Fuler's formula and under the usual assumption that p(w) is symmetric and has support on the entire real line, the above reduces to:
1 2/2 [ p(w) jimta (') dw + / p(w) e -io((===') dw = p(w) jur(a-a) dw. (10)
<(x, x') =
Again, by letting r = x x- x', we find that we recover the spectral density decomposition of stationary process given by the Wiener-Khined theorem. A detailed derivation can be found in App. A.2.
Triangle wave activation Inspired by the success of piecewise linear activation functions (e.g., ReL.U [36], leaky ReI.U [54], PReLU [17]), we show that the triangle wave, a periodic piecewise linear function, can be used instead of sinusoidal activations. The triangle wave given by:
4 p 2
2x
(x)
(11)
where is the period and denotes the floor function. By considering a period of 2m, assuming a uniform prior on the biases, Uniform( (-T1, x). and appropriate rescaling, we get the activation
1 1
Oriangle (x)
r π
(12)
2
for which we obtain direct correspondence of the network weight priors to the spectral density. As shown the derivations in App. A.3, we find that we again recover the spectral density decomposition theorem.
of stationary process given by the

---

Exponemtil
Matérn-
RBF
Rxaet Sinusoid
Exponential
Matém-
RBF
Exact
Triangle ware
Sine- Mine-Cosine
Periodic ReLU
Input, Jar 11
Input, x'1
Figure 4: Covariance functions calculated by MC integration with 5000 samples and compared to their exact dashed counterparts (peaks shifted for clarity). Regardless of the type of periodic activation function used, recover the behaviour of the stationary kernels (see also Fig. 8a).
Periodic ReLU activation To go even further, we define a piecewise linear periodic activation function with repeating rectified linear pattern, which we call the periodic ReLU function. It can be defined in terms of the sum of two triangle waves, with the second one being shifted by of period. The resulting periodic function is piecewise linear and defined as:
(x) 72 ( (1r
c + 5)
2.
(=+ + + -I
x
(1)
1 2
(13)
assuming period of 2T. It is again possible obtain correspondence between the network weight prior and the spectral density by considering weighted version of the equation above. In particular, show in App. A.4 that from the following weighted periodic ReLU activation function,
()RaLU(()= (x)= T (( T
(x + 5) 1
(2) +
x
(1))+++)))
π
π
(14)
we again recover the spectral density decomposition of stationary process given by the Wiener- Khinchin theorem, providing a direct connection between the prior on the weights and the spectral density. Note that, choosing a piecewise linear periodic activation function has potential computational benefits compared to sinusoidal activation functions and can help prevent vanishing gradients.
3.3 Kernel Functions
We have established that is it possible obtain direct correspondence of the prior on the weights and the spectral density of a stationary covariance function by using periodic activation functions in random neural networks. In App. B we show that by placing Student-t distribution on the weights with degrees of freedom of u = 2v we recover the spectral density of the Matérn family, i.e.
r( metll
w
[ + 1) [(v)
(2v)" (2v+ + +w? (1++)
1 SMar. (w),
p(w)
1
2 2T
V VuTI
2T
(15)
where p(w) denotes the probability density function of Student-t distribution and SMa. (w) denotes the spectral density of the Matérn family. This means that a Student-l prior the weights in combination with an appropriately scaled periodic activation function corresponds directly to a prior in the function space with Matérn covariance structure. Fig. 4 verifies this result and shows that we recover the exact covariance function (dashed) for various examples from the Matérn family by Monte Carlo (MC) integration (5000 samples) with all of the discussed periodic activation functions.
Recall that the Matérn family contains the exponential kernel (v 1/2) and the RBF Kernel (v oo) as special cases. Similarly, the Student-t distribution has the Cauchy (u = 1) and the Normal distribution (u = oo) as limiting cases, resulting in correspondence the Matérn family. Table 2 in App. B summarises the set of priors on the weights that correspond to the spectral density of kernel functions the Matérn family. This correspondence is exact for the sinusoidal and sine-cosine activations. For the triangle wave and periodic ReLU activations, the obtained correspondence approximate, and the introduced approximation error analyzed in detail in App. A.
the respective special cases in
7

---

Table 1: Examples of UCI regression tasks, showing the globally stationary NN model directly gives competitive RMSE and for most data sets also improved mean negative log predictive density (NLPD) compared non-stationary and locally stationary NN models. KFAC Laplace was used as the inference method. High std is due to variability in 10-fold-cv splits of the small data sets, not method performance.
BOSTON (306, 12) 50)
CONCRETE
AIRFOIL
ELEVATORS (16899, 18)
(HL,
(10500,5
(1503, 51 (1)
(1. 50)
500)
NLPD
RMSE
NLPD
RMSE
NLPD
RMSE
NLPD
RMSE
RoLU
0.5120032
0.3740.07
0.784:0.16
0.48400.4
0.5120.53
0.411:0.21
0.3840.03
0.354:0.01
RRF
0.5240.30
0.3740.08
0.7840022
0.44400.0
0.1040.15
0.264:0.03
0.4140.04
0.3540.001
glob (sin)
0.42:40.34
0.36:20.07
0.744:0.15
0.49+90.05
0.1420.17
0.29::0.05
0.38:20.08
0.35::0.01
glob RBF (prele)
0.39:200.0
0.30:10.07
0.744:0.14
0.491:0.04
0.08:10:12
0.26:10.03
0.74:0078
0.46:10.21
ooe Mat-M/2
0.71#:0.38
0.40==0.08
0.8440.28
0.42#0.04
0.11
0.26+:0.03
0.43+00.04
0.3540.01
Mat322 3/2
0.43:10.22
0.39:10.06
06
0.49:10.05
0.0710.11
0.274:0.03
0.37100.00
0.35:10.01
gloh Mar3/2 (pectu)
0.38:1.222
0.382:008
0.72:0.18
0.4810006
0.08:0.12
0.274:0.33
0.30:00003
0.361:0001
4 Experiments
To assess the performance of cuttionarit-indducinn activation functions, we compared the in-domain and out-of-domain predictive uncertainty to non-stationary and locally stationary models on various benchmark data sets. In all experiments, the compared models differ only by the activation function and the respective weight priors of the last hidden layer of the network. Replacing the prior effects the initialization of the NN weights and the training objective, detailed description of the experimental setup is available in App. D. The illustrative toy BNN examples are implemented using Turing.jl [14], GP regression results use GPflow [33], and all other experiments are implemented using Py PoTch [41].
maximize the log joint density.
Illustrative toy examples Fig. shows predictive densities for non-stationary. locally stationary, and globally stationary activation functions on the banana classification task. The top row illustrates the predictive densities of infinite-width BNNs (GP), the bottom row shows corresponding results for single hidden layer BNN using 30 hidden units. We observe that global stationarity-inducin activation functions revert to the prior outside the data, leading to conservative behaviour (high uncertainty) for out-of-domain samples. Moreover, we see that the mhite-width BNNs result in similar behaviour to their fhinite-width counterpart, while locally stationary activation functions in finite-widie BNNs exhibit a slower reversion to the mean than their anfiiitewwdtth corresponding GPs. Additionally, we include ID toy regression study that highlights the differences between different prior assumptions encoded by choice of the activation function. Fig. shows the corresponding prior covariance as well as posterior predictions for the infinite- width (GP) model. We replicated the experiment with finite-wiid width network and observed that the behaviour translates to finite width (see Fig. 8 in App. D.I).
UCI benchmarks Table 1 shows results on UCI [8] regression data sets comparing deep neural networks with ReLU, locally stationary RBF [60], and locally stationary Matérn-3/2 [34] against global stationary models. An extended results table can be found in App. D.2 and additional results
Mean confidence
Accuracy
0.6
0.3
NLPD
0.2 0.4
0.6
0"
90"
180"
270"
360
0"
90
180
270"
360
0"
90
180"
270"
360
Rotation angle
Rotation angle
Rotation angle
Figure 5: Rotated MNIST: The models have been trained on unrotated digits. The test-set digits are rotated at test time to show the sensitivity of the trained model to perturbations. All models perform equally in terms of accuracy, while ReLU (-) shows overconfidence in terms of mean confidence and NLPD. The stationary RBF models (- local, sin, sin-cos) capture uncertainty.

---

ReLU activation function
0.3
0.3
CIFAR-10 CIFAR-100
CIFAR-10
CIFAR- 100
0.2
SVHN
0.2
SVHN
0.1
0
0.5
1.5
2
0
0.01
0.02
Predictive entropy
Predictive marginal variance
Locally stationary Matérn-
3 activation function
0.3
0.3
0.2
0.1
0.1
0
0.5
1.5
2
0
0.01
0.02
Prodictive entropy
Predictive marginal variance
Periodic activation function (Matérn-
kernel)
0.3
0.3
0.2
0.1
0.1
0.5
1.5
0.01
0.02
Predictive entropy
Predictive marginal variance
7 48
9
8Cg
9
Most similar
CIFAR-10
Least simnllar
similar
CIFAR-100
Least similar
Most similar
SVHN
Least similar
Figure 6: OOD detection experiment results for models trained on CIFAR-10 and tested on CIFAR- 10. 100, and SVHN. Top rows show predictive entropy and predictive marginal variance histograms test images for in-domain and OOD data sets. Top to bottom: ReLU activation function, local stationary Matérn-32 activation function [34], and sinusoidal activation function that induces a globally stationary Matérn-3/ model. The last row shows sample images from each test set: left-side images with the lowest entropy/highest confidence and right-side images with the highest entropy/lowest confidence. See App. D.5 for histograms for more models.
on UCI classification tasks in App. D.3. Table lists root mean square error (RMSE) and negative log predictive density (NLPD), which captures the predictive uncertainty, while the RMSE only accounts for the mean. Global stationary models provide better estimates of the target distribution all cases while obtaining comparable RMSEs. It is important to note that the large standard deviations in Table I are due to the small number of data points and the fact that some splits in the 10-fold CV end up being harder than others. The posterior inference is performed using KFAC Laplace [45].
Detection of distribution shift with rotated MNIST Fig. 5 demonstrates the predictive behaviour of different activation functions under covariate shift. We evaluate the predictive accuracy, mean confidence of the predicted class, and NLPD on MNIST [25] for different rotations of the input. Models are trained unrotated digits using KFAC Laplace [45]. The results indicate that all models obtain similar accuracy results, while only local and global stationary models do not result in over- confident uncertainty estimates. For an ideally calibrated model, the mean confidence would decrease as low as the accuracy curve when the digits are rotated, keeping the NLPD values as low as possible.
Out-of-distributie detection using CIFAR-10, CIFAR-100, and SVHN Fig. 6 shows model performance on OOD detection in image classification for ReLU, locally stationary Matérn-3/2, and globally stationary Matérn-3/2 models. The models have been trained using SWAG [29] on CIFAR-10 [23], and tested on CIFAR-10. CIFAR-100 [23], and SVHN [39] test set images. Both CIFAR- 100 (more similar) and SVHN (more dissimilar to CIFAR- 10) images are OOD data, and the

---

models should show high uncertainties for the respective test images. We use two metrics to measure model uncertainty: predictive entropy and predictive marginal variance. Predictive entropy captures uncertainty present in the predicted mean class weights, while the predictive marginal variance captures uncertainty related to the variability in the predicted class weights. From these two metrics, the predictive marginal variance could be more suitable as an OOD detection metric, as predictive entropy can justifiably give high values for some in-distribution samples, for example, those that the model confidently predicts being in between two known classes.
The histograms of predictive entropies for different test sets show that all models can separate between in- on-distribution and OOD data to some extent, and all models consider SVHN more OOD than CIFAR- 100, which intuitive as CIFAR- 1-00 resembles CIFAR- 10 more. Also, the predictive marginal variance histograms show that the models have higher variance on OOD data compared to in-distribution data. However, the ReLU model considers CIFAR- more OOD than SVHN and the locally stationary Matérn-3/2 model has overall relatively small predictive marginal variance values for all data sets. We consider predictive marginal variance better metric for OOD detection than predictive entropy, as in-distribution samples that are hard to classify are expected to have high predictive entropy, not necessarily allowing the detection of OOD samples on this metric. See Fig. 14 in App. D.5 for results on different models, and Table 8 in App. D.5 for numerical results showing the area under the receiver operating characteristic curve and the area under the precision recall curve for the OOD detection experiment.
Discussion and Conclusions
We have shown that periodic activation functions in Bayesian neural networks (BNNs) establish direct connection between the prior on the network weights and the spectral density of the limiting stationary process. This link goes beyond the Fourier (sinusoidal) basis, which we have illustrated by deriving the correspondence also for the triangle wave and a new periodic variant of the ReLU activation function. Moreover, we have shown that for BNNs with a periodic activation function, placing a Student-t distribution on the network weights corresponds to a prior in the function space with Matérn covariance structure. This correspondence is exact for the sinusoidal and sin-cos activations and approximate for the triangle and periodic ReLU activations.
Our work can help build neural network models that dacu-occurring either naturally or maliciously. As demonstrated in the experiments, the modelling assumptions have practical importance in reducing sensitivity to input shift. Perturbation of inputs can be captured in the last layer a deep model, and the principled choice of a priori stationarity has shown to be effective based the experiments. Using periodic activation functions in BNNs induces global stationarity. i.e., the model not only locally behaviour across the input domain. The resulting model reverts the prior for -istt--diittibuttion (OOD) samples, resulting in well-specified uncertainties.
robust and less vulnerable to OOD
which induces conservative
In contrast to models with non-stationary or locally stationary activations, optimising and performing approximate inference models with periodic activation functions has proven to be more challenging. In particular, found that the uncertainty over the bias terms is often not accurately estimated when using KFAC Laplace or SWAG, which required us to use a relatively large number of hidden units. However, the experiments using dynamic HMC indicate that the number of hidden units can be drastically reduced using a more accurate approximate inference scheme, as seen in Fig. 1 and Fig. 11 in App. D.1.
The codes and data to replicate the results are available under MIT license at fttaps://githann com//aaltoMl/Periodee
Acknowledgments and Disclosure of Funding
We acknowledge the computational resources provided by the Aalto Science-IT project. This research was supported by the Academy of Finland grants 324345 and 339730, and Saab Finland Oy. We thank William J. Wilkinson and the anonymous reviewers for feedback on the manuscript.
10

---

References
[1] L. Aitchison. A statistical theory of cold posteriors in deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
E. J. Candès. Harmonic analysis of neural networks. Applied and Computational Harmonic Analysis, 6(2): 197-218, 1999.
Y. Cho and L. K. Saul. Kemel methods for deep learning. In Proceedings of Advances Neural Information Processing Systems (NIPS), pages 342-350. Curran Associates, Inc., 2009.
[4] C. Cortes and Vapnik. Support-vector networks. Machine Learning, 20(3):273-295 297, 1995.
[5] N. A. Cressie. Statistics for Spatiaaa Data. Wiley series in probability and mathematical statisties. Wiley- Interscience, 1991.
[6] G. Da Prato Zabczyk. Stochastic Equations in Infinite Dimensions, volume 45 Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1992.
[7] A. de Matthews, Hron, M. Rowland, Turner, and Ghahramani, Gaussian process behaviour in wide deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[8] D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://atcchivee iics. www.uciiuuuuuu edu/ml.
[9] A. Y. Foong, D. R. Burt, Y. Li, and R. E. Turner. On the expressiveness of approximate inference in Bayesian neural networks. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), pages 15897-15908. Curran Associates, Inc., 2020.
[10] V. Fortuin, A. Carriga-Alonso, F. Wenzel, G. Rätsch, R. E. Turner, M. van der Wilk, and L. Aitchison. Bayesian neural network priors revisited. In Third Symposium on Advances in Approximate Bayesion Inference, 2021.
[11] D. Gabor. Theory of communication part 1: The analysis of information. Journal of the Institution of Electrical Engineers- Part Radio and Communication Engineering. 93(26):429 441, 1946.
[12] Y. Gal and Z. Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning (ICML), volume of Proceedings of Machine Learning Research, pages 1050-1059. PMLR, 2016.
[13] A. R. Gallant and H. White. There exists a neural network that does not make avoidable mistakes. In Proceedings of International Conference on Neural Networks (ICNN), pages 657-664, 1988.
[14] H. Ge, K. Xu, and Z. Ghahramani. Turing: A language for flexible probabilistic inference. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 84 of Proceedings of Machine Learning Research, pages
PMLR, 2018.
[15] M. G. Genton. Classes of kernels for machine learning: A statistics perspective. Journal of Machine Learning Rexearch, 2:299-312, 2001.
[16] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
[17] K. Hc, X. Zhang, S. Ren, and Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[18] M. Hein. M. Andriushchenko, and Bitterwolf. Why ReLU networks yield high-confidence predictions away from training data how to mitigate the problem. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 41-50, 2019.
[19] Hensman, N. Durrande, and Solin. Variational Fourier features Gaussian processes. Journal of Machine Learning Research, 18:1-52, 2018.
[20] T. Hofmann, B. Schölkopf, and A. Smola. Kernel methods in machine learning. The Annals of Statistics, pages 1171-1220, 2008.
[21] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernl: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31, pages 8571 15800 Curran Associates, Inc., 2018.

---

[22] A. Kristiadi, M. Hein, and P. Hennig. Being bayesian, even just bit, fixes overconfidence in ReLU networks. In Proceedings of the International Conference Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 5436-5446. PMLR, 2020.
[23] A. Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 2012.
[24] M. Lázaro-Gree Greailla, J. Suufionero-Candele C. E. Rasmussen, and A. R. Figueiras-Vidde Sparse spectrum Gaussian process regression. Journal of Machine Learning Research, 1 11:1865-188) 1881, 2010.
[25] Y. LeCun, C. Cortes, and C. J. Burges. The MNIST database of handwritten digits, 1998. URL http: //yann. lecun. com/ wwt.cotc//c//eceecttttccoco.o exdb/mnist/.
[26] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and Sohl-Dickstein. Deep neural networks as Gaussian processes. International Conference on Learning Representations (ICIR), 2018.
[27] Lee- Sho-Th J. Ainslic, I. Eckstein, and S. Ontañón. Fnet: Mixing tokens with Fourier transforms. ArXiv preprint arXiv: 100..5508, 2021.
[28] L. Loomis. An Introduction to Abstract Harmonic Analysis. D. van Nostrand Company, Inc., New York, 1954.
[29] W. J. Maddox, P. Izmailov. T. Garipov, D. P. Vetrov, and A. G. Wilson. simple baseline for Bayesian uncertainty deep learning. In Advances in Neural Information Prucessing Systems 32 (NewrIPS), pages 13132-13149 143. Curran Associates, Inc., 2019.
[30] H. Maennel. Uncertainty estimates and duitt--distribution detection with Sine Networks. In (CML.2019 Workshop on Uncertainty and Robustness in Deep Learning, 2019.
[31] G. Marcus. The next decade in AI: Four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
[32] B. Matérn. Spatial variation: Stochastic models and their applications to some problems forest surveys and other sampling investigations. Meddelanden från statens skogsforskninggsisstitt, 49:1- 14:4,14,, 1960.
[33] A. G. d. G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P. León- Velll-ill,gáá, Z. Ghahra- mani, and J. Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Rexearch, 18(40):1-6, 2017.
[34] L. Meronen, C. Irwanto, and A. Solin, Stationary activations for uncertainty calibration in deep learning. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), pages 2338-2350. Curran Associates, Inc., 2020.
[35] P. Morales-Alvarez, D. Hermández-I R. Molina, and J. M. Hernánder- obato. Activation-Ievel uncer- tainty in deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[36] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. Proceedings of the International Conference on Machine Learning (ICML), pages 807-814. Omnipress, 2010.
[37] E. T. Nalisnick. Gordon, and J. M. Bernández-Lobate Predictive complexity priors. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130 of Proceedings of Machine Learning Research, pages 694 694-702. PMLR, 2021.
[38] R. M. Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, Toronto, Canada, 1995.
[39] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2. URL http://ufidl. stanford. adu/hhossenubeer.
[40] A. Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw-Hill New York, 1991.
[41] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. DeVito, M. Raison, A. Tejani, Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, perfooerformance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 8024 8035. Curran Associates, Inc., 2019.
12

---

[42] T. Pearce, R. Tsuchida, M. Zaki, A. Brintrup. and A. Neely, Expressive priors in Bayesian neural networks: Kernel combinations and periodic functions. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), volume 115 of Proceedings of Machine Learning Research, pages 134-144. PMLR. 2019.
[43] A. Rahimi and B. Recht. Random features for large-scale kemel machines. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), pages 1177-1184. Curran Associates, Inc., 2008.
[44] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge. MA, 2006.
[45] H. Ritter, A. Botev, and D. Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018.
[46] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Nenworks, 61:85-117, 2015.
[47] V. Sitzmann, J. Martel, A. W. Bergman, D. B. Lindell, and Wetzstein. Implicit neural representations with periodic activation functions. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 7462-7473. Curran Associates, Inc., 2020.
[48] A. Solin and Särkkä. Hilbert space methods for reduced-rank Gaussian process regression. Statistics and Computing, 30(2)419 446, 2020.
[49] M. L. Stein. Interpolation of Spatial Data. Springer Series in Statistics. Springer, New York, 1999.
[50] S. Sun, G. Zhang. J. Shi, and R. B. Grosse. Functional variational Bayesian neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
[51] D. J. Sutherland and J. G. Schneider. On the error of random Fourier features. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pages 862-871. AUAI Press, 2015.
[52] C. Szegedy, W. Liu, Y. Jia, P. Sermanct, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9,20 2015.
[53] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi,
J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 7537-7547. Curran Associates, Inc., 2020.
[54] R. Tsuchida, F. Roosta, and M. Gallagher. Invariance of weight distributions in rectified MLPs. In Proceedings of the International Conference Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages
PMLR, 2018
[55] D. Ulmer and G. Cinà. Know your limits: Uncertainty estimation with ReLU classifiers fails at reliable OOD detection. In Proceedings of the Thirty-Sevemth Conference on Uncertainty in Artificial Intelligence, volume 161 Proceedings of Machine Learning Research, pages 1766-1776. PMLR, 2021.
[56] M. Uteuliyeva, A. Zhumekenov, R. Takhanow, Z. Assylbekov, A. J. Castro, and O. Kabdolov. Fourier neural networks: A comparative study. Intelligent Data Analysis, 24(5):1107-111 2020.
[57] F. Wenzel, K. Roth, S. Veeling, J. Swiętkowski, L. Tran, S. Mandt, J. Snoek. T. Salimans, R. Jenatton. and S. Nowozin. How good is Bayes posterior in deep neural networks really? In Proceedings of the International Conference on Muchine Learning (ICML), volume 119 of Proceedings of Muchine Learning Research, pages (0248-10255 PMLR, 2020.
[58] N. Wiener. Generalized harmonic analysis. Acta Mathematica, 55:1177258, 1930.
[59] C. K. I. Williams. Computing with infinite networks. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 295-222333 301. MIT Press, 1997.
[60] C. K. I. Williams. Computation with infinite neural networks. Neural Computation, 1005):1203-1216 1998.
[61] K. Wong, C. Leung, and S. Chang. Handwritten digit recognition using multi-layer feedforward neural networks with periodic and monotonic activation functions. In Proceedings of International Conference on Pattern Recognition (ICNN), pages 106-109, 2002.
[62] L-C. Yeh. Modeling of strength of high-hhihhonreeee performance concrete using artificial neural networks. Cement and Concrete Research, 28:1797-1808. 1998.
13


Reviews:
[{"review_id": "sKt0jtXTHTu", "reviewer": "Reviewer_C4fJ", "summary": "The authors attempt to draw a connection between \n(1) the spectrum $S(\\omega)$ of stationary kernels, e..g K(x, x') = k(x-x') and \n(2) the prior $p(\\omega)$ of the weights in random initialized networks of the form $\\psi(wx +b)$, where the activation function $\\psi$ is periodic\nby using the well-known Bochner’s theorem. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I summarize (simplify) the arguments of the paper and points out a couple mistakes of the paper. The main result of the paper is trying to say that the feature (or NNGP) kernel of a randomly initialized one-hidden layer network with weight prior $p(w)$ can be approximated written as (up to some scalars, positive/negative signs)\n\\begin{align}\n K(x, x') \\approx  C_{\\psi} \\int p(w) e^{i\\omega^T (x -x')} d\\omega \n\\end{align}  \nThis approximation is not very correct in general. Even it is, the formula itself is not significant. \n\nHere is the main argument to \"get\" the above formula. \n\\begin{align}\nK(x, x') = \\int _\\omega p(w) \\int_b \\psi(wx+b) \\psi(wx'+b) p(b) db dw\n= \\int _\\omega p(w)  \\psi \\ast \\bar \\psi (\\omega (x-x'))  dw \n\\end{align}\nwhere the following equation used the fact that $b$ is uniformly supported on $[-\\pi, \\pi]$ and $\\psi$ is a periodic function on that and thus the inner integral is a convolution on the torus \n\\begin{align}\n\\int_b \\psi(wx+b) \\psi(wx'+b) db  = \\psi \\ast \\bar \\psi (\\omega x-  \\omega x'))\n\\end{align}\nHere $\\bar \\psi( x) = \\psi(-x) $.  We can then apply Fourier expansion to $\\psi \\ast \\bar \\psi $ since it is periodic and write\n\\begin{align}\n\\psi \\ast \\bar \\psi (z) =  \\sum  c_k  \\bar c_k e^{ik\\pi z } \n\\end{align}\nPlugging back the original equation, one has \n\\begin{align}\n K(x, x') =  \\int p(\\omega) \\sum c_k \\bar c_k e^{ik\\pi \\omega(x-x') }  d\\omega \n\\end{align}  \nThis clearly implies $K$ is stationary. Here is what goes wrong in the paper. The authors made the claim that \n\\begin{align}\n K(x, x') =  \\int p(\\omega) \\sum c_k \\bar c_k e^{ik\\pi \\omega(x-x') }  d\\omega  \\sim   \\int p(\\omega) c_1 \\bar c_1 e^{i\\pi \\omega(x-x') }  d\\omega \n\\end{align}  \nThat is keeping $k=1$ frequency and getting rid of all others! This is OK for $\\cos/\\sin$ combination seems $c_k=0$ for $k\\neq1$. For others,  the case $k=0$ is ok but need to assume $\\psi$ has mean zero (I haven't seen this assumption in the paper while all examples in the papers are odd functions.) but NOT OK for other small $k$ (e.g. k=2, 3 etc.) In equation (40) of the appendix, the authors conclude $c_k$ can be dropped due to rapid decay, which is clearly WRONG! Even though the tail terms could be dropped, but the lower frequency terms shouldn't. \n\nEmpirically, applying periodic activation functions to OOD problem seems interesting, but this is idea seems not original; see e.g. \"Stationary Activations for Uncertainty Calibration in Deep Learning\". \n\nOverall, the originality, quality and significancy of the paper is not very high and I don't recommend acceptance of it. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work hasn't been brought up in the paper; \nOne side note: \nThere has been significant advances in infinite-width networks that dramatically generalize Neal's work, which hasn't been discussed / cited in the paper. See e.g. \"Deep Neural Networks as Gaussian Processes\", \"Gaussian Process Behaviour in Wide Deep Neural Networks\" and reference therein.  \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "b9IuYcV0xdF", "reviewer": "Reviewer_6Fuy", "summary": "The author shows that periodic activation functions in a single layer Bayesian neural networks induce global stationarity in the limiting Gaussian Process. They show a direct link between the weight distribution (before training) and the spectral density of the covariance limiting GP, for various periodic activation functions. They leverage this observation to build models that are more robust to out-of-domain data.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper: I found it clearly written and with appropriate background for a non-expert audience. This work is timely and, I think relevant to the sub-community interested in coordinated-based networks/\"implicit representations\" as those bloom in many different applications (NerF, occupancy networks, SDF shape representation, GANs etc.).\n\nThe author demonstrates an equality between the prior weight distribution in a single layer neural network and the spectral density of the stationary covariance function of the limiting Gaussian Process, for four types of periodic activation functions. Thanks to this equality, the author shows that taking a Student-t distribution over the weights induces a Matern covariance function for the limiting GP. They also demonstrate that BNN with stationary limiting GP are less vulnerable to out-of-domain data.\n\nDrawing an explicit parallel between the weight distribution in a neural network and the spectral density of the covariance function of its limiting GP is a strong theoretical result. As the author shows, one can deduce a weight distribution from the stationary covariance kernel they want to induce, for any fully-connected neural network having periodic activation functions.\n\nThe author do not use the proven parallel between Student-t distribution and Matern covariance in their experiments. However, they experimentally show the advantage of stationarity in Bayesian neural network by proving robustness to our-of-domain data for different regression and classification task.\n\nThe main theoretical result in this paper is a significant step towards a better understanding of the link between neural architecture, weight distribution and inductive bias. The scope of the experiments described in this paper is limited to proving the advantage of a stationary limiting GP.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "XoHjHItqoUD", "reviewer": "Reviewer_qVhs", "summary": "The paper derives a class of Gaussian processes, derived as infinite-width limits of neural networks, that exhibit stationary covariance functions. The key observation comes from a comparison between the expression of a stationary kernel given by Bochner's theorem, and the infinite-width kernel of a neural network. Stationary covariance functions can be guaranteed whenever the activation function is chosen to be periodic. Numerical experiments suggest that these GPs enjoy improved performance on regression tasks, and more accurate uncertainty prediction on out-of-distribution inputs. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I very much enjoyed reading this paper. The authors present a simple thought-provoking observation with sufficient mathematical justification and an adequate selection of experiments to highlight the advantages. The paper is well-written, with excellent visual illustrations throughout to demonstrate the authors' claims. I found the literature review fairly comprehensive. Overall, I believe the authors have ticked all the boxes for a great conference paper. \n\nA few comments:\n\n- I would suggest being more careful about claiming improved NLPD in Table 1. The error margins due to the CV are large, and the 'improvements' shown in mean don't seem to be sufficient to suggest a significant difference. \n\n- It's unfortunate that Figure 6 does not contain a comparison to a locally stationary model; this comparison being relegated to the supplementary material. I understand this was done for space constraints, but I feel a better compromise could be made. Maybe by restricting to predictive marginal variance in the main text, and relegating entropy to supplementary material? \n\n- 'Banana' (dataset) is in typewriter format earlier in the paper, but not in the experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One common issue with Bayesian neural networks lies in its occasional overconfidence. This paper aims to address that issue with a simple trick. There are always some risks associated with uncertainty prediction, but this comes with the territory. The authors have pointed out a few limitations of their methodology in the conclusions; I believe this suffices. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "JfQ3dpmVlJq", "reviewer": "Reviewer_8dcJ", "summary": "The authors show that under the infinitely wide Bayesian neural network framework, periodic activation functions combined with student-t distributed readout parameters result in a GP with the Matern class of covariance functions. Periodic activations are considered in a relatively wide sense, including not only cosine and sine but also triangle and deep-learning inspired \"periodic ReLU\". The Matern class, being a stationary covariance function with intuitive and controllable smoothness properties, sees application in a diverse range of tasks. The authors compare predictive uncertainties and performance of their new infinite-width and finite-width models against corresponding locally stationary and non-stationary models on some toy and standard benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\na. The paper addresses an important gap in research. More specifically, it is known (Meronen et al.) that a certain activation function induces a locally stationary covariance function corresponding with an appropriately decayed Matern kernel. However, it is not known how to construct the globally stationary Matern class. This paper addresses this problem. \nb. The paper is very well written and easy to read. I did not spot any typos or awkward sentences. The paper uses elegant tools that are accessible to most readers in the community.\nc. The experiments and illustrations are insightful.\n\nLimitations/suggestions for improvement:\n1. I am not convinced that stationarity of the infinite-width model transfers to stationarity of the finite-width model when the width is small. More importantly, it is not necessarily true that the mean-reverting property holds for finite-width models. Ideally the authors would provide some quantification of error of this fact in the finite-width setting.\n2. The authors do not mention the related infinite-width neural tangent kernel (NTK) models. While not Bayesian, these are also GPs in the infinite width and do offer what some would call an uncertainty. The form of the covariance function is very similar to the Bayesian setting. Do your stationarity results carry over to this setting? If not, what breaks or presents a technical difficulty? At the least, I think the authors should mention this work and describe how it relates to their results.\n3. There are some minor issues concerning the rigour of the proofs in A.3 and A.4. I believe the result is correct, but the argument can be tightened. In (32), the authors approximate the wave by a truncated series. In (35) the put this truncation inside the kernel. This is the truncated series, not the full infinite series. There should be a limit inside the integral which needs to be interchanged with the integral via dominated convergence argument or similar.  There is another issue around (45) where are limit should be present and then the \\approx may be removed. The same criticism applies to A.4.\n\nDisclaimer:\nI did not review the code provided in the supplementary material. I did not read appendix D.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that periodic activations in a Bayesian neural network induce global stationarity in the Gaussian Process induced by an infinite-width limit. Stationary covariance functions in GP can be guaranteed whenever the activation function is chosen to be periodic (beyond sinusoidal). In a concrete example,  periodic activation functions combined with student-t distributed readout parameters result in a Matern class of covariance functions. The paper goes on to show experiments demonstrating that these stationary models are good for capturing uncertainty for out-of-domain data. 

The reviewers pointed out the fact that periodic activation functions inducing stationary kernels  may not be mathematically surprising. However, some reviewers also argued that, methodologically, this paper indeed describes a simple solution to an important and challenging problem of ensuring good uncertainty prediction for neural networks.

There were initial concerns regarding the approximation error of ignoring higher frequency effects. However, in the discussion period, authors have clarified the issue 1) providing alternative perspective 2) adding error estimates induced by approximation. Moreover the authors point out that numerical verification in the paper also indicates in reality error is indeed small. In the end, all reviewers agree that the paper should be accepted. 

Overall, the reviewers agree that the paper is clearly written with well performed experiments and comprehensive literature review.  The paper would be interesting to various NeurIPS sub-communities e.g. coordinated-based networks (implicit representations), Bayesian deep learning, infinite-width theory.
Decision: Accept (Poster)