Periodic Activation Functions Induce Stationarity
Authors: Lassi Meronen
Martin Trapp
Arno Solin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian deep learning
Gaussian process
uncertainty quantification

Abstract: Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.

Periodic Activation Functions Induce Stationarity
Lassi Meronen
Martin Trapp Aalto University Espoo, Finland martin. trappraaltppaaa.fff
Arno Solin Aalto University Espoo, Finland arno. www.solll.aa..o.oooooom..
Aalto University / Saab Finland Oy Espoo, Finland
lassi aaasi..eroneenaalto...
Abstract
Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that 'know what they do not know' by introducing inductive biases the function space. We show that periodic activation functions in Bayesian neural networks establish connection between the prior on the network weights and iossllaton-invariinn stationary Gaussian process priors. Furthermore, show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In series of experiments, functions obtain comparable performance for in-domain data and capture sensitivity perturbed inputs in deep neural networks for out-of-domain detection.
show that periodic activation
Introduction
Deep feedforward neural networks [46, 16] are an integral part of contemporary artificial intelligence and machine learning systems for visual and auditory perception. medicine, and general data analysis and decision making. However, when these methods have been adopted into real-world use, concerns related robustness (with respect to data that has been seen during training), fairness (hidden biases in data being reinforced by the model), and interpretability (why the model acts as it does) have taken a central role. The knowledge gathered by contemporary neural networks has even been characterised as never truly reliable [31]. These issues relate to the sensitivity of the trained model to perturbed inputs being fed through or the lack thereof.
This motivates Bayesian deep learning, where the interests are two-fold: encoding prior knowledge into models and performing probabilistic inference under the specified model. We focus on the former. Probabilistic approaches to specifying assumptions about the function space of deep neural networks have gained increasing attention the machine learning community, comprising, among others, work analysing their posterior distribution [57, 1], discussing pathologies arising in uncertainty quantification [9], and calls for better Bayesian priors (e.g., [42, 50, 37, 10]).
In this paper, we focus on stationary models, which act as a proxy for capturing sensitivity. Stationarity indicates mranllaton-invariande meaning that the joint probability distribution does not change when the inputs are shifted. This seemingly naive assumption has strong consequences in the sense that it induces conservative behaviour across the input domain, both in-distribution and outside the observed data. The resulting model is mean-reverting outside the training data (reversion to the prior), directly leading to higher uncertainty for out-of-distribution (OOD) samples (see Fig. I for examples). These features (together with some direct computational benefits) have made stationary models/priors the standard approach kernel methods [4, 20]. spatial statistics [5], and Gaussian process (GP) models [44], where the kernel is often chosen to induce stationarity.
Neural networks parametric models, which typically fall into the class of non-stationary models. Non-stationarity increases flexibility and is often a sought-after sopperty-especinn if the interest is solely in optimizing for accuracy in-domain test data. In fact, all standard neural network
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Non-stationarity
Local statiouarity
Global/full stationarity (this puper)
Finite-widi Infmitt-width
ArcCos-1 kernel /
Matém- J-NN
RBF-NN
Exponential (Cauchy prior)
Marérn-B & (t-distribution prior)
RBF (Gaussian prior)
ReLU activation [36] kerneVactiivtionn [34]
kemellactivation [59]
Figure 1: Posterior predictive densities of single hidden layer Bayesian neural networks (BNNs) with 30 hidden units and their infinite-width corresponding GPs on the banana classification task. Different activation functions induce different prior assumptions. Estimates obtained through HMC sampling [14] for 10k iterations.
activation functions (ReLU, sigmoid, etc.) induce non-stationarity (see [3, 59]). However, non- stationary models can easily result in over-confidence outside the observed data [18], [55] or spurious relationships between input variables (as illustrated in Fig. where darker shades show higher confidence).
Stationarity in neural networks has been tackled before under the restriction of local stationarity, i..,
only local, induced by a Gaussian envelope (as realized by [59] for the RBF kernel/activation). Recently. Meronen et al. [34] expanded this approach and derived activation functions corresponding to the widely used Matérn class of kernels [32, 44], We go one step further and derive activation functions that induce globul stationarity. To do so, we leverage theory from harmonic analysis [58] periodic functions, which helps expand the effective support over the entire input domain. We also realize direct links previous works leveraging harmonic functions in neural networks [13, 2, 61, 47, 56], and Fourier features in other model families [43, 51].
The contributions of this paper are: (i) We show that periodic activation functions establish direct correspondence between the prior on the network weights and the spectral density of the covariance function of the limiting stationary Gaussian process (GP) of single hidden layer Bayesian neural networks (BNNs). (ii) We leverage this correspondence and show that placing Student-t prior on the weights of the hidden layer corresponds to prior on the function space with Matérn covariance. (iii) Finally, we show a range of experiments that periodic activation functions obtain comparable performance for in-domain data, do not result in overconfident predictions, and enable robust out-of- domain detection.
Related Work
We build upon prior work on exploring the covariance function induced by different activation functions, starting with the seminal work by Williams [59], who discussed sigmoidal (ERF) and Gaussian (RBF) activation function, resulting in locally stationary kernel modulated by Gaussian envelope. Cho and Saul [3] introduced non-stationary GP kernels corresponding to the ReLU and step activations, and [54] later extended the approach to the leaky ReI.U and analysed alternative weight prior specifications. More recently, [34] derived activation functions corresponding to kernels from the Matérn family, which are locally stationary modulated by Gaussian envelope. In addition to the work connecting neural networks to GPs at initialisation, [21] created a connection between the neural network training and kernel methods by introducing the Neural Tangent Kernel (NTK).
At the same time, uninformative priors have been widely criticised in Bayesian deep learning [10]. and alternative strategies have been proposed to incorporate prior knowledge. Pearce et al. [42] proposed compositions of BNNs to mimic compositional kernels their limiting case. Sun et al. [50] proposed a variational objective for BNNs acting on the function space rather than the network parameters. Nalisnick al. [37] proposed tractable priors on the functional complexity, obtained through the change of variables formula applied to the KL-divergence from reference model/prior.

---

Highly nue statonury
ArcCas-1 ReL.U
ArcCss-O ~ Step
NN n Signoodd
ArcCos- ~ RaLU
ArdCze-l Step
NX Sigmoid
Locally stationry
XBF-NN
Maténa-3/a NNNN
Esp-NN (local)
RBF-NN (loca))
Mat5m-3/2-NN (local)
Exp-NN
Statonary
RBF
Maiém-3/ 3/2
Exponeniall
RDF
Masérn-3/2
Exponential
(a) Gram matrices
(b) Regression results with corresponding prior
Figure 2: Left: Gram/covariance matrices (for k(x, x')) reflecting different prior beliefs on how infinite-widde NN model functions ought to behave outside the observed data. Right: ID regression results corresponding to the model induced by the priors, showing the posterior and predictive 95% intervals. See Fig. appendix for the corresponding finite-width results.
Furthermore, Morales-Alvaree et al. [35] proposed to encode prior assumptions on the activation level rather than the model parameters.
Recent works have shown the potential of periodic activation functions neural networks, including approaches learn implicit neural representations [47], methods for recognition of handwritten digits [61]. and out-of ontoo--distribution detection [30]. Earlier work used periodic activation functions to resemble the Fourier series [13, 56] and showed that such models universal function approximators [2]. In broader terms, [27] recently showed that the use of the Fourier transform, replacing self- attention, in Transformers can drastically improve the runtime. Inspired by the Fourier duality of covariance and spectral density, [43] proposed random Fourier features to approximate GP with an RBF kernel using Monte Carlo integration. [51] later analysed the quality of the approximation, and various works have explored (random deterministic) Fourier features, e.g., in the context of kernel methods (e.g., [24, 19, 48)). [53] investigated the use of Fourier features multilayer perceptrons (MLPs) to help represent high-frequency functions.
To the best of our knowledge, none of the previous works has shown that periodic activation functions in BNNs establish a direct correspondence between the prior on the network weights and the density of the spectral density decomposition, thus, inducing global stationarity and enabling principled prior choices for BNNs.
Encoding Inductive Biases into Infinitely-wide Networks
are concerned with the notions of similarity and bias in input space when performing supervised learning. For this, we consider modelling assumptions present in random (untrained) models that induce the priori (before observing data) characteristics the model. We argue that these assumptions, when combined with data, either reinforce suppress data biases depending the prior assumptions. For analysis, we consider the mean function, (4.)), and covariance (kernel) function, <(,)) -), induced by model functions f(x), representing the input-output mapping of single- (f()))
where x, x' E IRd are two inputs and the expectations are taken over model functions. probabilistic machine learning, rather than inferring the covariance structure from the expectations over model functions, one typically directly encodes assumptions by choosing a suitable parametric kernel family. However, both approaches, building model by choosing form for f choosing kernel family, equivalent.
layer: (xx) = Elf [x)] and <(x, x') =

---

Specification of prior knowledge through the covariance function and doing inference under this model, is at the core of Gaussian process (GP) models [44]. These models admit the form of a prior f(x) ~ (P()
and a likelihood (observation) model y | ~ T P(yi [(x))),
where the data D = {(x; vi) i==1 are input- output pairs and X, E Rdd This non-par non-parametric machine learning paradigm covers regression and classification tasks, under which GPs are known to offer a convenient machinery for learning and inference, while offering meaningful uncertainty estimates.
Neal [38] showed that an untrained single-layer network converges GP in the limit of infinite width. This link is well-understood and generalizes to deep models [7], [26]. By placing an i.i.d. Gaussian prior on the weights and biases of a network, the distribution on the output of an untrained network converges to Gaussian distribution in the limit of infinite width. By application of the multivariate Central Limit Theorem, the joint distribution of outputs for any collection of inputs is multivariate Gaussian, as in GP, and completely characterized by some kernel function k(:;) .). Let (-) be some non-linear (activation) function, such as the ReLU or sigmoid, and w and be the network weights and biases. Then, the associated kernel for the infinite-width network can be formulated terms of [38, 12]:
/ p(()) p(b) o(wTx +
<( x²)
+ +b) dw db,
where priors p(w) and p(b) are chosen suitably. From the formulation in Eq. (1) one can read that the covariance function corresponding to an untrained (random) single-layer neural network is fully characterized by the choice of activation function o-)) and the priors on the network weights and
biases.
Many of the covariance functions corresponding to commonly used activation functions under Gaussian priors on the weights have closed-form representations. The ArcCos kernel [3] covers the ReLU (AreCos-1) and the step (ArcCos-O) activations, the so-called 'neural network kernel' [59, 44] links to sigmoidal activations. Gram matrices K., = x(xr, dfj), evaluated for input pairs x and x' over x, x' E 1, 3]. are visualized in Fig. 2a to illustrate these covariance structures. The models induced by the ReLU, step, and sigmoid activations are by nature local, with the concentration of the non-linearity around the origin. As can be seen in the ID regression results in Fig. 2b, the strong inductive bias in the prior (model) is reflected in the posterior. The ReLU extrapolates by matching the local bias of the data, while the step and sigmoidal saturate outside the data, and priors are clearly sensitive to translations and highly non-stationary
Stationarity For stationary (homogeneous) covariance function the covariance structure of the model functions (x) is the same regardless of the absolute position in the input space, and thus the parametrization can be relaxed to x(x,x') & n(x - x') = <(r). Stationarity implies translation- invariance, i.e. the notion of similarity between inputs x, and x' is only a function of their distance x t-x x'. This implies weak stationarity under stochastic process theory [40], and results in reversion to the prior outside observed data (see Fig. 2b). For stationary covariance functions, the best-known family is the Matérn [32, 49] family of kernels, which features models with sample functions of varying degrees of differentiability (smoothness):
Maat. (x, x')
21
(V22
xx11 - L
|x
K, 22
x x'Il
(2)
I'2))
where v is a smoothness parameter, a characteristic length-scale parameter, (.) the modified Bessel function, and I(-) the gamma function. For the Matérn class, the processes are [v]] 1 times mean-square differentiable, and the family includes the RBF (squared exponential) and the exponential (OrnttennUUhlenbeete kernels as limiting cases as + oo and = 1/2.
Local stationarity Neural networks do not naturally exhibit stationarity. Formally, this stems from the problem of representing non- non-linnasisies over infinite input domain with a finite set of local non-linear mappings. However, to bridge the gap between neural networks and widely-used stationary kernels, the typical approach is to restrict the invariance to be local (see Fig. 2a). This is the approach in the RBF-NN [60, 44] and general Matérn-NN [34] activation functions, where the prior is a composite covariance function with a Gaussian decay envelope (see discussion in [15]), Le..
(3) where 2 = 20 + l (see Fig. 2a for the decay effect of the envelope). One motivation for this approach is to assume Gaussian input density the training/testing inputs, thus restricting our
MM (x, x') x exp(-x x/ x/20"n) KMa. (x, x') exp(-x' Tx 1220

---

M
W
WWW
W
-2
Sinusoidal activation
Sinc-Cosine activation
Triangle wave activation
Periodic ReLU activation
Figure 3: Sketches of types of periodic activations functions a--) visualized over two repeating periods each. The two leftmost infinitely differentiable and the two rightmost piecewise linear. The dashed line overlays the Sinusoidal activation for comparison.
interest only to inputs the vicinity of the training data. The local behaviour is highlighted in Fig. 2b, where the locally stationary models revert to the mean (as expected), but the marginal uncertainty of the model functions drops to zero when far enough from training samples. This motivates us to seek activation functions that induce (globally) stationary behaviour (bottom-row in Fig. 2).
Global Stationarity-inducin Activation Functions
We derive a direct connection between (fully) stationary covariance functions and periodic activation functions. Our main result leverages the spectral duality of the spectral density of stationary covariance functions to establish direct connection between the weight prior and the induced prior on the function space. Furthermore, basis and establishes an equivalence relationship between a Student-t distribution on the weights and prior the function space with Matérn covariance structure.
show that this connection is not restricted to the (sinusoidal) Fourier
3.1 Spectral Duality Under Harmonic Functions
For stationarity-inducinn priors, the covariance function be written equivalently in terms of its spectral density function. This stems from Bochner's theorem (see [6] and 36A in [28] for more general formulation) which states bounded continuous positive definite function x(r) can be represented through the following norm-presse corm-preserving isomorphism
<(r) = = ((T) p(dw),
(4)
where (s a positive measure and r E Rdd the measure (uw) has density, it is called the spectral density S(w) corresponding the covariance function s(r). With w(r) (2x) (or))=exp(iod (i r). this gives rise to the Fourier duality of covariance and spectral density, which is known as the Wiener-Khindl Khinchin theorem (e.g., [44]):
<(r) (2 / ((w)eew'r de and S(w) - (<(T) -iw'r dr.
(5)
For d > 1, if the covariance function is isotropic (it only depends on the Euclidean norm ||||| such that (()r) A ([[[]]])) then it is invariant all rigid motions of the input space. Moreover, the spectral density will also only depend on the norm of the dual input variable w. In the neural network case, assume the previous layer will take care of scaling the inputs, and thus effectively interested in the isotropic case, which brings us to analysing ID projections.
An important question is whether we are restricted to the (sinusoidal) Fourier basis. Let (z)(2) be any sequence of bounded, qquare-integrabee (over its period) periodic functions, such that these functions define an inner product space (generalising a basis of vector space). In signal processing terms, they define a frame that provides a redundant yet stable way representing a signal. From a generalised harmonic perspective [58], we could resort to any convenient basis, such as Gabor wavelets [1 1]. However, in this work we seek choose (xx) suitably, such that we have uniform convergence to w(r) (cf., Eq. (4)), ensuring that we retain the spectral density S(w).
5

---

3.2 Types of Periodic Activation Functions
We show how various periodic activation functions (see Fig. 4) under the view of Eq. (1) directly link back to the spectral duality. These periodic activation functions are continuous, bounded, and centered zero, while not necessarily differentiable and never monotonic over R (but monotonic within half-periods
Sinusoidal activation By assuming the activation function to be a sinusoid, (si((z) = 22 (a) (see Fig. 4), placing uniform prior the bias term, p(b) UUiffrmm-n; x), and substituting this into Eq. (1), we can obtain an expression for the covariance function <(z,x"):
x(x, x') = p(()) [
sin (we + b) sin(wa' + b) db dw.
(6)
By solving the inner integral and applying Euler's formula (cos(z) = _ee²= = i=))), we obtain:
<(r, x') = ((w)eeetees dw + ʃ (owwe-lu(e-- dwc
which simplifies under the assumption that the prior on the weight is symmetric and has support on the real line, i.e.,
k(I, x') - f (u())il(eree dw.
(8)
By letting r x x', we find that we recover the spectral density decomposition of a stationary process given by the Wiener-Khiin Khinchin theorem in Eq. (5), where p(w) Σम obtain a direct connection between the prior on the weights and the spectral density. A detailed derivation can be found in App. A.I.
3 S(w). Therefore, we
Sine-Cosine activation An alternative way of using sinusoidal activation functions is the use of composite sine-cosine activation, i.e., Jsin cos (z) = sin(x) + cos(r) (Fig. 4 compares the sin-cos to the sinusoidal activation). Using such construction has the benefit of removing the need to integrate over the bias, which can result in reduced variance [51] of the estimates, giving us covariance function in the form of:
k(x,x')= = f p(w) [sin(wz) + cos(wwr)) (x:)] [sin(wx') + cos(wa')) dw = p(w) sin(w(( + x²)) dw + p(w) cos(w(z x²)) dw.
(9)
By application of Fuler's formula and under the usual assumption that p(w) is symmetric and has support on the entire real line, the above reduces to:
1 2/2 [ p(w) jimta (') dw + / p(w) e -io((===') dw = p(w) jur(a-a) dw. (10)
<(x, x') =
Again, by letting r = x x- x', we find that we recover the spectral density decomposition of stationary process given by the Wiener-Khined theorem. A detailed derivation can be found in App. A.2.
Triangle wave activation Inspired by the success of piecewise linear activation functions (e.g., ReL.U [36], leaky ReI.U [54], PReLU [17]), we show that the triangle wave, a periodic piecewise linear function, can be used instead of sinusoidal activations. The triangle wave given by:
4 2 p 2
2x
(x)
(11)
where is the period and denotes the floor function. By considering a period of 2m, assuming a uniform prior on the biases, Uniform( (-T1, x). and appropriate rescaling, we get the activation
1 1
Oriangle (x)
r π
(12)
2
for which we obtain direct correspondence of the network weight priors to the spectral density. As shown the derivations in App. A.3, we find that we again recover the spectral density decomposition theorem.
of stationary process given by the

---

Exponemtil
Matérn-
RBF
Rxaet Sinusoid
Exponential
Matém-
RBF
Exact
Triangle ware
Sine- Mine-Cosine
Periodic ReLU
Input, Jar 11
Input, x'1
Figure 4: Covariance functions calculated by MC integration with 5000 samples and compared to their exact dashed counterparts (peaks shifted for clarity). Regardless of the type of periodic activation function used, recover the behaviour of the stationary kernels (see also Fig. 8a).
Periodic ReLU activation To go even further, we define a piecewise linear periodic activation function with repeating rectified linear pattern, which we call the periodic ReLU function. It can be defined in terms of the sum of two triangle waves, with the second one being shifted by of period. The resulting periodic function is piecewise linear and defined as:
(x) 72 ( (1r
c + 5)
2.
(=+ + + -I
x
(1)
1 2
(13)
assuming period of 2T. It is again possible obtain correspondence between the network weight prior and the spectral density by considering weighted version of the equation above. In particular, show in App. A.4 that from the following weighted periodic ReLU activation function,
()RaLU(()= (x)= T (( T
(x + 5) 1
(2) +
x
(1))+++)))
I
π
π
(14)
we again recover the spectral density decomposition of stationary process given by the Wiener- Khinchin theorem, providing a direct connection between the prior on the weights and the spectral density. Note that, choosing a piecewise linear periodic activation function has potential computational benefits compared to sinusoidal activation functions and can help prevent vanishing gradients.
3.3 Kernel Functions
We have established that is it possible obtain direct correspondence of the prior on the weights and the spectral density of a stationary covariance function by using periodic activation functions in random neural networks. In App. B we show that by placing Student-t distribution on the weights with degrees of freedom of u = 2v we recover the spectral density of the Matérn family, i.e.
r( metll
w
[ + 1) [(v)
(2v)" (2v+ + +w? (1++)
1 SMar. (w),
p(w)
1
2 2T
V VuTI
2T
(15)
where p(w) denotes the probability density function of Student-t distribution and SMa. (w) denotes the spectral density of the Matérn family. This means that a Student-l prior the weights in combination with an appropriately scaled periodic activation function corresponds directly to a prior in the function space with Matérn covariance structure. Fig. 4 verifies this result and shows that we recover the exact covariance function (dashed) for various examples from the Matérn family by Monte Carlo (MC) integration (5000 samples) with all of the discussed periodic activation functions.
Recall that the Matérn family contains the exponential kernel (v 1/2) and the RBF Kernel (v oo) as special cases. Similarly, the Student-t distribution has the Cauchy (u = 1) and the Normal distribution (u = oo) as limiting cases, resulting in correspondence the Matérn family. Table 2 in App. B summarises the set of priors on the weights that correspond to the spectral density of kernel functions the Matérn family. This correspondence is exact for the sinusoidal and sine-cosine activations. For the triangle wave and periodic ReLU activations, the obtained correspondence approximate, and the introduced approximation error analyzed in detail in App. A.
the respective special cases in
7

---

Table 1: Examples of UCI regression tasks, showing the globally stationary NN model directly gives competitive RMSE and for most data sets also improved mean negative log predictive density (NLPD) compared non-stationary and locally stationary NN models. KFAC Laplace was used as the inference method. High std is due to variability in 10-fold-cv splits of the small data sets, not method performance.
BOSTON (306, 12) 50)
CONCRETE
AIRFOIL
ELEVATORS (16899, 18)
(HL,
(10500,5
(1503, 51 (1)
(1. 50)
500)
NLPD
RMSE
NLPD
RMSE
NLPD
RMSE
NLPD
RMSE
RoLU
0.5120032
0.3740.07
0.784:0.16
0.48400.4
0.5120.53
0.411:0.21
0.3840.03
0.354:0.01
RRF
0.5240.30
0.3740.08
0.7840022
0.44400.0
0.1040.15
0.264:0.03
0.4140.04
0.3540.001
glob (sin)
0.42:40.34
0.36:20.07
0.744:0.15
0.49+:0.05
0.1420.17
0.29::0.05
0.38:20.08
0.35::0.01
glob RBF (prele)
0.39:200.0
0.30:10.07
0.744:0.14
0.491:0.04
0.08:10:12
0.26:10.03
0.74:0.78
0.46:10.21
ooe Mat-M/2
0.71#:0.38
0.40==0.08
0.8440.28
0.42#0.04
0.11
0.26+:0.03
0.43+00.04
0.3540.01
Mat32/ 3/2
0.43:10.22
0.39:10.06
06
0.49:10.05
0.0710.11
0.274:0.03
0.37100.00
0.35:10.01
gloh Mar3/2 (pectu)
0.38:1.222
0.382:008
0.72:0.18
0.4810006
0.08:0.12
0.274:0.33
0.30:00003
0.361:0001
4 Experiments
To assess the performance of cuttionarit-indducinn activation functions, we compared the in-domain and out-of-domain predictive uncertainty to non-stationary and locally stationary models on various benchmark data sets. In all experiments, the compared models differ only by the activation function and the respective weight priors of the last hidden layer of the network. Replacing the prior effects the initialization of the NN weights and the training objective, detailed description of the experimental setup is available in App. D. The illustrative toy BNN examples are implemented using Turing.jl [14], GP regression results use GPflow [33], and all other experiments are implemented using Py PoTch [41].
maximize the log joint density.
Illustrative toy examples Fig. shows predictive densities for non-stationary. locally stationary, and globally stationary activation functions on the banana classification task. The top row illustrates the predictive densities of infinite-width BNNs (GP), the bottom row shows corresponding results for single hidden layer BNN using 30 hidden units. We observe that global stationarity-inducin activation functions revert to the prior outside the data, leading to conservative behaviour (high uncertainty) for out-of-domain samples. Moreover, we see that the mhite-width BNNs result in similar behaviour to their fhinite-width counterpart, while locally stationary activation functions in finite-widie BNNs exhibit a slower reversion to the mean than their anfiiitewwdtth corresponding GPs. Additionally, we include ID toy regression study that highlights the differences between different prior assumptions encoded by choice of the activation function. Fig. shows the corresponding prior covariance as well as posterior predictions for the infinite- width (GP) model. We replicated the experiment with finite-wiid width network and observed that the behaviour translates to finite width (see Fig. 8 in App. D.I).
UCI benchmarks Table 1 shows results on UCI [8] regression data sets comparing deep neural networks with ReLU, locally stationary RBF [60], and locally stationary Matérn-3/2 [34] against global stationary models. An extended results table can be found in App. D.2 and additional results
Mean confidence
Accuracy
0.6
0.3
NLPD
0.2 0.4
0.6
0"
90"
180"
270"
360
0"
90
180
270"
360
0"
90
180"
270"
360
Rotation angle
Rotation angle
Rotation angle
Figure 5: Rotated MNIST: The models have been trained on unrotated digits. The test-set digits are rotated at test time to show the sensitivity of the trained model to perturbations. All models perform equally in terms of accuracy, while ReLU (-) shows overconfidence in terms of mean confidence and NLPD. The stationary RBF models (- local, sin, sin-cos) capture uncertainty.

---

ReLU activation function
0.3
0.3
CIFAR-10 CIFAR-100
CIFAR-10
CIFAR- 100
0.2
SVHN
0.2
SVHN
0.1
0
0.5
1.5
2
0
0.01
0.02
Predictive entropy
Predictive marginal variance
Locally stationary Matérn-
3 activation function
0.3
0.3
0.2
0.1
0.1
0
0.5
1.5
2
0
0.01
0.02
Prodictive entropy
Predictive marginal variance
Periodic activation function (Matérn-
kernel)
0.3
0.3
0.2
0.1
0.1
0.5
1.5
0.01
0.02
Predictive entropy
Predictive marginal variance
7 48
9
8Cg
9
Most similar
CIFAR-10
Least simnllar
similar
CIFAR-100
Least similar
Most similar
SVHN
Least similar
Figure 6: OOD detection experiment results for models trained on CIFAR-10 and tested on CIFAR- 10. 100, and SVHN. Top rows show predictive entropy and predictive marginal variance histograms test images for in-domain and OOD data sets. Top to bottom: ReLU activation function, local stationary Matérn-32 activation function [34], and sinusoidal activation function that induces a globally stationary Matérn-3/ model. The last row shows sample images from each test set: left-side images with the lowest entropy/highest confidence and right-side images with the highest entropy/lowest confidence. See App. D.5 for histograms for more models.
on UCI classification tasks in App. D.3. Table lists root mean square error (RMSE) and negative log predictive density (NLPD), which captures the predictive uncertainty, while the RMSE only accounts for the mean. Global stationary models provide better estimates of the target distribution all cases while obtaining comparable RMSEs. It is important to note that the large standard deviations in Table I are due to the small number of data points and the fact that some splits in the 10-fold CV end up being harder than others. The posterior inference is performed using KFAC Laplace [45].
Detection of distribution shift with rotated MNIST Fig. 5 demonstrates the predictive behaviour of different activation functions under covariate shift. We evaluate the predictive accuracy, mean confidence of the predicted class, and NLPD on MNIST [25] for different rotations of the input. Models are trained unrotated digits using KFAC Laplace [45]. The results indicate that all models obtain similar accuracy results, while only local and global stationary models do not result in over- confident uncertainty estimates. For an ideally calibrated model, the mean confidence would decrease as low as the accuracy curve when the digits are rotated, keeping the NLPD values as low as possible.
Out-of-distributie detection using CIFAR-10, CIFAR-100, and SVHN Fig. 6 shows model performance on OOD detection in image classification for ReLU, locally stationary Matérn-3/2, and globally stationary Matérn-3/2 models. The models have been trained using SWAG [29] on CIFAR-10 [23], and tested on CIFAR-10. CIFAR-100 [23], and SVHN [39] test set images. Both CIFAR- 100 (more similar) and SVHN (more dissimilar to CIFAR- 10) images are OOD data, and the

---

models should show high uncertainties for the respective test images. We use two metrics to measure model uncertainty: predictive entropy and predictive marginal variance. Predictive entropy captures uncertainty present in the predicted mean class weights, while the predictive marginal variance captures uncertainty related to the variability in the predicted class weights. From these two metrics, the predictive marginal variance could be more suitable as an OOD detection metric, as predictive entropy can justifiably give high values for some in-distribution samples, for example, those that the model confidently predicts being in between two known classes.
The histograms of predictive entropies for different test sets show that all models can separate between in- on-distribution and OOD data to some extent, and all models consider SVHN more OOD than CIFAR- 100, which intuitive as CIFAR- 1-0 resembles CIFAR- 10 more. Also, the predictive marginal variance histograms show that the models have higher variance on OOD data compared to in-distribution data. However, the ReLU model considers CIFAR- more OOD than SVHN and the locally stationary Matérn-3/2 model has overall relatively small predictive marginal variance values for all data sets. We consider predictive marginal variance better metric for OOD detection than predictive entropy, as in-distribution samples that are hard to classify are expected to have high predictive entropy, not necessarily allowing the detection of OOD samples on this metric. See Fig. 14 in App. D.5 for results on different models, and Table 8 in App. D.5 for numerical results showing the area under the receiver operating characteristic curve and the area under the precision recall curve for the OOD detection experiment.
Discussion and Conclusions
We have shown that periodic activation functions in Bayesian neural networks (BNNs) establish direct connection between the prior on the network weights and the spectral density of the limiting stationary process. This link goes beyond the Fourier (sinusoidal) basis, which we have illustrated by deriving the correspondence also for the triangle wave and a new periodic variant of the ReLU activation function. Moreover, we have shown that for BNNs with a periodic activation function, placing a Student-t distribution on the network weights corresponds to a prior in the function space with Matérn covariance structure. This correspondence is exact for the sinusoidal and sin-cos activations and approximate for the triangle and periodic ReLU activations.
Our work can help build neural network models that dacu-occurring either naturally or maliciously. As demonstrated in the experiments, the modelling assumptions have practical importance in reducing sensitivity to input shift. Perturbation of inputs can be captured in the last layer a deep model, and the principled choice of a priori stationarity has shown to be effective based the experiments. Using periodic activation functions in BNNs induces global stationarity. i.e., the model not only locally behaviour across the input domain. The resulting model reverts the prior for -fstt--diittibuttion (OOD) samples, resulting in well-specified uncertainties.
robust and less vulnerable to OOD
which induces conservative
In contrast to models with non-stationary or locally stationary activations, optimising and performing approximate inference models with periodic activation functions has proven to be more challenging. In particular, found that the uncertainty over the bias terms is often not accurately estimated when using KFAC Laplace or SWAG, which required us to use a relatively large number of hidden units. However, the experiments using dynamic HMC indicate that the number of hidden units can be drastically reduced using a more accurate approximate inference scheme, as seen in Fig. 1 and Fig. 11 in App. D.1.
The codes and data to replicate the results are available under MIT license at fttaps://githann com//aaltoMl/Periodee
Acknowledgments and Disclosure of Funding
We acknowledge the computational resources provided by the Aalto Science-IT project. This research was supported by the Academy of Finland grants 324345 and 339730, and Saab Finland Oy. We thank William J. Wilkinson and the anonymous reviewers for feedback on the manuscript.
10

---

References
[1] L. Aitchison. A statistical theory of cold posteriors in deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
E. J. Candès. Harmonic analysis of neural networks. Applied and Computational Harmonic Analysis, 6(2): 197-218, 1999.
Y. Cho and L. K. Saul. Kemel methods for deep learning. In Proceedings of Advances Neural Information Processing Systems (NIPS), pages 342-350. Curran Associates, Inc., 2009.
[4] C. Cortes and Vapnik. Support-vector networks. Machine Learning, 20(3):273-295 297, 1995.
[5] N. A. Cressie. Statistics for Spatiaaa Data. Wiley series in probability and mathematical statisties. Wiley- Interscience, 1991.
[6] G. Da Prato Zabczyk. Stochastic Equations in Infinite Dimensions, volume 45 Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1992.
[7] A. de Matthews, Hron, M. Rowland, Turner, and Ghahramani, Gaussian process behaviour in wide deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[8] D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://atcchivee -ics. www.uciiuuuuuu edu/ml.
[9] A. Y. Foong, D. R. Burt, Y. Li, and R. E. Turner. On the expressiveness of approximate inference in Bayesian neural networks. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), pages 15897-15908. Curran Associates, Inc., 2020.
[10] V. Fortuin, A. Carriga-Alonso, F. Wenzel, G. Rätsch, R. E. Turner, M. van der Wilk, and L. Aitchison. Bayesian neural network priors revisited. In Third Symposium on Advances in Approximate Bayesion Inference, 2021.
[11] D. Gabor. Theory of communication part 1: The analysis of information. Journal of the Institution of Electrical Engineers- Part Radio and Communication Engineering. 93(26):429 441, 1946.
[12] Y. Gal and Z. Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning (ICML), volume of Proceedings of Machine Learning Research, pages 1050-1059. PMLR, 2016.
[13] A. R. Gallant and H. White. There exists a neural network that does not make avoidable mistakes. In Proceedings of International Conference on Neural Networks (ICNN), pages 657-664, 1988.
[14] H. Ge, K. Xu, and Z. Ghahramani. Turing: A language for flexible probabilistic inference. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 84 of Proceedings of Machine Learning Research, pages
PMLR, 2018.
[15] M. G. Genton. Classes of kernels for machine learning: A statistics perspective. Journal of Machine Learning Rexearch, 2:299-312, 2001.
[16] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
[17] K. Hc, X. Zhang, S. Ren, and Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[18] M. Hein. M. Andriushchenko, and Bitterwolf. Why ReLU networks yield high-confidence predictions away from training data how to mitigate the problem. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 41-50, 2019.
[19] Hensman, N. Durrande, and Solin. Variational Fourier features Gaussian processes. Journal of Machine Learning Research, 18:1-52, 2018.
[20] T. Hofmann, B. Schölkopf, and A. Smola. Kernel methods in machine learning. The Annals of Statistics, pages 1171-1220, 2008.
[21] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernl: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31, pages 8571 15800 Curran Associates, Inc., 2018.

---

[22] A. Kristiadi, M. Hein, and P. Hennig. Being bayesian, even just bit, fixes overconfidence in ReLU networks. In Proceedings of the International Conference Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 5436-5446. PMLR, 2020.
[23] A. Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 2012.
[24] M. Lázaro-Gree Greailla, J. Suufionero-Candele C. E. Rasmussen, and A. R. Figueiras-Vidde Sparse spectrum Gaussian process regression. Journal of Machine Learning Research, 1 11:1865-188) 1881, 2010.
[25] Y. LeCun, C. Cortes, and C. J. Burges. The MNIST database of handwritten digits, 1998. URL http: //yann. lecun. com/ wwt.cotc//c//eceecttttccoco.o exdb/mnist/.
[26] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and Sohl-Dickstein. Deep neural networks as Gaussian processes. International Conference on Learning Representations (ICIR), 2018.
[27] Lee- Sho-Th J. Ainslic, I. Eckstein, and S. Ontañón. Fnet: Mixing tokens with Fourier transforms. ArXiv preprint arXiv: 100..5508, 2021.
[28] L. Loomis. An Introduction to Abstract Harmonic Analysis. D. van Nostrand Company, Inc., New York, 1954.
[29] W. J. Maddox, P. Izmailov. T. Garipov, D. P. Vetrov, and A. G. Wilson. simple baseline for Bayesian uncertainty deep learning. In Advances in Neural Information Prucessing Systems 32 (NewrIPS), pages 13132-13149 143. Curran Associates, Inc., 2019.
[30] H. Maennel. Uncertainty estimates and duitt--distribution detection with Sine Networks. In (CML.2019 Workshop on Uncertainty and Robustness in Deep Learning, 2019.
[31] G. Marcus. The next decade in AI: Four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
[32] B. Matérn. Spatial variation: Stochastic models and their applications to some problems forest surveys and other sampling investigations. Meddelanden från statens skogsforskninggsisstitt, 49:1- 14:4,14,, 1960.
[33] A. G. d. G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P. León- Velll-ill,gáá, Z. Ghahra- mani, and J. Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Rexearch, 18(40):1-6, 2017.
[34] L. Meronen, C. Irwanto, and A. Solin, Stationary activations for uncertainty calibration in deep learning. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), pages 2338-2350. Curran Associates, Inc., 2020.
[35] P. Morales-Alvarez, D. Hermández-I R. Molina, and J. M. Hernánder- obato. Activation-Ievel uncer- tainty in deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[36] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. Proceedings of the International Conference on Machine Learning (ICML), pages 807-814. Omnipress, 2010.
[37] E. T. Nalisnick. Gordon, and J. M. Bernández-Lobate Predictive complexity priors. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130 of Proceedings of Machine Learning Research, pages 694 694-702. PMLR, 2021.
[38] R. M. Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, Toronto, Canada, 1995.
[39] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2. URL http://ufidl. stanford. adu/hhossenubeer.
[40] A. Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw-Hill New York, 1991.
[41] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. DeVito, M. Raison, A. Tejani, Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, perfooerformance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 8024 8035. Curran Associates, Inc., 2019.
12

---

[42] T. Pearce, R. Tsuchida, M. Zaki, A. Brintrup. and A. Neely, Expressive priors in Bayesian neural networks: Kernel combinations and periodic functions. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), volume 115 of Proceedings of Machine Learning Research, pages 134-144. PMLR. 2019.
[43] A. Rahimi and B. Recht. Random features for large-scale kemel machines. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), pages 1177-1184. Curran Associates, Inc., 2008.
[44] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge. MA, 2006.
[45] H. Ritter, A. Botev, and D. Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018.
[46] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Nenworks, 61:85-117, 2015.
[47] V. Sitzmann, J. Martel, A. W. Bergman, D. B. Lindell, and Wetzstein. Implicit neural representations with periodic activation functions. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 7462-7473. Curran Associates, Inc., 2020.
[48] A. Solin and Särkkä. Hilbert space methods for reduced-rank Gaussian process regression. Statistics and Computing, 30(2)419 446, 2020.
[49] M. L. Stein. Interpolation of Spatial Data. Springer Series in Statistics. Springer, New York, 1999.
[50] S. Sun, G. Zhang. J. Shi, and R. B. Grosse. Functional variational Bayesian neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
[51] D. J. Sutherland and J. G. Schneider. On the error of random Fourier features. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pages 862-871. AUAI Press, 2015.
[52] C. Szegedy, W. Liu, Y. Jia, P. Sermanct, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9,20 2015.
[53] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi,
J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 7537-7547. Curran Associates, Inc., 2020.
[54] R. Tsuchida, F. Roosta, and M. Gallagher. Invariance of weight distributions in rectified MLPs. In Proceedings of the International Conference Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages
PMLR, 2018
[55] D. Ulmer and G. Cinà. Know your limits: Uncertainty estimation with ReLU classifiers fails at reliable OOD detection. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 Proceedings of Machine Learning Research, pages 1766-1776. PMLR, 2021.
[56] M. Uteuliyeva, A. Zhumekenov, R. Takhanow, Z. Assylbekov, A. J. Castro, and O. Kabdolov. Fourier neural networks: A comparative study. Intelligent Data Analysis, 24(5):1107-111 2020.
[57] F. Wenzel, K. Roth, S. Veeling, J. Swiętkowski, L. Tran, S. Mandt, J. Snoek. T. Salimans, R. Jenatton. and S. Nowozin. How good is Bayes posterior in deep neural networks really? In Proceedings of the International Conference on Muchine Learning (ICML), volume 119 of Proceedings of Muchine Learning Research, pages (0248-10255 PMLR, 2020.
[58] N. Wiener. Generalized harmonic analysis. Acta Mathematica, 55:1177258, 1930.
[59] C. K. I. Williams. Computing with infinite networks. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 295-222333 301. MIT Press, 1997.
[60] C. K. I. Williams. Computation with infinite neural networks. Neural Computation, 1005):1203-1216 1998.
[61] K. Wong, C. Leung, and S. Chang. Handwritten digit recognition using multi-layer feedforward neural networks with periodic and monotonic activation functions. In Proceedings of International Conference on Pattern Recognition (ICNN), pages 106-109, 2002.
[62] L-C. Yeh. Modeling of strength of high-hhihhonreeee performance concrete using artificial neural networks. Cement and Concrete Research, 28:1797-1808. 1998.
13


Reviews:
[{"review_id": "pbeg_E4vfe4", "reviewer": "Reviewer_KDY4", "summary": "The paper utilizes periodic activation functions in concert with BNNs to induce global stationarity. Most of the paper is devoted to explaining and deriving the connection between the priors of the BNN parameters (+ activations) and the GP's kernel. The paper additionally assesses the impact of global stationarity on in- and out-of-distribution examples.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nThe method combines/links several methods (BNNs, periodic activations, GPs, etc).\nThe combinations and connections appears to be novel.\nThe paper is well-cited.\n\n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nMost of the paper is spent explaining and developing global stationarity. The analysis seems correct though I did not review the derivations in the Appendices. The experimental results show comparable performance to standard methods while better informing model uncertainties. The toy, UCI, and MNIST experiments are good illustrations of the impact of the method. Figure 5 is especially nice for demonstrating model performance under distribution shift.\n\nCIFAR-10 vs similar datasets are a standard test for OOD assessment. The paper includes some of the standard OOD datasets for comparison (CIFAR-100 and SVHN) but does not provide any quantitative comparison which makes it difficult to compare to other OOD methods. While Figure 6 provides an encouraging qualitative comparison of the datasets, the lack of (or justification for the lack of) quantitative comparisons is strange. I would like to see some combination of AUROC, AUPR, etc. from whatever the OOD detection criteria is (predictive entropy/marginal variance?).\n\n\nClarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nThe development and writing make sense and are accessible. Clarification of the model architectures would be helpful. From the Appendix, the more complex models appear to be a feature extractor followed by the stationarity-inducing FC layer(?). Overall, the supplemental helped to clarify a few confusing points.\n\n\nSignificance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\nThe authors make a good argument for global stationarity and develop how to accomplish it. They provide an interesting connection\nbetween BNNs and GPs.\n\nThey do a reasonable job exploring their method and the results are encouraging. However, the lack of baseline comparisons (especially for the OOD task) makes it difficult to position the method.\n\n\nOverall:\n\nThe paper is reasonably well-written and clear about its intentions. I appreciate the analysis and justification in how the activations and priors are chosen, instead of brute-force empirical evaluation. I think they have done a good job explaining, justifying, and developing their ideas. My main concern is the lack of quantitative assessment and comparisons on the CIFAR-10 OOD task.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ainRt_AI24H", "reviewer": "Reviewer_pm5N", "summary": "Update: I have read the author feedback and am quite satisfied with it and thus raised my score.\n\n******\n\nThis paper relates to Bayesian Neural Networks (BNN). It proposes to use periodic activation functions in order to deal with out-of-domain (OOD) samples and uncertainty. Series of experiments hint that with the method OOD performance is improved while in-domain performance is not worse than with the baseline methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of OOD uncertainty for neural networks is an important one and this paper describes an interesting method towards solving this.\n\nThe theoretical part seems original, especially related to exploring different kinds of periodic activation functions in detail.\n\nThe experimental part of the paper has some drawbacks. For example, it should have a quantitative comparison to other OOD approaches, since that is the main focus of the method.\n\nTable 1 states that globally stationary model gives improved NLPD for most datasets. Could the authors explain how these 4 datasets were chosen and how do the methods’ performance change if some other datasets, e.g., from UCI are chosen.  Also, while the statement of improved perf is true  when compared to relu, the difference to locally stationary models is lower especially given that different activations seem to suit different datasets. Since these datasets are not explicitly dealing with OOD conditions, it would be enough to demonstrate similar performance to locally stationary activations and similar improvement over relu, not consistently improved results.\n\nIn Fig.1 the locally stationary activation function BNNs do not seem to reach the performance of the GP counterparts, while this seems not to hold for globally stationary ones. Could the authors explain this behaviour?\n\nThe main problem with the paper is that quantitative evaluation of the main benefit of the method, namely proper uncertainty for OOD samples is too thin. Most of the OOD results in the paper require qualitative assessment and I think this is unacceptable given OOD is the main focus of the method.\n\nIn Fig. 5, the rotated MNIST experiment seems to hint at overconfidence with the ReLU baseline model, although it would be good to quantify this overconfidence. Also, it seems that in this case both locally and globally stationary models seem to work as well in this OOD case, while the main claim of this paper is that the globally stationary activations should lead to better performance in OOD cases.\n\nFig. 6 seems to provide the most important experimental results of the paper related to deep learning and OOD. Unfortunately the main paper only shows single model results and while the supplement shows the same results for all models, the results require interpretation.  It would be better to have quantified results that compare locally and globally stationary and baseline models. Could the OOD detection task be quantified into a single measure and then the different methods could be easily compared in the main paper.  \n\nOverall, decent paper, but would require more work in the results section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "3FUteKo1437", "reviewer": "Reviewer_S2q4", "summary": "The paper shows that periodic activation functions in the last hidden layer of a Bayesian neural network are related to stationary Gaussian process priors. With the periodic activation functions Bayesian inference correctly produced less confident predictions at out-of-distribution test data.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed usage of the periodic activation functions in the last hidden layer of a neural network is practical.\nThe obtained results are comparable to ReLU networks and are correctly less confident on out-of-distribution test data.\nThe usage with KFAC Laplace does not require a special network training to perform the approximate Bayesian inference.\n\nClarity: The paper is well written. The experiments are well explained.\n\nQuality: The theory is well explained and numerically verified in Figure 4.\n\nOriginality: The Gaussian process literature is aware of the benefits of stationary kernels. Previously, Meronen (NeurIPS 2020) proposed an activation function for a locally stationary model. The current paper goes one step further, proposing activation functions for global stationarity.\n\nSignificance: The paper is practical and relevant to the commonly used deep neural networks.\n\n\nA question for the authors:\nDo you see benefits from the periodic activation functions on OOD data only if using the Bayesian inference?\nIt would be informative to compare the Bayesian inference to NLPD results on the Rotated MNIST when using only the mode of the parameters.\n\nMinor comments to help to improve clarity:\n- Define the BNN abbreviation before using it in Figure 1.\n- Line 94: Define \"x\" and define early that the E[...] expectations are over model functions.\n  Make it clear that \"x\" and \"x_i\" have the same dimensionality.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper helpfully mentions that the periodic activations functions required a relatively large number of hidden units.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1-SJtT49H1Q", "reviewer": "Reviewer_L92o", "summary": "In this paper, the authors present the effect of using periodic activation functions in bayesian neural networks (BNN). More precisely, they show a connection between the priors on the network weights and translation invariance stationary Gaussian Process priors.\nThe authors show how this applies not only for sinusoidal, but also triangular and ReLU based periodic activation functions. \nEmpirically, they demonstrate how this architecture choice still recovers good in-distribution behavior, while improving the out-of-distribution sensitivity. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One significant research area in machine learning is the one of model robustness. In the context of bayesian neural networks, we are interested in the modeling assumptions that can be introduced as priors, as well as the posterior predictive behavior.\nUnfortunately, the choice of activation functions have a significant impact on the sensitivity of the model.\n\nIn this paper, the authors show how certain periodic activation functions in connection with the Student-t distribution on the weights, can induce global stationarity. In particular, the look at sinusoidal, sine-cosine, triangle wave, and, periodic ReLU activations.\n\nThe authors show in synthetic datasets how the posterior predictive density behaves with regards to the distribution of the data and how the uncertainty increases far from the distribution. On real datasets, the results are also promising especially when comparing against distribution shifts and OOD tests.\n\nOne question that follows from this type of approaches is whether the use of the periodic activations makes the training of the network harder?\nIn any case, the paper is well-written, technically sound and it provides an interesting stepping stone in model robustness.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not discussed, but not a significant issue here.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an interesting link between periodic activation functions, which have shown excellent empirical performance in fitting low-dimensional functions in models like SIREN, and stationarity in the infinite-width limit, connecting these networks with the Gaussian Process literature. All reviewers agreed that the contribution is interesting and novel, and many of the reviewers raised their scores after the author rebuttal. So long as the authors include the promised results in the camera-ready version, all reviewers were happy to recommend acceptance. I agree with their assessment.
Decision: Accept (Poster)