Periodic Activation Functions Induce Stationarity

Abstract: Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.


Reviews and author responses:

Update: I have read the author feedback and am quite satisfied with it and thus raised my score.

******

This paper relates to Bayesian Neural Networks (BNN). It proposes to use periodic activation functions in order to deal with out-of-domain (OOD) samples and uncertainty. Series of experiments hint that with the method OOD performance is improved while in-domain performance is not worse than with the baseline methods.  The problem of OOD uncertainty for neural networks is an important one and this paper describes an interesting method towards solving this.

The theoretical part seems original, especially related to exploring different kinds of periodic activation functions in detail.

The experimental part of the paper has some drawbacks. For example, it should have a quantitative comparison to other OOD approaches, since that is the main focus of the method.

Table 1 states that globally stationary model gives improved NLPD for most datasets. Could the authors explain how these 4 datasets were chosen and how do the methods’ performance change if some other datasets, e.g., from UCI are chosen.  Also, while the statement of improved perf is true  when compared to relu, the difference to locally stationary models is lower especially given that different activations seem to suit different datasets. Since these datasets are not explicitly dealing with OOD conditions, it would be enough to demonstrate similar performance to locally stationary activations and similar improvement over relu, not consistently improved results.

In Fig.1 the locally stationary activation function BNNs do not seem to reach the performance of the GP counterparts, while this seems not to hold for globally stationary ones. Could the authors explain this behaviour?

The main problem with the paper is that quantitative evaluation of the main benefit of the method, namely proper uncertainty for OOD samples is too thin. Most of the OOD results in the paper require qualitative assessment and I think this is unacceptable given OOD is the main focus of the method.

In Fig. 5, the rotated MNIST experiment seems to hint at overconfidence with the ReLU baseline model, although it would be good to quantify this overconfidence. Also, it seems that in this case both locally and globally stationary models seem to work as well in this OOD case, while the main claim of this paper is that the globally stationary activations should lead to better performance in OOD cases.

Fig. 6 seems to provide the most important experimental results of the paper related to deep learning and OOD. Unfortunately the main paper only shows single model results and while the supplement shows the same results for all models, the results require interpretation.  It would be better to have quantified results that compare locally and globally stationary and baseline models. Could the OOD detection task be quantified into a single measure and then the different methods could be easily compared in the main paper.  

Overall, decent paper, but would require more work in the results section. Not applicable.

 Thank you for the MAP experiment.

 We thank the reviewer for the helpful comments and address the comments below in the order received.

> Do you see benefits from the periodic activation functions on OOD data only if using the Bayesian inference? It would be informative to compare the Bayesian inference to NLPD results on the Rotated MNIST when using only the mode of the parameters.

We observed that the MAP estimate of the globally stationary models (periodic activation functions) was often better than the MAP estimate of locally stationary and ReLU models. However, we did not analyse the results systematically but will add the results for the rotated MNIST experiment to the appendix of the camera-ready paper.

We greatly appreciate the comments to improve the clarity of our paper, as we consider the readability of our paper to be highly important. We will add these improvements to the camera-ready version.


 We appreciate the thorough review and the detailed comments. We address the comments and concerns below in the order received.

### Comment 1

> The experimental part of the paper has some drawbacks. For example, it should have a quantitative comparison to other OOD approaches, since that is the main focus of the method.

The main contribution of our work is to show that the use of periodic activation functions in neural networks induces stationarity and establishes a connection between the prior distribution on the weights and the spectral density of the limiting stationary Gaussian process. Therefore, in the experiments, we focused on assessing the implications of the inductive bias (stationarity) of neural networks with periodic activation functions in the OOD settings and for density estimation. Our experiments intentionally do not aim at comparing the performance of a specific Bayesian neural network architecture with periodic activation functions to other models for OOD. Instead, we aim to show that the use of periodic activation functions empirically helps in OOD settings and allows for accurate density estimation compared to activation functions that do not induce global stationarity, such as ReLU activations. Moreover, the performance of Bayesian neural networks is heavily influenced by the inference algorithm; we refer to recent works on Bayesian inference (e.g. [27,42]) for details. Therefore, we have decided to systematically compare the induced biases under equivalent settings (same architecture and same inference algorithm) instead of a comparison to existing OOD methods.

### Comment 2

> Could the authors explain how these 4 data sets were chosen and how do the methods’ performance change if some other data sets, e.g., from UCI are chosen. Also, while the statement of improved perf is true when compared to relu, the difference to locally stationary models is lower especially given that different activations seem to suit different data sets.

The four UCI regression data sets (and the four classification data sets in the appendix) are typical benchmark data sets from the Gaussian process literature. The list intentionally includes data sets in the small and the large data regime to assess the performance of globally stationary neural networks under various realistic settings. 

### Comment 3

> In Fig.1 the locally stationary activation function BNNs do not seem to reach the performance of the GP counterparts, while this seems not to hold for globally stationary ones. Could the authors explain this behaviour?

The difference between infinite/finite width models for locally stationary models arises since we approximate the binary white prior with a standard Gaussian distribution. The use of a binary white prior would establish direct correspondence, while the use of standard Gaussian introduces a certain error. We refer to Meronen et al. [31] for a detailed discussion on using a standard Gaussian prior instead of a binary white prior distribution.

### Comment 4

> In Fig. 5, the rotated MNIST experiment seems to hint at overconfidence with the ReLU baseline model, although it would be good to quantify this overconfidence. Also, it seems that in this case both locally and globally stationary models seem to work as well in this OOD case, while the main claim of this paper is that the globally stationary activations should lead to better performance in OOD cases.

We consider NLPD values suitable for quantifying the overconfidence in models when a distribution shift occurs, as this metric penalizes both overconfident incorrect predictions and correct predictions with low confidence. Our experiments focus on comparing different inductive biases, i.e., non-stationarity (ReLU), local stationarity, and global stationarity. The NLPD curves in Figure 5 show that the non-stationarity model is significantly more overconfident under distribution shift than the locally or the globally stationary models. In this experiment, the locally and globally stationary models seem to perform equally well. The similar behaviour can be explained by the mean-reversing property of both locally and globally stationary models. As illustrated in Figure 2 (and Figure 7 in the appendix), locally and globally stationary models exhibit mean-reversion (the mean of the prediction distribution converges back to the prior mean outside the data distribution), which drives the results shown in Figure 5. Thus, both models are expected to perform similarly well on this task. However, while globally stationary models revert to the prior (not only the mean) outside the data distribution, locally stationary models only exhibit mean-reversion, and their variance estimate collapses (see Figure 2). Consequently, globally stationary models perform better in OOD detection tasks, as shown in Figure 6 (and Figure 9 in the appendix). We will clarify the results in the camera-ready version and use the additional page to show a direct comparison between the globally stationary and the locally stationary model in the OOD task.

### Comment 5

> Fig. 6 seems to provide the most important experimental results of the paper related to deep learning and OOD. Unfortunately the main paper only shows single model results and while the supplement shows the same results for all models, the results require interpretation. It would be better to have quantified results that compare locally and globally stationary and baseline models. Could the OOD detection task be quantified into a single measure and then the different methods could be easily compared in the main paper.

We will address this issue by using the additional page in the camera-ready version as well as adding the calculated AUROC numbers (see the reply to reviewer KDY4) for the OOD detection task to show quantifiable results in the main paper.
ion. The use of a binary white prior would establish direct correspondence, while the use of standard Gaussian introduces a certain error. We refer to Meronen et al. [31] for a detailed discussion on using a standard Gaussian prior instead of a binary white prior distribution.

### References

[27] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 13132–13143. Curran Associates, Inc., 2019.

[31] L. Meronen, C. Irwanto, and A. Solin. Stationary activations for uncertainty calibration in deep learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 2338–2350. Curran Associates, Inc., 2020.

[42] H. Ritter, A. Botev, and D. Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018.


 We thank the reviewer for the thorough review and comments. We address the comments and concerns below in the order received.

### Comment 1

> I would like to see some combination of AUROC, AUPR, etc. from whatever the OOD detection criteria is (predictive entropy/marginal variance?).

We calculated AUROC values for OOD detection in the CIFAR10 experiment to provide additional quantitative results, treating SVHN as the OOD data set. We calculated the AUROC measure using the marginal variance as the metric to determine whether a sample is OOD or not. We consider this a better metric for OOD detection than predictive entropy, as in-distribution samples that are hard to classify are expected to have high predictive entropy, not necessarily allowing the detection of OOD samples based on this metric. These calculations result in the following AUROC numbers: ReLU 96.1, locally stationary RBF 98.7, globally stationary RBF 98.8, locally stationary Matérn-3/2 98.3, and globally stationary Matérn-3/2 98.1. The results indicate a clear difference between the non-stationary (ReLU) and the stationary (local and global) models. The globally stationary RBF model achieves the best AUROC score. We will utilize the additional page in the camera-ready version to add AUROC and AUPR results.

### Comment 2

> Clarification of the model architectures would be helpful. From the Appendix, the more complex models appear to be a feature extractor followed by the stationarity-inducing FC layer(?).

We agree that it’s potentially difficult to understand the model architecture only from the text. Therefore, we will add an illustrative figure showing the model architectures in the appendix.

### Comment 3

> They do a reasonable job exploring their method and the results are encouraging. However, the lack of baseline comparisons (especially for the OOD task) makes it difficult to position the method.

> My main concern is the lack of quantitative assessment and comparisons on the CIFAR-10 OOD task.

The main contribution of our work is to show that the use of periodic activation functions in neural networks induces stationarity and establishes a connection between the prior distribution on the weights and the spectral density of the limiting stationary Gaussian process. Therefore, in the experiments, we focused on assessing the implications of the inductive bias (stationarity) of neural networks with periodic activation functions in the OOD settings and for density estimation. Our experiments intentionally do not aim at comparing the performance of a specific Bayesian neural network architecture with periodic activation functions to other models for OOD. Instead, we aim to show that the use of periodic activation functions empirically helps in OOD settings and allows for accurate density estimation compared to activation functions that do not induce global stationarity, such as ReLU activations. Moreover, the performance of Bayesian neural networks is heavily influenced by the inference algorithm; we refer to recent works on Bayesian inference (e.g. [27,42]) for details. Therefore, we have decided to systematically compare the induced biases under equivalent settings (same architecture and same inference algorithm) instead of a comparison to existing OOD methods.

### References
[27] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 13132–13143. Curran Associates, Inc., 2019.

[42] H. Ritter, A. Botev, and D. Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018.


 We thank the reviewer for the review and the comments on our work.

> One question that follows from this type of approaches is whether the use of the periodic activations makes the training of the network harder?including inductive biases

In our experiments, we observed that sometimes training neural networks with periodic activation functions can be a bit harder compared to ReLU networks, but usually, slightly adjusting the learning rate is enough to solve these issues. Other than this, we did not encounter any systematic difficulties compared to ReLU when training networks with periodic activation functions.


In this paper, the authors present the effect of using periodic activation functions in bayesian neural networks (BNN). More precisely, they show a connection between the priors on the network weights and translation invariance stationary Gaussian Process priors.
The authors show how this applies not only for sinusoidal, but also triangular and ReLU based periodic activation functions. 
Empirically, they demonstrate how this architecture choice still recovers good in-distribution behavior, while improving the out-of-distribution sensitivity. 

  One significant research area in machine learning is the one of model robustness. In the context of bayesian neural networks, we are interested in the modeling assumptions that can be introduced as priors, as well as the posterior predictive behavior.
Unfortunately, the choice of activation functions have a significant impact on the sensitivity of the model.

In this paper, the authors show how certain periodic activation functions in connection with the Student-t distribution on the weights, can induce global stationarity. In particular, the look at sinusoidal, sine-cosine, triangle wave, and, periodic ReLU activations.

The authors show in synthetic datasets how the posterior predictive density behaves with regards to the distribution of the data and how the uncertainty increases far from the distribution. On real datasets, the results are also promising especially when comparing against distribution shifts and OOD tests.

One question that follows from this type of approaches is whether the use of the periodic activations makes the training of the network harder?
In any case, the paper is well-written, technically sound and it provides an interesting stepping stone in model robustness.
 Not discussed, but not a significant issue here.

The paper utilizes periodic activation functions in concert with BNNs to induce global stationarity. Most of the paper is devoted to explaining and deriving the connection between the priors of the BNN parameters (+ activations) and the GP's kernel. The paper additionally assesses the impact of global stationarity on in- and out-of-distribution examples.  Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

The method combines/links several methods (BNNs, periodic activations, GPs, etc).
The combinations and connections appears to be novel.
The paper is well-cited.


Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

Most of the paper is spent explaining and developing global stationarity. The analysis seems correct though I did not review the derivations in the Appendices. The experimental results show comparable performance to standard methods while better informing model uncertainties. The toy, UCI, and MNIST experiments are good illustrations of the impact of the method. Figure 5 is especially nice for demonstrating model performance under distribution shift.

CIFAR-10 vs similar datasets are a standard test for OOD assessment. The paper includes some of the standard OOD datasets for comparison (CIFAR-100 and SVHN) but does not provide any quantitative comparison which makes it difficult to compare to other OOD methods. While Figure 6 provides an encouraging qualitative comparison of the datasets, the lack of (or justification for the lack of) quantitative comparisons is strange. I would like to see some combination of AUROC, AUPR, etc. from whatever the OOD detection criteria is (predictive entropy/marginal variance?).


Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

The development and writing make sense and are accessible. Clarification of the model architectures would be helpful. From the Appendix, the more complex models appear to be a feature extractor followed by the stationarity-inducing FC layer(?). Overall, the supplemental helped to clarify a few confusing points.


Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

The authors make a good argument for global stationarity and develop how to accomplish it. They provide an interesting connection
between BNNs and GPs.

They do a reasonable job exploring their method and the results are encouraging. However, the lack of baseline comparisons (especially for the OOD task) makes it difficult to position the method.


Overall:

The paper is reasonably well-written and clear about its intentions. I appreciate the analysis and justification in how the activations and priors are chosen, instead of brute-force empirical evaluation. I think they have done a good job explaining, justifying, and developing their ideas. My main concern is the lack of quantitative assessment and comparisons on the CIFAR-10 OOD task.
 N/A

The paper shows that periodic activation functions in the last hidden layer of a Bayesian neural network are related to stationary Gaussian process priors. With the periodic activation functions Bayesian inference correctly produced less confident predictions at out-of-distribution test data.  The proposed usage of the periodic activation functions in the last hidden layer of a neural network is practical.
The obtained results are comparable to ReLU networks and are correctly less confident on out-of-distribution test data.
The usage with KFAC Laplace does not require a special network training to perform the approximate Bayesian inference.

Clarity: The paper is well written. The experiments are well explained.

Quality: The theory is well explained and numerically verified in Figure 4.

Originality: The Gaussian process literature is aware of the benefits of stationary kernels. Previously, Meronen (NeurIPS 2020) proposed an activation function for a locally stationary model. The current paper goes one step further, proposing activation functions for global stationarity.

Significance: The paper is practical and relevant to the commonly used deep neural networks.


A question for the authors:
Do you see benefits from the periodic activation functions on OOD data only if using the Bayesian inference?
It would be informative to compare the Bayesian inference to NLPD results on the Rotated MNIST when using only the mode of the parameters.

Minor comments to help to improve clarity:
- Define the BNN abbreviation before using it in Figure 1.
- Line 94: Define "x" and define early that the E[...] expectations are over model functions.
  Make it clear that "x" and "x_i" have the same dimensionality.
  The paper helpfully mentions that the periodic activations functions required a relatively large number of hidden units.

Meta-review: The paper provides an interesting link between periodic activation functions, which have shown excellent empirical performance in fitting low-dimensional functions in models like SIREN, and stationarity in the infinite-width limit, connecting these networks with the Gaussian Process literature. All reviewers agreed that the contribution is interesting and novel, and many of the reviewers raised their scores after the author rebuttal. So long as the authors include the promised results in the camera-ready version, all reviewers were happy to recommend acceptance. I agree with their assessment.

Decision: accept