Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors present the effect of using periodic activation functions in bayesian neural networks (BNN). More precisely, they show a connection between the priors on the network weights and translation invariance stationary Gaussian Process priors.
The authors show how this applies not only for sinusoidal, but also triangular and ReLU based periodic activation functions. 
Empirically, they demonstrate how this architecture choice still recovers good in-distribution behavior, while improving the out-of-distribution sensitivity. 


main_review: One significant research area in machine learning is the one of model robustness. In the context of bayesian neural networks, we are interested in the modeling assumptions that can be introduced as priors, as well as the posterior predictive behavior.
Unfortunately, the choice of activation functions have a significant impact on the sensitivity of the model.

In this paper, the authors show how certain periodic activation functions in connection with the Student-t distribution on the weights, can induce global stationarity. In particular, the look at sinusoidal, sine-cosine, triangle wave, and, periodic ReLU activations.

The authors show in synthetic datasets how the posterior predictive density behaves with regards to the distribution of the data and how the uncertainty increases far from the distribution. On real datasets, the results are also promising especially when comparing against distribution shifts and OOD tests.

One question that follows from this type of approaches is whether the use of the periodic activations makes the training of the network harder?
In any case, the paper is well-written, technically sound and it provides an interesting stepping stone in model robustness.

limitations_and_societal_impact: Not discussed, but not a significant issue here.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper utilizes periodic activation functions in concert with BNNs to induce global stationarity. Most of the paper is devoted to explaining and deriving the connection between the priors of the BNN parameters (+ activations) and the GP's kernel. The paper additionally assesses the impact of global stationarity on in- and out-of-distribution examples.
main_review: Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

The method combines/links several methods (BNNs, periodic activations, GPs, etc).
The combinations and connections appears to be novel.
The paper is well-cited.


Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

Most of the paper is spent explaining and developing global stationarity. The analysis seems correct though I did not review the derivations in the Appendices. The experimental results show comparable performance to standard methods while better informing model uncertainties. The toy, UCI, and MNIST experiments are good illustrations of the impact of the method. Figure 5 is especially nice for demonstrating model performance under distribution shift.

CIFAR-10 vs similar datasets are a standard test for OOD assessment. The paper includes some of the standard OOD datasets for comparison (CIFAR-100 and SVHN) but does not provide any quantitative comparison which makes it difficult to compare to other OOD methods. While Figure 6 provides an encouraging qualitative comparison of the datasets, the lack of (or justification for the lack of) quantitative comparisons is strange. I would like to see some combination of AUROC, AUPR, etc. from whatever the OOD detection criteria is (predictive entropy/marginal variance?).


Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

The development and writing make sense and are accessible. Clarification of the model architectures would be helpful. From the Appendix, the more complex models appear to be a feature extractor followed by the stationarity-inducing FC layer(?). Overall, the supplemental helped to clarify a few confusing points.


Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

The authors make a good argument for global stationarity and develop how to accomplish it. They provide an interesting connection
between BNNs and GPs.

They do a reasonable job exploring their method and the results are encouraging. However, the lack of baseline comparisons (especially for the OOD task) makes it difficult to position the method.


Overall:

The paper is reasonably well-written and clear about its intentions. I appreciate the analysis and justification in how the activations and priors are chosen, instead of brute-force empirical evaluation. I think they have done a good job explaining, justifying, and developing their ideas. My main concern is the lack of quantitative assessment and comparisons on the CIFAR-10 OOD task.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Update: I have read the author feedback and am quite satisfied with it and thus raised my score.

******

This paper relates to Bayesian Neural Networks (BNN). It proposes to use periodic activation functions in order to deal with out-of-domain (OOD) samples and uncertainty. Series of experiments hint that with the method OOD performance is improved while in-domain performance is not worse than with the baseline methods.
main_review: The problem of OOD uncertainty for neural networks is an important one and this paper describes an interesting method towards solving this.

The theoretical part seems original, especially related to exploring different kinds of periodic activation functions in detail.

The experimental part of the paper has some drawbacks. For example, it should have a quantitative comparison to other OOD approaches, since that is the main focus of the method.

Table 1 states that globally stationary model gives improved NLPD for most datasets. Could the authors explain how these 4 datasets were chosen and how do the methods’ performance change if some other datasets, e.g., from UCI are chosen.  Also, while the statement of improved perf is true  when compared to relu, the difference to locally stationary models is lower especially given that different activations seem to suit different datasets. Since these datasets are not explicitly dealing with OOD conditions, it would be enough to demonstrate similar performance to locally stationary activations and similar improvement over relu, not consistently improved results.

In Fig.1 the locally stationary activation function BNNs do not seem to reach the performance of the GP counterparts, while this seems not to hold for globally stationary ones. Could the authors explain this behaviour?

The main problem with the paper is that quantitative evaluation of the main benefit of the method, namely proper uncertainty for OOD samples is too thin. Most of the OOD results in the paper require qualitative assessment and I think this is unacceptable given OOD is the main focus of the method.

In Fig. 5, the rotated MNIST experiment seems to hint at overconfidence with the ReLU baseline model, although it would be good to quantify this overconfidence. Also, it seems that in this case both locally and globally stationary models seem to work as well in this OOD case, while the main claim of this paper is that the globally stationary activations should lead to better performance in OOD cases.

Fig. 6 seems to provide the most important experimental results of the paper related to deep learning and OOD. Unfortunately the main paper only shows single model results and while the supplement shows the same results for all models, the results require interpretation.  It would be better to have quantified results that compare locally and globally stationary and baseline models. Could the OOD detection task be quantified into a single measure and then the different methods could be easily compared in the main paper.  

Overall, decent paper, but would require more work in the results section.
limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper shows that periodic activation functions in the last hidden layer of a Bayesian neural network are related to stationary Gaussian process priors. With the periodic activation functions Bayesian inference correctly produced less confident predictions at out-of-distribution test data.
main_review: The proposed usage of the periodic activation functions in the last hidden layer of a neural network is practical.
The obtained results are comparable to ReLU networks and are correctly less confident on out-of-distribution test data.
The usage with KFAC Laplace does not require a special network training to perform the approximate Bayesian inference.

Clarity: The paper is well written. The experiments are well explained.

Quality: The theory is well explained and numerically verified in Figure 4.

Originality: The Gaussian process literature is aware of the benefits of stationary kernels. Previously, Meronen (NeurIPS 2020) proposed an activation function for a locally stationary model. The current paper goes one step further, proposing activation functions for global stationarity.

Significance: The paper is practical and relevant to the commonly used deep neural networks.


A question for the authors:
Do you see benefits from the periodic activation functions on OOD data only if using the Bayesian inference?
It would be informative to compare the Bayesian inference to NLPD results on the Rotated MNIST when using only the mode of the parameters.

Minor comments to help to improve clarity:
- Define the BNN abbreviation before using it in Figure 1.
- Line 94: Define "x" and define early that the E[...] expectations are over model functions.
  Make it clear that "x" and "x_i" have the same dimensionality.
 
limitations_and_societal_impact: The paper helpfully mentions that the periodic activations functions required a relatively large number of hidden units.
review_title: null

Meta-review: The paper provides an interesting link between periodic activation functions, which have shown excellent empirical performance in fitting low-dimensional functions in models like SIREN, and stationarity in the infinite-width limit, connecting these networks with the Gaussian Process literature. All reviewers agreed that the contribution is interesting and novel, and many of the reviewers raised their scores after the author rebuttal. So long as the authors include the promised results in the camera-ready version, all reviewers were happy to recommend acceptance. I agree with their assessment.
Decision: Accept (Poster)