Implicit Neural Representation Inference for Low-Dimensional Bayesian Deep Learning
Authors: Panagiotis Dimitrakopoulos
Giorgos Sfikas
Christophoros Nikou
Venue: ICLR 2024 poster (2024)
Keywords: Bayesian Deep Learning
Implicit neural representations
Probabilistic machine learning
Hypernetworks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Bayesian inference is the standard for providing full predictive distributions with well calibrated uncertainty estimates.
	However, scaling to a modern, overparameterized deep learning setting 
	typically comes at the cost of severe and restrictive approximations, sacrificing model predictive strength.
	With our approach, we factor model parameters as a function of deterministic and probabilistic components;
	the model is solved by combining maximum a posteriori estimation of the former,
	with inference over a low-dimensional, Implicit Neural Representation of the latter.
	This results in a solution that combines both predictive accuracy and good calibration,
	as it entails inducing stochasticity over the full set of model weights while being comparatively cheap to compute.
	Experimentally, our approach compares favorably to the state of the art,
	including much more expensive methods as well as less expressive posterior approximations over full network parameters.

Published as a conference paper at ICLR 2024
IMPLICIT NEURAL REPRESENTATION INFERENCE FOR LOW-DIMENSIONAL BAYESIAN DEEP LEARNING
Panagiotis Dimitrakopoulos' Giorgos Sfikas" & Christophoros Nikou Dept. of Computer Science & Engineeringg University of loannina, loannina Greece Dept. of Surveying Geoinformatics, University of West Attica, Athens Greece [(p www.miiaaaooooolooooooo @ gr, "gsfikas@uniwa.c
ABSTRACT
Bayesian inference is the standard for providing full predictive distributions with well calibrated uncertainty estimates. However, scaling a modern, overparame- terized deep learning setting typically comes at the cost of severe and restrictive approximations, sacrificing model predictive strength. With approach, we fac- tor model parameters as function of deterministic and probabilistic components; the model solved by combining maximum a posteriori estimation of the former, with inference over a fow-dimensional, Implicit Neural Representation of the latter. This results in a solution that combines both predictive accuracy and good calibra- tion, as it entails inducing stochasticity over the full set of model weights while being comparatively cheap to compute. Experimentally, our approach compares favorably to the state of the art, including much more expensive methods as well as less expressive posterior approximations over full network parameters.
INTRODUCTION
Bayesian Neural Networks (BNNs) a class of models that propose elegant solutions to the patholo- gies of standard NNs (Ritter et al. 2018a Jospin et al., 2022 Gawlikowski et al.) (023). In BNNs, model parameters are defined random variables that follow a prior (posterior) distribution, which encodes knowledge about the model before (after) having "seen" the training data. Learning is cast as an inference problem, where the task is to compute efficiently the posterior distribution. turn, mak- ing predictions on new data is replaced by computing a predictive distribution. Advantages include that uncertainty estimates are calibrated and robust, and hyperparameter estimation can be performed through a principled evidence maximization framework. In BNNs, Bayesian inference is not exact, and direct application of Bayes' law leads an intractable computation. An approximation has to be applied, and in this respect numerous solutions have been proposed. A factor that complicates this problem is that the approximation must lead to a scalable, practical implementation that must take into account that the data and model size may be far larger than what was the norm in methods and models that dominated Bayesian inference in the pre-deep learning era. Several solutions have been proposed in this respect, rehashing and adapting older methods (Betancourt 2017 Daxberger et al. 2021a) or putting forward completely fresh approaches (Maddox et 2019).
Implicit Neural Representations (INRs) related to different line of research that is orthogonal to that involving Bayesian networks (Sitzmann et al. 2020 Dupont et al.] 2021b). With INRs, the goal is represent a signal in terms of a trained neural network. Unlike standard representations discrete sets of values over a canonical grid, an INR accepts continuous coordinates as inputs. Therefore, the INRs allow for a continuous representation, with the underlying NN providing values of the represented signal at theoretically any granularity. Related breakthroughs in improving representation of high frequencies have contributed to the popularity of the approach (Sitzmann al., 2020, Mildenhall et al., 2021). Numerous signal representation use-cases have been explored, including images, video, 3D shapes or Neural Radiance Fields (NeRFs). With the latter, NN tasked with mapping ray position and direction to color and density values. Part of the parameters of a larger NN can also be encoded with an INR; in Romero et al. (2021a), convolutional kernels are represented in terms of Multiplicative Anisotropic Gabor Networks. In this case, the implicit representation allows for kernels that generalize well to higher resolutions than the ones originally trained with. Aside from allowing for continuous representation at multiple scales, another major focus involves the

---

Published as a conference paper at ICLR 2024
INR's s capability of producing compressed, low-dimensional representation (Benbarka et al. Strümpler et al. 2022).
2022
In this work, we propose class of Bayesian Neural Network is parameterized using combination of deterministic and stochastic parameters. In recent work, similar partitions are employed (Kristiadi al. 2020 Dusenberry al. 2020 Daxberger et al., 202Tb), where a specific subnetwork is set be stochastic while the rest of the network is deterministic. Unlike these works, we define all parameters as functions conditioned over both deterministic and probabílistic components. Normally, this is very much desired but computationally prohibitive due to the huge number of parameters in modern NNs; in our work, this is made feasible due to the probabilistic component being parameterized through an INR hypernetwork, which compresses probabilistic factors through a -iweenmnsiooal SIREN representation (Sitzmann et al.. 2020). It is over this representation that we assume prior distribution, and perform inference. As the number of prohabilistic factors is kept low, we are allowed to make fewer concessions w.r.t. constraining the form of the posterior and the predictive. The result is process that is comparatively closer to exact inference, leading to more accurate estimates and better uncertainty calibration.
In nutshell, the deterministic model component is responsible for ensuring accurate results, while the low-dimensional probabilisic component is responsible for inducing stochasticity to the entirety of the network. We validate our claims and model across variety of experimental trials, where we show that our model produces accurate and well-calibrated uncertainty estimates.
2 BACKGROUND AND MOTIVATION
We consider supervised learning setting, where we have a training dataset D = (X,Y), with inputs X = {In |== and outputs Y = {um) ) and we define a mapping 9u: : + y. where X' and y are the input and output domains respectively. This mapping is modelled as a NN with parameters (weights and biases) w E _ Under the BNN paradigm, we assume that the mapping parameters are probabilistic, so we can say that they follow some prior distribution p(w), while we aim to compute (in practice, estimate) their posterior distribution p(w) DD) Given the posterior distribution, we can then opt to find the predictive distribution for some unseen datum x², formally:
p plylD, = plylow(x") (x))) (ppp)p((ddu))
(1)
In contrast to a Maximum a Posteriori (MAP) solution, which would optimize a cost combining log-likelihood and log-prior terms:
w arg max maxx[loggei p(ylg (x)) + logp(w)),
w
a Bayesian solution aims to compute distributions for both the posterior and the predictive.
Several options are available to proceed. The Stochastic Weight Averaging- Gaussian method (SWAG) assumes Gaussian posterior for the weights, with the distribution mean and covariance approximated as a function of the objective optimization method (Stochastic Gradient Descent) with a modified learning schedule (Maddox al.) 2019). Laplace Approximation (LA) also assumes Gaussian distributed parameters, with a precision matrix that is computed as the negative Hessian of the loss. After having weight posterior, an option can be to sample the predictive distribution and either obtain point estimates for test data, or perform Bayesian model averaging (Maddox et al. 2019). Additional simplifying assumptions can lead closed form also for the predictive. The Generalized Gauss-Newton approximation is closely related to linearizing assumption for the output layer of the NN (Immer et al.) 2021). which conveniently leads a Gaussian approximation for the predictive distribution. The covariance of the predictive is then dependent on combination of two factors: the covariance of the posterior (negative loss Hessian in LA) and the Jacobian for the specific point. Interestingly, a relation between the subspace spanned by the SGD trajectory vectors (used by SWAG) and the corresponding one to the most important eigenvectors of the Hessian (used by LA) is discussed in Gur-Ari et al. (2018). Normalizing Flows (NFs) represent powerful framework for density estimation (Dinh et al. 2016), that may in principle also be used to model the posterior of a large NN.
Scalability is crucial factor when it comes to learning methods in the context of NNs. Assuming an entire network to be probabilistic implies significant overhead in terms of various factors. Common
2

---

Published as a conference paper at ICLR 2024
remedies include assuming Gaussian form combined with low-rank approximation of the Hessian, and using a simplified, even diagonal covariance structure. Soonecker-Factore Approximate Cur- vature (KFAC) expresses useful tradeoff, which neglects only cross-layer correlations and uses block-diagonal covariance matrix (Ritter et al., 2018b). Another option involves treating only part of the network non-deterministic (Kristiadi et al. 2020, Daxberger et al. 202Tb). We then have uncertainty only in the last layer neurons, treating the rest of the network as feature extractor (Weber) al. 2018). As a consequence, and the degree that these assumptions are overly simplistic, the approximate distributions may turn out to be very far from the actual posterior and predictive. This often translates to a dramatic reduction of predictive strength in practice.
In the following Section, we shall discuss our approach to dealing with these issues.
3 PROPOSED MODEL: LOW-DIMENSIONAL BAYESIAN DEEP LEARNING
We propose to move from the high-dimensional setting of full inference in a modern Neural Network to low-dimensional inference, by assuming auxiliary Implicit Neural Representation alongside the main network. We perform density estimation over the parameters of the INR hypernetwork, while treating the factors corresponding the original weights as deterministic parameters. This allows us to employ powerful inference methods (we discuss LA, SWAG, NFs) with minimal approximation concessions, by leveraging on the small size and representational strength of the INR.
3.1
INR MODELING
Given the NN that models the mapping Iw (cf. Section 2), the first step of our approach is to augment each weight w with multiplicative nuisance factor & (Srivastava et al., 2014, Kingma al. 2015 Louizos & Welling 2017). In particular, we use w E. where o is point-wise multiplication, and the dimensionality of is identical that of w.
The & factor parameterized using an INR (Dupont et al. 2022), obtained as the output of a function
SwINK - IR, where tensor coordinates (domain 7) are mapped to layer values. More specifically for a convolutional main network, the INR hypernetwork learns a mapping from a 5 dimensional I to a scalar value which corresponds the nuisance factor associated with the weight o, A located at the kernel position k; ky at channel c of filter in layer 1 of the main/primary network (in the case of culle-co layer, dimensions k; and k- are omitted). With the above modeling choice, the hypernetwork can be easily shared across each layer of the main network and reduce the overall modeling complexity. The architecture of the INR defined as a multi-layer perceptron with sinusoidal activations, with the SIREN model of Sitzmann al. (2020). Formally, the input vector 2 for layer is transformed according to 2₁-1 + sin (Wo( W 2 + bf b) where w/ ni - by N denote weights and biases of the INR layer i, and wp a fixed hyperparameter.
In INRs, any target quantity can be modelled regardless of its size, while in traditional networks parameter size coupled with target dimensionality. This characteristic, in combination with the stochastic character of & allows us choose the complexity of SwiNR(( (.) to be (much) lower than that of its target (dwINR a < de). Thus, WINR parameters can also be interpreted as a fow-dimensional representation of factors E.
3.2 BAYESIAN INFERENCE
In our method, we treat the product w & stochastic random variable coming from a parametric distribution p(w, 5) = p(w)pp ((E). Here we are taking advantage of the INR hypernetwork modeling of { and implicitly place prior over those variables, by defining a prior over the INR parameters w N R- This allows us to reason about but in the much lower dimensional space of WINN- Following the supervised learning setting of Section 2 our aim remains to compute the posterior p(w, WIN D). Since the posterior distribution cannot be obtained in closed form, we cannot apply exact inference methods. Thus we resort to approximate inference, under an additional assumption that we only require deterministic estimate approximate posterior distributions q(w) and N where g (ww) = 8(w (ww--e S(w-w), and 6(-) is the Dirac delta function. This forces to be deterministic, equal point estimate The full approximate
w. We encode this constraint a factorization over separate
3

---

Published as a conference paper at ICLR 2024
posterior is then written
p w, WInnlD) se q(w, - = ((( - w).
(3)
Laplace Approximation. One way to proceed is by constructing Laplace approximation ((WTNR) We approximate p(WrNRl D) by
((WINR) = ((WINN.A-))
(4)
N
== A
V2 WINR log
WINR:
(5)
where we have assumed a prior WINR ~ No, C). Mean WINa is found as the Maximum a Posteriori
solution (eq.2) (p. Under this scheme, q(w, WINN) is expressed by a product of a Gaussian and Dirac delta distribution, which can be seen alternatively as single Gaussian distribution with precision w and zero covariance between and WINR terms by assumption (eq. 3 Concerning the weights and biases that directly parameterize the "main" network {), we note that these are in general non Gaussian,, even under LA assumptions. The INR fwrwn (-) transforms the (approximately) Gaussian WINR into a non- Gau-Gaus density g(5). This is multiplied by deterministic w where the result follows a density that is scaled version of (()). The first and second moments are equal to (E{{} and (V{ where W diag [w] and { V(-} denote expectation and covariance respectively. Once we have computed posterior over the weights, we can estimate the predictive (eq. by acquiring & samples by first sampling WINR ~ q(WINR) and evaluating { = fwINER (-). We finally scale them by w, then the product is used compute (zx) and p(ylg(z)) in Monte Carlo fashion.
Y +00 for variates corresponding
(i.e. the product
Alternatively, the predictive distribution (eq. D can be computed in closed form, as long as we impose linearizing assumption over the network output. Specifically, this involves first-order Taylor expansion of network output g(-) around WINd- As by LA assumption, parameters w linear transformation over them through linearization would result in a Gaussian predictive as well; linearization over other variables (w, {) would not have been fruitful due their being non-Gaussian. Hence, only require parameters WINR R vary in this approximation, while we assume the rest of the parameters w to be constant at their MAP solution. Formally we write:
a posteriori crussian-distribune
gim Jo.WINg (x) + JWINA (F)( WINR),
(6)
where used Juixn (I) 8go (INR (x) OWINR
WINR- For the predictive then have:
= -(to ) (oonn(x")) (x*)A~ -wwNN ( (x")).
Stochastic Weight Averaging. An alternative over LA use SWAG (Maddox al. 2019) over INR parameters. In this context, this amounts to approximating p( N(WENRll D) by a Gaussian q( ((WINR) as in eq. but with inverse equal the sample covariance over the SGD trajectory:
T
A
1
( WINR WINR)(W WINR)",
T
where {WINR- w WINR
w WINRi are training updates of INR parameters. The predictive distribution estimated by Bayesian model averaging through Monte Carlo sampling. Formally we have:
(2)
(T)
K
Ply
Σ E(vvgaa.c(a"))
K
where K samples (E1, .2...., } are drawn from the approximate posterior q(E) by evaluating WINR q(WINR) described in the previous paragraph.
Normalizing Flows. Normalizing Flows are another poweful modeling choice for q(WINR). In this context, ((WNNR) is freed from the Gaussian restriction and can be any parameterized flexible parametric distribution. A normalizing flow transforms an initial random variable 2, typically sampled from standard Normal, by applying chain of invertible parameterized transformations.
4

---

Published as a conference paper at ICLR 2024
The RealNVP model (Dinh et al. 2016) is based on flow composed of series of affine coupling layers defined as: y - m 2;-1 stand for scale and translation, which are typical linear mappings, while m is channel-wise masking scheme. The flow parameters can be computed by directly optimizing the variational lower hound:
m. o (2i-1 exp(s(m o 24-1) + t(m 2ς-1)) where and t
Lw,, WINR) = (g(wera) log ply) Iw Sw,wiNa (2"))
(10)
where the carefully designed coupling layers ensure that the inverse and the Jacobian of the determi- nant of each transformation can be efficiently computed. The predictive distribution is estimated by Bayesian model averaging through Monte Carlo sampling similar to eq
Table I:
Numerical results for classification on CIFARIO (top) and Corrupted CIFARIO (bottom) for different design choices. Likeliidl (1) and Expected Calibration Error (4) are reported.
Modeling w(
Noise Structure Chunnel
Type of INR Individual
Noise Type Add
Activation Type
ç
Rank-1
Full
Shared
Mult
ReLU
Sine
LL ECE
-1.29
-0.37
-0.44
-0.47
-0.40
-0.37
-0.29
-0.37
-0.37
-5.28
-0.289
-0.287
0.01
0.05
0.06
0.06
0.05
0.05
0.06
0.05
0.05
0.19
0.032
0.034
11.
-0.97
-1.43
-1.18
-0.97
00.77
-6.29
-0.50
-0.47
ECE
0.17
0.11
0.20
0.20
0.15
0.11
0.10
0.11
0.11
0.26
0.06
0.05
EXPERIMENTAL RESULTS
In this Section, we provide numerical results for the proposed INR-based scheme, in comparison to recent Bayesian inference methods. Namely, we compare ourselves versus the following methods: MC Dropout (Gal & Ghahramani] 2016); Bayes by Hypernet (BbH) (Pawlowski et al., 2017 7) Deep Ensembles (Lakshminarayanan et al. 2017) considered among the state-of-the-arr methods for uncertainty estimation in Deep Learning (Ovadia et al.] 2019 Ashukha et al. (020) and last layer Laplace approximation (LL). We start by experimenting with different types of modeling choices and evaluate each one on bascline classification task, in order to quantify how our method performs under different modeling scenarios. For our main numerical analysis, we deployed three different experimental setups. First, we evaluate our predictive uncertainties for our method on a 1D synthetic regression task. We carried out experiments regression UCI datasets. Last, we ran image classification trials (CIFAR 100, CIFARIO 10 and MNIST) where we compare ResNet variants for prediction and diitri--iittittiionn robustness. We test three variants of the proposed INR- based model, namely INR- INR-Laplace (eq. 4577. INR-SWAG (eq. ( and NNR-ReeINVP (eq. 1O). The three variants differ w.r.t. the approximation strategy for the posterior and the predictive (cf. Section For the first two cases we compute the full Gaussian covariance for the weight posterior (avoiding e-g. KFAC or low-rank approximations (Daxberger et al. aa). Throughout our experiments, we found that the proposed model provides good predictive uncertainties on a variety of settings, highlighting the benefits of low-dimensional Bayesian inference. Concerning implementation details of the proposed and compared models and benchmark setup in general, we have moved additional information to the Appendix (App. B).
evaluate INR performance on different types of
4.1 DESIGN CHOICES
In this Section we carry out ablation studies that justify the particular modeling and INR architecture described in subsection 3.1] and help us understand the behavior of the hypernetwork under different settings. We numerically evaluate each different potential modeling scenario by training ResNet-20 model at CIFAR- 10 according to subsection(4.4 and and evaluate its MAP solution in both in and out of distribution data. Table inncludes the main results.
Our first ablation study aims to justify the introduction and use of & variables i.e. we investigate how the BNNs perform with only the INR for the posterior (see Table [under Jnder the column "Modeling"). As the & variables only serve to induce stochasticity, removing weights result in model which is not able to capture any information from the training data. Furthermore, augmenting w with g results in more sophisticated model which yields better calibrated predictions. We choose the INR hypernetwork to be shared across all the layers of the main network. Sharing the INR hypernetwork, besides being efficient, can also reduce significantly the dimensionality of WIN nt as the total d(WNNR)
5

---

Published as a conference paper at ICLR 2024
for the individual hypernetwork will by a multiple of the number of layers of the main network. As an example, for Wide-ResNett the magnitude of this figure can be up to hundreds of variates. Despite having less parameters, the shared version of the hypernetwork is highly comparable to its more expensive counterpart as Table (columm labelled as "Type of INR") indicates. Also, we introduce independent nuisance factors ş for every single weight In Table T (column labelled as "Noise Structure") we measure the benefits of our full-rank multiplicative noise versus other low-rank modeling options used in related works (Dusenberry same Table (column labelled as "Noise Type"), we can see results for evaluation of two different types of noise injection the main model, namely multiplicative noise ("Mult"" and additive noise ("Add"). The additive noise hugely underperforms where multiplicative noise factors seem to provide good and calibrated solutions. Because in the multiplicative structure during training Vg depends on W, we argue that as W is responsible for fitting the data, it can pass valuable information to the hypernetwork weights in the multiplicative case leading to significant increase in overall performance. Furthermore we find that Sine/Periodic activations the "default" choice in Sitzmann et al. (2020) slightly outperforms a hypernet with ReLU activations as we can see in Table (column labelled as "Activation Type"), even though results are still very close.
al. 2020; Louizos & Welling 2017). In the
Finally, we evaluate the effects of INR network size on uncertainty estimates. We want to measure how increasing the number of paramters of the hypernetwork will affect the predictive behavior of the model. We trained 3 different INR models, with increasing numbers of trainable parameters.
Following Fort al. 2019 and Dusenberry et al. 2020 in Figure we examine the normalized
INR 350
INR 4K
INR 10K
(L(CT)
(C)
10 Deversity (f)
Cocppted CIFAR 10
(f)
ECE (1)
Diversiy (t)
0.4
2.5
CIFAR 10
0.35
-1.05
D
-3.15
0.3
-1.1
0.5
1.5
-3.2
-1.2
SWAG
MAP SWAG
SWAG
MAP
SWAG
SWAG
SWAG
Figure 1: Comparison of Log-Likelihood (t), Expected Calibration Error (L) and Normalized Diversities between INR networks of increasing size, over CIFARIO and corrupted CIFARIO datasets. INR-z represents an INR with parameters.
diversity of INRs of increasing size, where the posterior over w o was estimated via INR-SWAG and INR-MAP. Increasing the size of the INR hypemeeworr results in more complex weight posteriors. which is depicted with better scores across all metrics in out-of cuttof-diisributtin data. Nevertheless, small INR with only 350 trainable parameters competitive this training setup.
4.2 VISUALIZING UNCERTAINTY
We use synthetic 1D regression task with three disjoint clusters of input data as proposed in Izmailov et al. (2020). This dataset is carefully designed to test "in-between" uncertainty, i.e. model
confidence in between these disjoint clusters of data (Foong et al. (011).. Ideally, want model to predict high uncertainty values as test data move away from the observed data. In this test, we
Dropout
Deep Ensembles
INR Laplace
Ganssian Processes
0.5
0.5
0.5
0.5
00.5
0.55
Visualization of the predictive distribution for the "toy" regression task. The data are denoted as purple circles, predictive mean is the solid orange line and the shaded region is + std.
Figure 2:
use a fully- connected architecture with hidden layers that have [200, 50, 50, 50) neurons respectively. Following Izmailov et al. (2020). the network takes two inputs & (ar, x² and outputs single

---

Published as a conference paper at ICLR 2024
real value y fā). The INR network has 3 layers consisting of [2. 10, 4] neurons respectively, resulting totally in 160 training parameters (equal to only 1% of the number of the & parameters, cf. Section Results are shown in Figure _ We also include Gaussian Process (GP) with Radial Basis Function (RBF) kernel as the state of the art for this problem. Our INR-Lap preserves more of the uncertainty regarding both "out" and "in-hetween" of the observed data. Other methods, like Deep Ensembles and MC Dropout infer desirable uncertainty structure but still remain quite overconfident. Furthermore, the proposed INR model is able to maintain the appealing characteristics of the approximate inference methods applied, specifically the stationary structure (or in netbee uncertainty) benefits of the Linearized Laplace approximation as shown in multiple works (Kristiadi) et al. 2020 Daxberger et al. 2021b).
4.3
UCI REGRESSION
We next test our method on the UCI regression tasks (Asuncion Newman 2007). We experiment evaluationnennsttttttttee splits from Hernandez- Adams (213)) and their GaP-variants (Foong et al., 2019). To measure performance we deployed Gaussian test log- likelihood (LL). Our training strategy follows the work of Daxberger et al. (202 (0).b).
with 8 UCI regression datasets using standard
The INR network has 4 layers consisting of [5, 5, 5, neurons respectively, resulting totally in 70
training parameters (equal to only 2% of the number of the & parameters, ef. Section
Doep Ensembees
RhH
Laplace
Laplace
INR RealNVP
Dropont
Laplace
Concrete
Energy
Power
-3.2
33
0.9
-36
-..1
0.95
3.7
-3.4
3.9
3.8
0.8
3.5
1.05
Gap
2Ap
Energy Gap
Power
Wine Giop
Yachr Gap
Kin8om Gap
Noval
-3.2
3.7
-3
+0..5
3.5
3.8
3.5
3.7
0.S
3.9
3.4
1.05
3.8
U.7
2.5
1.1
-3.5
3.6
3.9
Numerical results for regression trials on UCI datasets (Asuncion & Newman 2007). Mean values of test Likelihod (1) are shown with + I standard deviation error bars, obtained over standard (Hemándezz-Lobtoo Adams, 2015) and GAP (Foong et al. 2019) splits.
Figure 3:
The main results are depicted in Figure 3 GGN matrix in the Laplace approximation of the main network and add it as baseline. As we can see, INR combined with RealNVP or LA achieves better test log- ligeliiklihoodd - a metric which considers both uncertainty and accuracy compared to BbH and LL Laplace approximation, while followed closely by MC Dropout. Furthermore, the proposed INR remain competitive with Deep Ensembles networks, even surpassing them in five out of eight datasets while overall being close enough, in both standard and gap splits, as standard deviation bars indicate.
The small MLP network enabled us to compute the full
4.4 IMAGE CLASSIFICATION UNDER DISTRIBUTION SHIFT
We evaluate our method on standard image classification tasks over the CIFARIO, CIFARI00 (Krizhessky et al. 2009) datasets. We use ResNet-50 (He et al. 2016) in order to test the ability of proposed INR-based method scale into larger models. capable Bayesian inference technique is critical when applied in deep models, as they tend to exhibit less accurate calibration in this context Guo et al. 2017). We provide experiments in a context of high degree of distribution shift, as under these conditions the evaluation of predictive uncertainty is the most useful in practice (Ovadia et al.) 2019). Our INR hypernetwork (Sitzmann et al. 2020). has 4 layers with [10, 10, 10, neurons each, resulting in 260 training parameters (only 0.001% of the parameters E). Following Ovadia jet al.] (2019); Antorán et al. (2020), we train ResNet50 on CIFARIO/CIFA,
100 and evaluate on data subject to 16 different types of corruption with levels of noise intensity each (Hendrycks &
7

---

Published as a conference paper at ICLR 2024
Deep Ensembles
INR RealNVP
Dropout
LL Laplace
INR SWAG
Liror
0.9
0.3
CIFAR 100
0.8
0.25
-3
0.7
0.6
0.15
35
2
3
2
3
2
Figure 4: Numerical results for classification trials on Corrupted CIFARI00 dataset. The x--axis of each plot corresponds to increasingly corruption levels.
Dietterich 2018). As Fig. 4 indicated, one of the proposed variants, non-INR methods terms, log-likell eegliikelioodd and expected calibration error. Both INR-based methods outperform I.I. Laplace and MC Dropout which are overconfident in their predictions and more often erroneous while still being competitive w.r. Deep Ensembles. Overall, these results suggest that the proposed approach produces more calibrated and accurate models than other popular uncertainty quantification approaches.
outperforms
Deep Ensembiess
INK RezINVP
Dropout
LL Laplace
INR SWAG
MNIST vs FASHIOON
CIFARIO SVIN
SVENN
CIFARIOD SVHN
Accuncy
0.8
0.6
0.2
100
Rejected
Rejectod
%6 Rejested
Figure 5: CejeettinnCClasification plots
We quantify the quality of uncertainty estimates by jointly evaluating the predictive entropy of our model on an in-distribution and an OOD test set. Ideally, we want predictive entropy to be high on OOD data as predictions should be more uncertain, and vice versa. Following Antorán et al. (2020) Nadeem et al. (2009), we deployed the OOD rejection scenario by jointly evaluating the entropy of our model on an in an-distribution and OOD test set, where we allow the models to reject an increasing proportion of the data based solely on their entropy values. Ideally, we want highly calibrated and robust models to reject all the OOD examples, as well as the in-distributional examples when the corresponding predictions are inaccurate. Figure 5 iillustrates on what percentage of the remaining non-rejected examples the predictions are accurate. On CIFARIO-S have the same performance, while on CIFAR 100 the INR-ReaINVP model fails to distinguish very uncertain in-distribution data from low uncertainty OOD ones. On MNIST-Fashion the proposed methods INR-SWAG and -NR-ReeINVP perform best, followed by LL Laplace and Dropout.
all methods
Finally, we tried to measure the quality of proposed low- cow-dimensional spaces in terms of predictive uncertainty. Specifically, we compare our INR low dimensional space with: rank-1 (Dusenberry et al. 2020) Wasserstein subnetwork (Daxberger al. (021b) and partially stochastic Resnets from Sharma et al. (2023). We trained (each method) combined with a Resnet for 100 epochs in CIFAR 100 while keeping the approximate inference method the same across all low-dimensional spaces. Results in Table 2 show a trend in favor of both proposed INR-z methods and validate to a considerable degree the premise of our method: instead of choosing subset or subnet following the rationale of the corresponding methods, the INR produces & outputs that endow the full network with the desirable stochasticity, while keeping the dimensionality of the random process that we want to do inference upon at low level.
5 RELATED WORK
Hypernetworks. Hypemnetoorss are NNs that are used to predict deterministically the parameters of another, typically larger network, termed the "primary" network. The terminology is due to Ha et al.
8

---

Published as a conference paper at ICLR 2024
Table 2: Numerical results for classification trials CIFARI00 for different proposed low- dimensional spaces alongside their inference time.
Standard
Corputed
Subspacc
Inference
Time
LL
Error J
Brier
ECE
LL
Error J
Brier
ECB
Rank1
SWAG
-2.29
0.34
0.55
0.22
-4.77
0.57
0.92
0.39
0.28
Laplace
0.97
0.66
1.25
0.58
0.97
0.40
0.55
SWAG
-2.09
0.30
0.50
0.22
-4.18
0.53
0.84
0.36
0.11
INR
Laplace
3.91
0.30
0.96
0.67
4.119
0.58
0.97
0.39
0.51
SWAG
2.14
0.49
0.20
3.97
0.82
0.34
Subnetwork
Laplace
- 3.95
0.32
0.96
0.65
-413
0.51
0.97
0.46
0.42
Partially Stochastic
SWAG
-2.14
0.30
0.49
0.20
3.97
0.51
0.82
0.34
0.28
Laplace
-3.99
0.34
0.97
0.63
4.18
0.51
0.98
0.47
0.49
(2016), however the main idea can be traced back to earlier works (see discussion in c.g. Krueger et al. 2017. Karaletsos et 20,(20 Krueger et al. (2017) have been among the first to extend hypernetworks to Bayesian setting. Their Bayesian hypernetwork, modelled as a normalizing flow, learns to predict distributions of weights for the primary network. The flow predicts scaling per-neuron factors for the primary network weights. This is similar to the closely related (Louizos & Welling (017). which however require an extra inference network to estimate the entropy term of the VLB. Almost concurrently, Pawlowski et al. (2017) proposed BbH for VI with implicit distributions. They use a discriminator network for density ratio estimation (DRE) in the context of prior-constrastive VI (Huszár 2017), and generator to model the variational distribution. [hh al. (2017) use a kernel
method for DRE instead of a discriminator, Karaletsos et al.] (2018) Kara serralesoss.... & Buil (2020)
explore hierarchical prior modeling using NN-based implicit distributions and Gaussian processes.
INRs have also been used for approximating model parameters of deep NNs (Romero et al. 202 la b).
Low-Dimensional Inference. Bayesian inference in a low-dimensional space is another important related concept, with often considerable overlap to works that can be understood as forms of hyper- networks. Dusenberry al. 20200 in the spirit of Wen al. 22200) employ rank-1 multiplicative noise components, before attempting to estimate an approximate posterior over the weights. Izmailov et al. 2020)adopt post-hoc Bayesian inference by constructing subspace of the BNN weights. They apply high fidelity inference on these small subspaces. and were able to produce state-of-the at a moderately low computational cost. |Prddeer et al. 2018 learn a non-linear latent representation of network weights. Another subgroup of related work can be described as selecting portion of the BNN parameters to be treated as random variables, and leaving the rest of the model to work deterministically. One of the most popular and straightforward approaches are last last-layer BNNs. By selecting a priori only the last layer to have a probabilistic treatment, they resort to linear model which ensures analytical tractability of both inference and predictive distribution in the spirit of Gaussian processes, while the remaining NN structure acts as feature extractor (Watson et al. 2021 I Snock al., 2015 Lázaro-Gredilla & Figueiras- Vidal 2010; Weber et al. 2018). Finally, Daxberger et al. 202 first obtain a MAP estimate of all weights, then define subnetwork selected in way that aims to maximally preserve predictive uncertainty. The small size of the subnetwork allows for
results
the use of full-covariance Gaussian posterior in tandem with linearized LA (MacKay) 1992).
Stochastic INRs. INRs have been used as models for signal compression (Dupont et al., 2021a) la). and more recently they have been extended to the Variational Bayesian setting (Guo et al. (023). Shen et al. (202T) extend NeRFs to learning distributions of possible radiance fields. simple variational posterior is assumed, and the base model is extended to learn uncertainty estimates over scene parameters. Vasconcelos et al. (2022) use a BNN as an INR of computerized tomography.
CONCLUSION AND FUTURE WORK
We have presented an approach for scalable and efficient Bayesian Deep Learning, that leverages on the small size and representational strength of INRs. Our claims are corroborated by the reported experimental results, which show that the integration of the proposed method results in improving considerably overall uncertainty estimates. For future work, we aim at exploring other ways to integrate INRs (e.g. multiplicative filter networks (Fathony et al. (020)) as well as integrating with different types of approximations, such as Hamiltonian Monte Carlo (Neal al. 2011)
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Javier Antorán, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. Advances neural information processing systems, 33:10620-10634 2020.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov. and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling deep learning. arXiv preprint arXiv:2002.06477 2020.
Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
Nuri Benbarka, Timon Höfer, and Andreas Zell. Seeing implicit neural representations Fourier series. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 2041-2050, 2022.
Michael Betancourt. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint
17X1:0/701.02444 2017.
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089-20103 2021a.
Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antorán, and José Miguel Hernández- Lobato. Bayesian deep learning via subnetwork inference. In International Conference on Machine Learning, pp. 1551--2521. PMLR, 2021b.
Laurent Dinh, Jascha Dick-iic,,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.03123 2021a.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. arXiv preprint arXiv: 102.04776 2021
Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Inteonnti Conference on Machine Learning, pp. 1694-5722 PMLR, 2022.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp. 2782-2792. PMLR, 2020.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2020.
Andrew YK Foong, Yingzhen Li, José Miguel Hernández-Lobato. and Richard E Turner. 'in- between uncertainty in bayesian neural networks. arXiv preprint arXiv:1906. 11537, 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshninarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: 192.0777 2019.
Vincent Fortuin. Priors in bayesian deep learning: A review. International Statistical Review, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks, Artificial Intelligence Review, pp. 1-77, 2023.
10

---

Published as a conference paper at ICLR 2024
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 02677, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and José Miguel Hernández-Lobato, Com- pression with bayesian implicit neural representations. arXiv preprint arXiv:2305. 2023.
Guy Gur-Ari, Daniel A Roberts. and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint wrXiv:1812.04777 2018.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. CoRR, abs/ 01216669999100 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
José Miguel Hermández-Le Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning, pp. 1861-1869. PMLR, 2015.
Ferene Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: met:::002003333 2017.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization. In International Conference Artificial Intelligence and Statistics, pp. 703-711. PMLR, 2021.
Pavel Izmailov, Wesley Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace inference for bayesian deep learning. In Uncertainty in Artificial Intelligence, pp. 1 1169-11799 PMLR, 2020.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands-on bayesian neural networks: A tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29-48, 2022.
Theofanis Karaletsos and Thang D Bui. Hierarchical gaussian process priors for bayesian neural network weights. Advances in Neural Information Processing Systems, 33:17141-17152, 2020.
Theofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani. Probabilistic meta-representations of neural networks. arXiv preprint arXiv:1810.0055: 2018.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and local reparameterization trick. Advances in neural information processing systems, 28, 2015.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in ReLU networks. In International conference machine learning, pp. 5436- 5446. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Chin- Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 1eXiv:1710.. 170.00479,, 2017.
11

---

Published as a conference paper at ICLR 2024
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
2010. Miguel Lázaro-Gredilla and Aníbal R Figueiras- Vidal. Marginalized neural network mixtures for
large-scale regression. IEEE transactions neural networks,
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp. 2218-2227. PMLR, 2017.
David IC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):488-44) 4724 1992.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, (5(1):99-106 2021.
Malik Sajjad Ahmed Nadeem, Lean-Danied Zucker, and Blaise Hanezar. Accuracy-rejection curves (arcs) for comparing classification methods with reject option. In Machine Learning in Systems Biology, pp. 65-81. PMLR, 2009.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 2011.
Yaniv Ovadia. Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference Machine Learning, pp. 4979- PMLR, 2019.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1111.01997, 2017.
Melanie F Pradier, Weiwei Pan, Jiayu Yao, Soumya Ghosh, and Finale Doshi- Velez. Projected bnns: Avoiding weight-space pathologies by learning latent representations of neural network weights. arXiv preprint arXiv: 18/ 1.07066, 2018
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 8a.
Hippolyt Ritter, Aleksandar Botev, and David Barber. scalable laplace approximation for neural networks. In 6th International Conference on Leurning Representations, ICLR 2018- 1018-Cofference Track Proceedings, volume 6. International Conference on Representation Learning, 2018b.
David W Romero, Robert-Jan Bruintjes, Jakub M Tomezak, Erik J Bekkers, Mark Hoogendoorn, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059, 202
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub Mikolaj Tomezak. and Mark Hoogendoom. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2021b.
12

---

Published as a conference paper at ICLR 2024
Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In International Conference on Artificial Intelligence and Statistics, pp. 7692- PMLR, 2023.
Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francese Moreno-Noguer, Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 2021 International Conference on 3D Vision (3DV), pp. 972-981, IEEE, 2021.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. arXiv preprint arXiv: 10119 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Jasper Snoek, Oren Rippel. Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, 2171-2180. PMLR, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15())1999-11988 2014.
Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari. Implicit neural representations for image compression. In European Conference Computer Vision, pp. 74-91. Springer, 2022.
Francisca Vasconcelos, Bobby He, Nalini Singh, and Yee Whye Teh. Uncertainr: Uncertainty quantification of end-to-end implicit neural representations for computed tomography, arXiv preprint arXiv:2202. 10847, 2022.
Joe Watson, Jihao Andreas Lin, Pascal Klink, Joni Pajarinen, and Jan Peters. Latent derivative bayesian last layer networks. In International Conference on Artificial Intelligence and Statistics, pp. 1198-1206. PMLR, 2021.
Noah Weber, Janez Stare, Arpit Mittal, Roi Blanco, and Lluís Mårquez. Optimizing over bayesian last layer. In NeurIPS workshop Bayesian Deep Leurning, 2018.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach efficient ensemble and lifelong learning. arXiv preprint 15Xiv:2002.00715, 2020.
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Vedd Velez. Quality of uncertainty quantification bayesian neural network inference. arXiv preprint arXiv: 2019.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A INR HYPERNETWORK DETAILS
This section delves deeper into the INR hypernetwork. We analyze its functionality and provide graphical illustration of the process in Fig. Additionally, we present the training and inference procedures of our proposed low-dimensional inference scheme in two separate algorithms outlined in Figure
Algorithm Training procedure
Algorithm Inference procedure
Require: (Indices of main network weights),
Require: (Indices), Net (Main network), INR (INR hypernetwork), Testset Approximate Infer- ence (Approximate inference method) MC Sam- ples (Number of Monte Carlo samples).
Net (Main network), INR (INR hypernetwork), Dataset.
for each Epoch do
for (x, y) Dataser do
for in Testset do for j in range MC Samples do
y" = Ne)(2r
loss = (y-y")
E, = Approximate Inference(INR, I)
update INR w.r.t loss update w.r.t loss
y Net(z,E,)
end for end Calculate y statistics
end for end for
Figure 6: High level pseudo-code introduce method's behavior in training and inference settings (in this setting, a post-training Monte Carlo-based approximate inference method implied).
Main Network
&
INR Stochastic Hypernetwork
1
wia e &
14
(2) UINE
we (1)
19
E
PINR
Wis S &s
I
h
w., &
Figure 7: Illustration of the proposed INR model.
As for the weight coordinates 1, in practice these values are batched and computed separately for each layer. For the - th layer tttpee//inpu-coooddnee positions have the shape [n, Jasma] where n is the number of the total main network parameters of the th layer and Idime is the dimensionality of the indices. For example, for a convolutional main layer Iatim = 5, the first positions correspond to the kernel weights plus dimension act the layer position (conditional position for each layer).
14

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL SETUP
In this Section, we provide all the experimental details that were used in order to produce the results of the main paper. Our experimental setups and procedures are heavily influenced by current practices in the literature. In all experiments we chose the SIREN (Sitzmann al., 2020) network to serve as the INR hypernetwork, due its popularity and its ability to model highly complex signals without any use of positional encoding layers (for the coordinate tensors I). Furthermore, (Sizmaann et al. (2020) described a hypernetwork initialization scheme for the WIN R parameters, that results in values & that initially Normally distributed. This is in general beneficial for the training procedure and it is also common practice for initializing multiplicative noise (Kingma et al. 2015 Gal & Ghahramani
2016
B.1 DESIGN CHOICES.
For each different modeling scenario we trained a ResNet-20 (He et al., 2016) on the CIFARIO dataset. All models are trained using the Adam optimizer with learning rate equal to 10-3, weight decay equal to 10 6 and a batch size equal to 256 running for 100 epochs. We evaluated the MAP solution for each model in clear and cortupted test data. As for evaluating the size of the INR hypernetwork we also deployed SWAG. We used a full Gaussian covariance to approximate the distribution of WINR: and used 10 epochs of average with learning step of 0.01. We evaluated the effect of the increasing size of hypernetworks by using the I Log-likelihodd Expected Calibration Error and the Normalized Diversity. Concerning the latter, typical way quantify diversity is to compute the fraction of points where discrete predictions differ between two members, averaged over all possible pairs. This disagreement measure is normalized by (1 accuracy) of each prediction to take into account its sample predictive accuracy. Recent works (Pang et al.] [211 Fortt et al. (019)) point out that measuring the diversity of individual predictions obtained from each sampled network can highlight the quality of uncertainty.
B.2 VISUALIZING UNCERTAINTY.
This visualization task is highly suited to quantify "in-between" uncertainty of a model, as recent works found that standard numerical evaluation metrics such as log- likelihood struggle to fully capture this behavior, while at times overconfident methods may obtain better scores (Yao et al. 2019 Ashukha et al., 2020). We train a single, hidden layer network, with 50 hidden ReL.U units per layer using MAP inference until convergence. For the INR network fwiNe (:) we used a SIREN /Sizmann ct al. (2020) The INR network has 3 layers consisting of [2, 10, 4| neurons respectively, resulting totally in 160 training parameters. Concerning the hyperparameters we used Ω 30 for the first INR layer and Ω4 = for the rest while keeping the the parameter c = fixed for all layers. The INR weights WNNN are initialized uniformly as ~ U(-vc/nn00, / /c/n/2). The input coordinates e 2 are normalized to be in the range [-1, 1J3. We optimize the Gaussian log-likelihood of our data, where the mean is produced by the network and the variance is hyperparameter learnt jointly with NN parameters. We used a full batch Adam optimizer with a learning rate of 10 a = 0.9, = 0.999 and weight decay 10-4 We trained all models for 600 epochs (since the amount of training samples are less than the actual training parameters). We used the same strategy for all of the baselines. We deployed deep ensembles with an ensemble of 5 networks, as suggested by Ovadia et al. (201)). Dropout was set with dropping probability of 0.1. For the INR-ReaINVP. following the literature, we tempered the posterior by applying a weight on the Kullback-Leiberr term of the ELBO, equal to 0.1. For the INR equipped with the linearized Laplace we set the prior precision of X = 0.001, where C X 2 Idg. Methods that required Monte Carlo sampling for estimating the predictive distribution use 30 MC samples during testing and sample during training.
B.3 UCL.
We experiment with 8 UCI regression datasets using standard training- ciiiiing-vallattinn-eett splits from Hermández-Lobato & Adams (2013) and their gap versions (Foong et al. 2019). In this test, we use a fully-comnected architecture with hidden layers that have [50, 50, 20 neurons respectively followed by ReLU activation. All the training details are applied to all the regression datasets regardless of their individual characteristics such as size, input dimensions, etc. We used homoscedastie regression
15

---

Published as a conference paper at ICLR 2024
methods with global variance parameter, (y): Iw (rx)) where the logarithm of global log- variance logø" (in order to ensure positivity) is jointly optimized with the model parameters. Our training strategy follows Daxberger et al. (02 We trained all methods for 50 epochs, except INR- RealNVP, which needed approximately additional epochs to adapt. We employed early stopping if validation performance does not increase for 10 consecutive epochs. The weight settings which provide best validation performance in terms of log-likelihood are kept for testing. Again we used the Adam optimizer with learning rate equal to 10-3 weight decay equal 10-6, and batch size equal to 512 samples. For the INR equipped with SWAG, we used full Gaussian covariance to approximate the distribution of WINR. and used 25 epochs of average with a learning step of 0.01. For INR-Laplace, after trying several precision values we use prior with a precision value 0.005, as it yielded better validation results across all datasets.
B.4 IMAGE EXPERIMENTS.
Through our image experiments we deployed the ResNet50 architecture (He et al.) 2016). As is common practice, we applied several modifications to the original architecture such as replacing the kernel size of the first strided convolutional layer (7 x 7) to size 3 x 3. Additionally, we remove the first max- pax-pooling layer. The rest of the ResNet details were set according to Gooaa et al. (0177). For the INR network (.)) we used a SIREN (Sitzmann et al.] 2020) shared across each layer of the main network. We used variety of metrics, these include: test log- lgglikelihood (LL); Brier score (Brier et al.) 1950), which is a metric that measures accuracy of predictive probabilities by computing their mean squared distance from the one-hot class labels; the Expected Calibration Error (ECÉ,,Naeim et al. 2015), a metric which measures the difference between predictive confidence and empirical accuracy in classification. A detailed explanation of uncertainty evaluation metrics can be found in Antorán et al. (2020): AAshukha et al. (2020); Ovadia et al. (2019). In our experiments we emphasized -utto--distiibution performance, model that was well-calibrated on the training and validation distributions must ideally remain so on shifted data. Regarding the completely "out- of-distribution" (OOD) data, we expect the empirical entropy of the predicted distribution to be quite high. Essentially, a good model must be uncertain according to the degree that test inputs are far from the training distribution. For Dropout experiments, we add Dropout to the standard ResNet model in between the 2nd and 3rd convolutions in each ResNet block (Ashukha et al. 2020). We used an ensemble of elements for prediction. Ensemble elements differ from each other in their initialization, which is sampled from the He initialization distribution (He et al., 2013). All models are trained using the Adam optimizer with learning rate equal to 10 3 weight decay equal to 10 with batch size equal to 256 running for 50 epochs for MNIST and 150 epochs for the CIFARIO/CIFAL 100 experiments respectively. The weight settings which provide best validation performance in terms of likelihoddood are kept for testing. During training, also used plain data augmentation strategies including random image cropping and random horizontal flips. We used INR-SWAG for 10 epochs with learning rate equal to 10 For INR-ReaINVP, the base Gaussian distribution is set to .V (V( 0.1/), transformed with cascade of 4 coupling layers. Finally as for the experiments validating the uncertainty quality per low dimensional space we trained (each method) combined with a Resnet18 for 100 epochs in both Cifar 10 and Cifar100 datasets while keeping the approximate inference method fixed same across all low dimensional spaces. While for each subspace method we followed the hyperparameters proposed in the original papers, for SWAG and Linearized Laplace with GGN, in order to be able to run across low dimensional spaces we choose the covariance to have Diagonal structure. We used SWAG for 10 epochs with learning rate equal to 10-3 3 For the Laplace, we use a prior with a precision value À = 1.0. All hyperparameters stayed the same across each method for comparison. Inference time (Table 2 for Resnet18 combined with different stochastic subspaces and different approximate inference methods was measured in seconds and for batch of 10 CIFAR images.
C RELU AND SINUSOIDAL HYPERNETWORKS
This section delves deeper into the activation function used in the hypernetworks. Our ablation study, focusing on SIREN activation, suggests that the hypernetorrss need to model high-frequency representations of the weight perturbations. We begin by empirically quantifying the benefits of each activation type by evaluating the performance of the Maximum A Posteriori (MAP) estimate.
16

---

Published as a conference paper at ICLR 2024
We trained Resnet18 in both CIFARIO and CIFARI00 for 100 epochs to evaluate the predictive capabilities of the Sinusoidal hypernetwork versus each ReLU counterpart.
Table 3: Numerical results for classification trials with different hypernetwork activations.
Hypernet Activation
Accuracy
LL
Brier
ECE
ReLU Sine
91.11
-0.48
0.14
0.05
CIFARIO
91.70
-0.41
0.08
0.13
0.05
ReLU
67.79
-2.54
0.32
0.53
0.23
CIFARIO0
Sine
68.49
-2.39
0.31
0.52
0.22
In Table we find that Sine/Periodic activations (the "default" choice in SIREN) slightly outperforms a hypernet with ReLU activations. Still, results are very close, though there is trend in favor of sine both benchmarks. The original motivation behind using the sine activation related to modeling high-frequency content, which translates as details in structured signals such as images or video Sitzmann et al.] (2020). We can however see this "in the top of its head", so speak: in structured signals we care more for low-frequency content, and high-frequency is "good-to-have" content. We can interpret an input semantically we see its low frequencies, but not necessarily vice versa. For example, image compression will invariably throw away high frequencies first, and the last frequencies to lose will be the lower ones. Our conjecture is as follows: When using an INR to model perturbations, landscape" (perhaps even different than the one of model weights). In particular, we do not have any reason differentiate lower or higher frequency content in any respect. We "care" for all frequencies, so we need to have a good way to model high frequencies as well. Perhaps this is the reason the sine activation gives small edge over ReLU.
faced with different situation, that corresponds to different "frequency
To elaborate further on this argument, we constructed a setting where we can visualize the & param- eters and see if we can observe any meaningful connection between hypernetwork activation and frequencies modelled by the hypemeework.
Heperacteati
Hopemetecrk
109
159
103
190
156
100
29
54
190 20
Figure 8: Values of & as a function of input weight coordinates (channel-wise
In Figure 8 we plotted the the values of function of input weight coordinates. Specifically for Resnet18 trained on CIFAR we plotted the flattened values for each specific kernel position across channels (channel slice) for 2 different convolutional layers. Both types of hypernetworks produce well structured perturbation functions. The values produced from the sinusoidal hypernetwork expressed as a somewhat oscillatory behavior w.r.t. channel position, which translates as higher frequency content. As for the ReLU perturbations, while having some high frequencies due to the discontinuity of the ReLU activation, the overall signal has smooth structure less complicated that
17

---

Published as a conference paper at ICLR 2024
10
20
Figure Empirical variance of & as function of input weight coordinates (channel-wise).
the sinusoidal ones in some cases. Unsurprisingly, the ReLU result consists of practically piecewise linear components. This is what we believe that highlights the marginally better performance of SIREN hypernetworks. Furthermore, following the same experimental procedure we plotted alongside the mean values of & also their variance (Fig- 9. as this computed from the SWAG-diagonal approximate inference method, again as function of channel coordinates. We can observe that the variance has the same structural properties as the mean values of Thus, we believe that makes sense for the main network convolutional kernel to take advantage of its structure.
D EVALUATING INR HYPERNETWORK SIZE
We added an ablation w.r.t INR size following the UCI regression setting our method 4 We compare four different versions of INR hypemetworks with increasing number of parameters each, namely (BIG=2500 MED=625, SMALL=76XXMAALL- Laplace approximate posterior. From the experiments (Fig.
all combined with a Full GGN observe that there is
10
Conerrte
Energy
Kinšan
-3.2
2.96
2.98
0.9
1.04
3.22
1.02
-3
1111
3.4
3.02
-3.6
3.6
3.04
1.3
0.96
Concrete gap
Energy
Gap
Wine
Yactx
Kinflom Gap
-3
3.8
3.5
3.2
3.8
3.4
1.1
-3.6
Figure 10: Numerical results for regression trials UCI standard (Hermández-Lobato & Adams 2015 and GAP (Foong et al. 2019) splits for different hypemeework sizes.
limit to where easily scale the INR hypernetwork and simultancously gain performance. Individual characteristics play significant role to the INR size (main network size, dataset size, dataset dimension).
18

---

Published as a conference paper at ICLR 2024
E COMPUTATIONAL TIME
Regarding the computational time requirement of our method, it can be decomposed as follows:
Taal Hypemet evaluation + Approximmate Inference
(11)
Where hypernetwork evaluation time according to Table 4 makes the overall network in practice 1.2 slower than the vanilla network training. As for the approximate inference time although these methods we are using in our experiments are expensive, because in our method they are applied in the small dimensional INR space in general takes less time to evaluate. In Table [ we are considering the computational time of inference of our method versus standard inference popular ones.
Table 4: Indicative time requirements for INR- hypernetwork model.
Our method
Vanilla Network
method (fixed perturattion)) Forward
Forward
Backward
Forward
Backward
Backward
0.0069 + 0.0001
0.014 0.008
0.0046 0.0001
0.011 + 0.000
0.0043 + 0.0002
0.009 + 0.001
Table 5: Computational time of INR low dimensional inference versus other approximate inference methods.
Method
Deep Ensembles
Droouut
I.I. Laplace
INR SWAG
INR RealNVP
Inference Time
0.9014 0.0273
2.0030 0.0073
0.6393
1: 0.0043
For the Deep Ensembles method the obtained values include additional overhead such as ensemble element loading etc. as it is common practice. Furthermore, the Linearized LL Laplace is much slower than the other methods as computing the Jacobian for the ResNet50 reaches the limits of our computational budget at this time.
As for the overhead in terms of learnable parameters, we have: Winr (total number of the hyper- network parameters), and giur (number of approximate inference parameters applied the INR space), which as we mention in the main paper is in fact much less than Φw (number of approximate inference parameters applied on the full set of main network weights). Performance-wise our method is still being competitive w.r.t. methods like ensembles of D networks which at best is D times slower than the vanilla network.
Furthermore, because the main overhead of our method is the hypernetwork evaluation we investigated the following alternative training scheme, further improve our method in terms of time. Instead of training the main network weights W and W Winn n together we update the WiNn parameters every 10 epochs of the main network training, this significantly reduces the computational overhead of our method and we hypothesize it can scale to ImageNet models and datasets. Inference time for Resnet18 combined with different stochastic subspaces and different approximate inference methods (time is measured in seconds and for a batch of 10 CIFAR images).
Table 6: Numerical results for classification trials of ResNet18 in CIFAR
Training Scheme
Accuracy 1
LL
Error |
Brier
ECE
Full Training
69.01 68.59
-2.32
0.30
0.51
0.22
Altemative Training
-2.3
0.31
0.52
0.22
F ADDITIONAL EXPERIMENTS
Further Image Experiments. Following Antorán et al. (2020); Daxberger et al. (202 Ovadia al. (2019), train all methods MNIST and evaluate their predictive distributions increasingly rotated digits. We trained the models for 50 epochs using the Adam optimizer. The results are depicted in Figure 11 The importance of distributional shift expressed in this experiment via rotation of the
19

---

Published as a conference paper at ICLR 2024
original test set, which is highly informative all methods perform degradation shift reaches high intensity, where at this point methods begin to differentiate from one another. While the enror of the prediction remains the same, metries such ECE and LL favor INR inference and Dropout which surpass the Deep Ensembles and LL Laplace as degradation increases significantly.
less the same until the
Table 7: Numerical results for classification trials on different proposed cow-dimensional spaces (CIFARIO).
Standard
Comuprdd
Subspace
Inforence
LL
Error +
Brier
ECE 4
LL
Error
Brier
ECE
SWAG
0.41
0.08
0.13
0.05
-1.25
0.22
0.35
0.14
Rank1
Laplace
-1.56
0.09
0.68
0.68
- -1.70
0.22
0.73
0.57
SWAG
0.32
0.07
0.12
0.04
-1 -111
0.21
0.35
0.14
INR
Laplace
-1.66
0.11
0.68
0.66
-1.66
0.19
0.32
0.58
SWAG
-0.42
0.07
0.12
0.04
- -1.45
0.23
0.38
0.17
Subnetwork
Laplace
-1.65
0.71
Partially stochastic
SWAG Laplace
-0.42
0.07
0.12
0.04
-1.44
0,20
0.38
0.17
-1.56
0.09
0.68
0.70
-1.67
0.21
0.72
0.59
Deep Ensembles
Dropout
INR SWAG
I.p.
RealNV
L.L
Broor
Brier
ECB
0.8
-2
0.6
0.4
0.4
-4
0.5
0.2
10
0
10
10
40.5
0.5
-1.0
0.25
0.4
CIFAR
0.15
-1.5
0.1
0.15
Figure 11: Numerical results for classification trials Corrupted MNIST dataset. Log-Likelihood (1), Expected Calibration Error (L), Brier Score (L), Error (L) and Accuracy (f) are used for compari- son. The r-axis of each plot corresponds to increasingly levels of corruption intensity.
G QUALITATIVE EVALUATION OF EMPIRICAL DENSITIES
In this Section, we qualitatively inspect the approximate posterior distributions produced by INR variants regression and classification settings. First, in Figure 12 we plot the empirical density of o for the network trained the toy regression task. The variables are acquired by evaluating first eq/4[ with 400 samples. Then we transform each sample according to ș fiNR(-) and finally scale the resulted values by
As we can see, INR-based models produce non- Con-Gaussian approximate posterior distributions. Our results in line with works like Fortuin 22022) which analyzed the empirical weight distributions of SGD-trained networks with different architectures, suggesting that fully connected neural networks learn heavy-tailed weight distributions.
20

---

Published as a conference paper at ICLR 2024
We plotted empirical covariance matrices (see Figure 15 (specifically, the parameters that are "connected" to the first output neuron of the first layer of the main network). We can that even the with high-magnitude off-diagonal elements. This result validates the use of more expressive posterior distributions and highlights the performance of our hypernetwork method in the previous tasks.
that correspond to part of the wof parameters
models able to produce covariance matrices
We evaluated the empirical densities of convolutional layers following the classification setting of subsection 4.4 More specifically, trained ResNet-50 using the INR-RealNVP method on CIFARIO dataset and evaluated the approximate distribution of w for the first convolutional layer of the network, following the same sampling procedure as before. Results are depicted Figure 14 where the density histograms of the kernel values are Gaussian-lin like but still placing lot of probability mass towards the tails.
We plotted the empirical covariance (Figure 14 left) of values belonging to the same 3 x 3 kernel for nine different kernels. The covariance matrices indicated high spatial correlations of kernel values as was expected (Fortuin 2022).
NVP
100
-381
50
30
90
50
1000
-0.10
0.1%
01003
0100077 (U0033)
1000
50-
5000
-00
40-
1390
30
00
60
5000
Figure 12: Empirical Covariance for the INR-RealNVE for the first linear layer of the regression network.
IND Laplace
64-
0.25
6.4
0.20
0.-
1.4
0.-
6:2
0.10
6.1
0.10
0.0
0.2
86
1.50
10-
1.10
0.33
1.0
0.75
0.37
022
0.50
0.60
0.20
0.00
-30
-0.2
Figure 13: Empirical Covariance for the INR-Laplace for the first linear layer of the regression network.
21

---

Published as a conference paper at ICLR 2024
Empirical Density
Empirioa Conrlanne
0.55
0.35
000
-02
2.50
2.33
1.15
0.303
0.263
-122
Entropy
-044
Dannape
40-
-14
0.1
005
0.86
275 1129-112255
Vitrrpyy
Fortngg
101
00
14
0.95
30-
0.00
03
Excogge
Estropy
Panooys
Figure 14: Empirical density histogram and empirical covariance for of kernel values of the first convolutional layer of ResNet-50 using INR-RealNVP.
Figure 15: Empirical Covariance for the INR-ReaINVP and INR-Laplace for the first linear layer of the regression network.
22


Reviews:
[{"review_id": "f4n0kumTmI", "reviewer": "Reviewer_tKmG", "summary": "The authors propose a hypernetwork-based inference mechanism for BNNs, focusing on two key aspects: first, they want the hypernetwork to be compact and reusable across the main network; second, they perform approximate Bayesian inference over the compact hypernetwork. A key part of this work is the fact that the authors also keep around deterministic weights which are modulated by the stochastic nuisances sampled from the hypernetwork, allowing them to maintain good performance.\n\nThe method appears to be competitive in experiments.", "questions": "Given the relative dearth of new ideas in this work, I would like to identify its strengths in execution and overall quality of the work.\n\nCould the authors argue that their specific combination of techniques could be applied to a larger neural network, i.e. an imagenet model?\n\nCould the authors share a bit more about scalability and performance/memory constraints/ number of forward passes needed to obtain good performance?\n\nI would enjoy seeing more evidence that highlights the merits of the execution here, given the relatively modest technical contributions.\n\nAs the paper stands, I would find the contributions somewhat thin, but I would enjoy seeing this line of work with hypernetworks for BNNs be paired with a very strong empirical result in the style I ask for above to enrich the community and hope the authors can find something more to show that differentiates this work more from e.g. Dusenberry et al. empirically, which can also be interpreted as a specific hierarchical weight model (with layer-specific structure).\n\nAgain, I commend the authors for their quality scholarship and would enjoy more sound arguments to place this work into the trajectory of papers they have mentioned as a primarily empirical contribution.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors propose a relatively narrow idea: making a small hypernetwork probabilistic and sharing it across the network.\n\nWhat I like about this paper is the good exploration over ways to do this: the authors both try Laplace approximations, and stochastic weight averaging as pragmatic means to parametrize an approximate posterior over the hypernet. They also propose to use normalizing flows to shape the outputs of this hypernetwork to squeeze a bit more performance out of it.\n\nEmpirically, the work shows good performance.\n\nThe paper is overall well written and easy to follow and understand.\n\nI also want to call out that the authors are good scholars, the breadth of relevant work cited here is comprehensive and commendable.", "weaknesses": "The elephant in the room with this paper is that it is very narrow in its contribution.\n\nTiny shared hypernernetworks parametrizing individual implicit weight outputs via coordinate systems go as far back as the cited paper Karaletsos et al 2018. Bayesian inference over such hypernetworks has likewise been performed before, in the shape of GPs and BNNs which the authors both cite in their related work.\nBlending deterministic and stochastic weights coming from hypernetworks also has been done, again cited absolutely correctly by the authors in numerous papers.\n\nThe contribution I see here is not the novelty of any of the ideas then, but the specific engineering combinations tweaked to obtain good performance. For example, executing on practical pieces like Laplace for the hypernetwork or SWA and pairing that object with normalizing flows is good execution that probably helps with performance compared to previously mentioned works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvgYJQkTWe", "reviewer": "Reviewer_VEKs", "summary": "The authors present a new framework for approximate Bayesian inference in neural networks based on low-dimensional hypernetwork representations of weight perturbations. In this framework, hypernetworks take in weight coordinates as input and output perturbation factors for each weight. An approximate posterior is fitted for the hypernetwork and perturbations are repeatedly sampled and multiplied with the main network weights to perform Bayesian model averaging. Three alternative Bayesian hypernetworks are considered: sinusoidal representation networks (SIRENs) with Laplace and SWA-Gaussian approximate posteriors, and normalizing flows with Real Non-Volume Preserving (RealNVP) transformations. The posteriors are benchmarked on UCI regression and gap datasets, and out-of-distribution (OOD) detection is tested on CIFAR-10 image recognition.", "questions": "- Does the INR-Laplace model in the experiments employ a full Hessian or a GGN approximation? Do you make any additional approximations (e.g. Kronecker-factorization) ?\n- Can you further evaluate the effect of INR network size on model performance and quality of the uncertainty estimates? Figure 1 already does this for CIFAR-10 but it would be interesting to see for MLPs on UCI and UCI-gap datasets, with INR-Laplace and INR-RealNVP, and perhaps on wider sets of network sizes. Would also be helpful to include error bars for these figures.\n- Have you tried using ReLU hypernetworks? Do these fail to recover high frequency representations of the perturbations compared to SIREN?\n- Do you have an intuition for why multiplicative perturbations are so successful compared to additive perturbations of the weights? Perhaps uncertainty is improved when perturbations are scaled up by the magnitude of the weight?\n- Is there a reason for INR-SWAG appearing in some experiments and INR-Laplace in others?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed framework introduces a novel combination of approximate Bayesian inference and implicit neural representations as hypernetworks which, to the best of my knowledge, has not been explored in prior research.\n- The paper offers valuable insights, e.g. in considering the benefit of multiplicative over additive perturbances, and in finding that CNN models benefit from shared representations.\n- The evaluation of out-of-distribution (OOD) detection on the CIFAR-10 dataset is thorough and detailed. This plays a large role in showing the practical utility of the proposed approach.", "weaknesses": "Regarding the method:\n- The derivation of the Laplace approximation is confusing: In Equations (4) and (5) a Laplace approximation with full Hessian is derived. While a full Hessian for the hypernetwork weights may be tractable for small enough networks, this approximation does not correspond to the closed form posterior of Equation (7). This closed form corresponds to linearized Laplace inference with generalized Gauss-Newton (GGN) approximation to the Hessian. It is unclear if the model evaluated in the experiments uses a full Hessian or the GGN approximation with closed form.\n- The use of SIREN activation suggests that the hypernetworks need to model high frequency representations of the weight perturbations. The SIREN models presented in [Sitzmann et al., 2020] benefit from consistency and repeating patterns in the signals they are fitting, making interpolation easier. There might be some form of structural consistency in neighboring weight perturbations for CNN kernels, but it seems unlikely for the weights of linear layers. I also expect that RealNVP hypernetworks have a harder time fitting high frequency representations when comparing to SIREN. This matter is only very briefly brought up in the discussion of Figures 9 and 10.\n\nRegarding experiments:\n- The methods are benchmarked against a last-layer Laplace approximation, however a block-diagonal Kronecker-factorized (KFAC) Laplace approximation should also be considered since this corresponds better to a full network variant in the linearized Laplace family of approximations.\n- Both theoretical and experimental runtimes and memory requirements are not discussed. It is unclear if the proposed models take significantly more time for training or inference, since in theory a forward pass of the hypernetwork is required for every \"main\" network weight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsuF5bGbo1", "reviewer": "Reviewer_Ji8X", "summary": "This work proposes so-called implicit neural representation inference for Bayesian Neural Networks. In the past, several \"subspace\" inference frameworks have been devised, where the aim is to only model smaller part of the weight space in a Bayesian manner. This type of approaches promise to better scale the approximate Bayesian inference for deep learning, while making the inference procedure more accurate. Building upon, this paper proposes to obtain the \"subspace\" of weights using implicit neural representation. The paper shows how the proposed method can enhance popular frameworks such as Laplace Approximation, SWAG and normalizing flows. Experiments are conducted on UCI, cifar10 and cifar100 and the approach is compared against the baselines with performs inference over all the parameters of neural networks, as oppose to the subspace.", "questions": "1.In line with section 3.1, is it possible to explain why INR is advantages to learn the subspace?  I could not get why it might be a good idea.\n\n2. Another question is on expressiveness Vs accuracy of the Bayesian inference. Basically, the weight space is very large. Having a simple distribution in such a complex high dimensional space can be already advantageous in terms of expressiveness of the probability distribution. Of course, a natural direction has been also improving the expressiveness through structured distribution, correlations amongst layers, etc. On the other hand, what these subspace approaches do is to model with smaller part of the network, which can actually reduce the overall expressiveness of the distribution, though overall inference might be easier and accurate. Then the question is, when should we look into the approaches for expressiveness, and when should we look into the subspace approaches?\n\n3. Why not only take last few layers of the network and make them probabilistic? What are advantageous of using INR against very simple baselines as taking last one or three layers? I think it might be interesting to include them as a baselines in the experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In my view, the paper has the following strengths:\n\n- the idea of using implicit neural representation for improving Bayesian Neural Network is interesting and novel.\n\n- Implicit neural representations are currently a popular topic, and may be therefore relevant to many researchers.", "weaknesses": "On the other-hand, I think the paper has several rooms to improve for a publication.\n\n- The paper could improve in terms of its clarity.\n\nSpecifically, I find it difficult to parse section 3.1. I get the problem statement, but the parts on implicit neural representation with the SIREN model was difficult to understand. It would help to have a figure on this since there are many notations introduced. The text contains many mathematical symbols, which would have been explained differently. \n\nSIREN should be explained more in depth since I think it is an important technical detail. Also, some technical details on how implicit neural representation is obtained, and the general working principles, like an algorithm behind, would be helpful for the reader.\n\n- Experiments may become more solid with other choices of baselines and datasets.\n\nFirst, there has been many subspace approaches and also many approaches that attempts to sparsely represent model uncertainty. Some examples are references in the paper: Kristiadi, Dusenberry, Daxberger, etc. I think the experiments should compare to these baselines as well, which can really show the advantages of implicit neural representation over existing methods within the same class of approaches. Comparisons to full weight space seem not natural.\n\nMoreover, the paper uses UCI and CIFAR as main datasets. I would have liked the paper more if \"uncertainty baselines\" from google was used, as such works represent the more upto date standard in experimental protocol. Speaking of the protocol, the included baselines seem not very consistent, e.g., Figure 1 misses laplace approximation, figure 2 again selectively reports INR Laplace and contains no SWAG, figure 3 misses swag, INR swag, etc.  \n\nOverall, I recommend a weak rejection. Improving the technical quality and clarity of the presentation would be meaningful here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x4jofXBGSI", "reviewer": "Reviewer_wVmX", "summary": "The authors propose an approximate inference method for Bayesian neural networks. The idea is to introduce a second (auxiliary) network which computes an approximate posterior over the multiplicative noise applied to the deterministic weights of the main network. Such a multiplicative noise then induces an approximate predictive posterior distribution of the main network, they key object of interest in BNNs. The auxiliary network can be of much smaller size than the main one, and an approximate posterior over its weights can be approximated using one of the existing methods (e.g. Laplace, SWAG, Normalising Flows). The proposed method is shown to be competitive in comparison to a number of baselines.", "questions": "- I wonder about the integer inputs (i.e. the tensor coordinates) to the INR network. Do you normalise these coordinates somehow or directly input the integers into the INR network? Did such an integer input space cause any problems during training?\n- Why do you think that sinusoidal activations are particularly suitable for the INR network? Do you expect the result to deteriorate if you used other activations (e.g. sigmoid)? \n- Why do you think the INR with 350 outperformed 4k and 10k versions on CIFAR in Fig. 1?\n- I was very interested to see that the predictive uncertainty in Fig. 2 has a stationary structure (i.e. dependent on the distance from the observations) similar to a GP with a stationary kernel. The Dropout and Deep Ensembles baselines clearly don't have such a property, but I wonder how such a figure would look like if we used a Laplace approximation directly on some layers of the main network (without INR), e.g. similar to Kristiadi et al. (2020)? In other words, I wonder how specific this stationary uncertainty structure is to the inference using an auxiliary network?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ An interesting approach to approximate BNN inference showing competitive performance\n+ Clear presentation, the paper is easy to follow", "weaknesses": "The baselines used for comparison (Dropout, BbH, Ensembles) are relatively old methods (by Deep Learning standards of course). I'd be very interested to see how the proposed method compares to more modern approaches, e.g. those applying Laplace approximation directly on to weights of the main network.\n\nA couple of minor points:\n- Typo in the second to last line on page 3 (wf).\n- Typo in Eq. (5) (w_{INR} should be in the subscript I guess?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)