Variational Deterministic Uncertainty Quantification
Authors: Joost van Amersfoort
Lewis Smith
Andrew Jesson
Oscar Key
Yarin Gal
Venue:  (2021)
Keywords: Uncertainty estimation
gaussian processes
deep learning
variational inference

Abstract: Building on recent advances in uncertainty quantification using a single deep deterministic model (DUQ), we introduce variational Deterministic Uncertainty Quantification (vDUQ). We overcome several shortcomings of DUQ by recasting it as a Gaussian process (GP) approximation. Our principled approximation is based on an inducing point GP in combination with Deep Kernel Learning. This enables vDUQ to use rigorous probabilistic foundations, and work not only on classification but also on regression problems. We avoid uncertainty collapse away from the training data by regularizing the spectral norm of the deep feature extractor. Our method matches SotA accuracy, 96.2\% on CIFAR-10, while maintaining the speed of softmax models, and provides uncertainty estimates competitive with Deep Ensembles. We demonstrate our method in regression problems and by estimating uncertainty in causal inference for personalized medicine

Under review as a conference paper at ICLR 2021
VARIATIONAL DETERMINISTIC UNCERTAINTY QUANTIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Building on recent advances in uncertainty quantification using single deep deterministic model (DUQ), we introduce variational Deterministic Uncertainty Quantification (vDUQ). We overcome several shortcomings of DUQ by recast- ing it as a Gaussian process (GP) approximation. Our principled approximation is based on an inducing point GP in combination with Deep Kernel Learning. This enables vDUQ to use rigorous probabilistic foundations, and work not only classification but also regression problems. We avoid uncertainty collapse away from the training data by regularizing the spectral norm of the deep feature extractor. Our method matches SotA accuracy, 96.2% on CIFAR-10, while main- taining the speed of softmax models, and provides uncertainty estimates compet- itive with Deep Ensembles. We demonstrate our method in regression problems and by estimating uncertainty in causal inference for personalized medicine.
INTRODUCTION
Deploying machine learning algorithms as part of automated decision making systems, such as self driving cars and medical diagnostics, requires implementing fail-safes. Whenever the model presented with novel or ambiguous situation, would not be wise to simply trust its predic- tion. Instead, the system should try to get more information or simply withhold or defer judgment. While significant progress has been made towards estimating predictive uncertainty reliably in deep learning (Gal & Ghahramani 2016: Lakshminarayanan et al. 2017), there is no single method that shown to work on large datasets classification and regression without significant computa- tion overheads, such as multiple forward passes. We propose Variational Deterministic Uncertainty Quantification (vDUQ), a method for obtaining predictive uncertainty in deep learning for both clas- sification and regression problems in only a single forward pass.
In previous work, van Amersfoort et al. 2020) show that combining a distance aware decision func- tion with regularized feature extractor in the form of deep RBF network, leads to model (DUQ) that matches a softmax model in accuracy, but is competitive with Deep Ensembles for uncertainty on large datasets. The feature extractor is regularized using two-sided gradient penalty, which encourages the model to be sensitive to changes in the input, avoiding feature collapse, and en- couraging generalization by controlling the Lipschitz constant. This model, however, has several limitations; for example the uncertainty (a distance in feature space) cannot be interpreted proba- bilistically and it is difficult disentangle aleatoric and epistemic uncertainty. Additionally, the loss function and centroid update scheme are not principled and do not extend to regression tasks.
A probabilistic and principled alternative deep RBF networks are Gaussian Processes (GPs) in combination with Deep Kernel Learning (DKL) (Hinton & Salakhutdinov 2008 Wilson et al.) 2016b). DKI. was introduced as a "best of both worlds" solution: apply a deep model on the training data and learn the GP in feature space, ideally getting the advantages of both models. In practice, however, DKL suffers from the same failure as Deep RBF networks: the deep model free to map out of distribution data close to the feature representation of the training data, removing the attractive properties of GPs with distance sensitive kernels.
Using insights from DUQ, are able mitigate the problems of uncertainty collapse in DKL. In particular, we use direct spectral normalization (Gouk et al., 2018; Miyato et al. 2018)) in combina- tion with ResNet (He et al. 2016), variation that was suggested in Liu et al. (2020). The spectral

---

Under review as a conference paper at ICLR 2021
0.0
2
-25
-7.5
-2
-10.0
-12.5
-15
-10
-5
10 15 -15 -10
5
10
(a) Deep Ensembles
(b) vDUQ
Figure 1: We show results ID regression dataset of a sinusoid curve. In green data points of the dataset and blue the prediction including uncertainty (two standard deviations). As expected, vDUQ reverts to the prior away from the training data, while Deep Ensembles extrapolates arbitrarily and confidently. The uncertainty is the posterior variance the case of vDUQ, and the variance across the ensemble element predictions in Deep Ensembles.
normalization enforces smoothness, while the residual connections enforee sensitivity of the feature represenation to changes in the input, obtaining a similar effect as the gradient penalty of DUQ. We use inter-domain inducing point variational approximation of the GP predictive distribution LLázaro-Gredilla & Figueiras- Vidal 2009) Hensman et al.) 2013), which places inducing points in feature space leading to needing fewer inducing points than previous work (Wilson al. 6a) These two techniques combined speed up inference in the GP model and decouple it from the dataset size. We release our code_ and hope that will become drop in alternative for softmax models with improved uncertainty.
In Figure we show how vDUQ and Deep Ensembles @Lakshminarayanan et al. 2017). the current state of the art for uncertainty quantification (Ovadia et al. 2011)). perform on simple ID regression. This task is particularly hard for deep networks as shown Foong the desired behavior of reverting back to the prior away from the data, while the Deep Ensemble extrapolates arbitrarily and confidently. In between the two sinusoids, the Deep Ensemble certain while vDUQ increases its uncertainty.
(2019). vDUQ shows
In summary, our contributions are as follows:
We improve training DKL model and for the first time match the accuracy and speed of training deep network using regular softmax output on standard vision benchmarks.
We demonstrate excellent uncertainty quantification in classsification which matches or exceeds the state of the on CIFAR- 10, including ensembling approaches.
We show state of the performance causal inference for personalized medicine, exciting real world application. This task requires calibrated uncertainty in regression to be able defer treatment an expert when uncertainty is high.
BACKGROUND
Gaussian Processes (GPs) provide an interpretable, explainable and principled way to make predic- tions, and can work well even with little training data due their use of Bayesian inference. In contrast to deep neural networks, GPs have high uncertainty away from the training data and on noisy inputs.
There are however two main issues with the standard GP setup: poor performance on high dimen- sional inputs and inefficient computational scaling with large datasets. The poor performance on high dimensional inputs is due to the fact that most standard shift-invariant kernels are based on
Available anonymi zed- for review
2

---

Under review as a conference paper at ICLR 2021
(a) Input
(b) Features constraint
(c) Features with constraint
Figure 2: On the left we visualize a 2D classification task where the classes are two Gaussian blobs (drawn in green), and large grid of unrelated points (colored according their log-probability under the data generating distribution). In the center we see the features as computed by an un- constrained model. On the right the features computed by model with residual connections and spectral normalization. As we can see, the objective for the unconstrained model introduces large amount of distortion of the space, largely collapsing the input impossible to use distance-sensitive measures on these features. In contrast, a constrained mapping still makes the classes more separable, while maintaining the relative distances of the other points.
single line, making it almost
the Euclidean distance, which is poor metric for highly structured data like images (Theis et al. 2016). While kernels exist that better address this (Van der Wilk al.) [017]] Jacot et al. (2018). these are more computationally expensive and typically still underperform standard convolutional
neural networks.
Deep Kernel Learning (Calandra et al. 2016: Hinton & Salakhutdinov 2008 Wilson et al. 2016b)
way to combine the expressiveness of deep neural networks with the attractive properties of Gaussian Processes. The core idea is to use a deep neural network inside the kernel of GP, k(,,, + f. (xx)), where fa(-) is a deep neural network, such as a Wide ResNet Zagoruyko & Komodakis 2016) up to the last linear layer, parametrized by O. The kernel k(:,) )) can be any of the standard kernels, such as the RBF or Matérn kernel. With the deep network becomes possible to train the GP datasets that contain high dimensional points, such images. As the deep network is unconstrained, uncertainty away from the data collapses (Bradshaw et al.] 2017), and the model can be arbitrarily confident while data has been observed in that region during training. This has been a major drawback to using DKL in practice. In Section 3 we provide detailed discussion of this effect and ways to mitigate
While DKL is a potentially powerful solution to overcome the problem of inexpressive kernels, it does not address the other scaling problem of Gaussian Processes: large datasets. The poor computa- tional scaling comes from the fact that making predictions with an exact GP requires solving linear system the size of the training data, namely computing K(X, X x²) where K (XX, X) is the kernel matrix evaluated on the training dataset X and ((X, x²) the kernel between the training dataset and test point x². This a significant computational problem and principled alternative exact GP inference is inducing point GP approximate inference, where only a small set of inducing points in the input space U are used to represent the entire training set, reducing the linear system to be solved to K ((u, - (u, x²)). The new linear system is m by m, where m is the number of inducing points, rather than N by N. Finding the optimal set of points, which not necessarily part of our training set, be done by treating them variational parameters and maximizing a lower bound on the marginal log-likelihood (Titsias) 2009). We fol- low the variational inducing point formulation of SVGPC (Hensman et al. 2015) implemented in GPyTorch (Gardner et al. 2018).
(X,
its own right. A powerful
3 METHOD
vDUQ is an instance of Deep Kernel Learning, combining a constrained deep neural network with an inducing point GP. The model is learned end-to-end using variational inference by means of gradient
3

---

Under review as a conference paper at ICLR 2021
Algorithm Algorithm for training vDUQ
Initialization:
Residual NN fo x Re with feature space dimensionality d and parameters 0 initialized
using He al. (2015).
Approximate GP with variational parameters & and number of inducing points m.
Using random subset of k points of our training data, Xinit X, compute:
Initial inducing points: K-means on f(Xiiiit) with K = m. Use found centroids as initial inducing point locations in GP. Initial length scale:
Li
Lfrtt |f(X -Xx, f(X)m))
1: for minibatch B, = f21...., I6 from X and By = {1/1...... yb from Y do
2:
0' spectral normalization(()
3:
//
for(B,
(Bz)
> Compute feature representation
4:
P(Y BB)
GP (1))
Compute posterior labels, implemented in GPyTorch
5:
L 5
Compute loss, implemented in GPyTord > 1j the learning rate, alternatively use ADAM
6:
o,0 + Φ,
7.0
7: end for
descent. In this section we discuss how vDUQ overcomes previous problems with collapsing un- certainty in DKL (Bradshaw al.) (017) and explain how learn the model with no pre-training, few inducing points, and a standard minibatch size, avoiding some shortcomings of Wilson et al. (2016a).
Without restrictions, the deep network inside the kernel is free map input points that are far away from the training distribution to a feature representation that resembles those data points in the training distribution. This behavior is also exhibited in standard neural networks (Smith & Gal, 2018) and sometimes referred to as feature collapse. We visualize this in Figure 2 where we map 2D input space into feature space. With the feature space of the unconstrained model it is impossible to recover uncertainty: many points are collapsed on top of each other.
In DUQ (van Amersfoort al. 2020), the authors show that is possible reduce feature collapse by enforcing two constraints on the model: sensitivity and smoothness. Sensitivity implies that when the input changes the feature representation also changes: this means the model cannot simply collapse feature representations arbitrarily. Smoothness means small changes in the input cannot cause massive shifts the output. This appears to help optimization, and ensures the feature space accords with the implicit assumptions that for example RBF kernels make about the data. We discuss connection between these two requirements, sensitivity and smoothness, and bi hi-Liisshitz functions in Section 6
There is number of methods proposed the literature that attempt to satisfy these constraints and each comes with different trade-offs:
Two-sided gradient penalty: In DUQ (van Amersfoort et al. 2020), the properties are achieved by regularizing using two-sided gradient penalty, that penalizes the squared distance of the gradient from fixed value at every input point. This approach is easy to implement, but not guaranteed work, and in practice both the stability of training and its effectiveness as regularizer can be fairly sensitive to the weighting of this penalty.
Direct spectral normalization and residual connections: spectral normalization (Miy- ato et al. 2018 Gouk et al.) (018) the weights leads to smoothness and it is possible to combine this with an architecture that contains residual connections for the sensitivity constraint (Liu et al. 2020). This method is faster than the gradient penalty, and in practice a more effective way of mitigating feature collapse.
Reversible model: A reversible model is constructed by using reversible layers and avoid- ing any down scaling operations (Jacobsen et al.) 2018 Behrmann et al.) 2019). This approach can guarantee that the overall function is bi- Lipschitz, but the resulting model consumes considerably more memory and can be difficult train.
4

---

Under review as a conference paper at ICLR 2021
In this work, use direct spectral normalization and residual connections, find it to be more stable than a direct gradient penalty and significantly more computationally efficient than reversible models. In Figure 2 we show that a constrained model is unable to collapse points top of each other in feature space, enabling the GP to correctly quantify uncertainty. Using stationary kernel, the model then reverts back to the prior away from the training data just like a standard GP.
The regularized feature extractor allows us to offload computational complexity from the GP onto the deep model without sacrificing uncertainty. An expressive deep model is able to find feature representation which generalizes across all intra class variation, and thus cluster inputs of the same class in feature space. We define the inducing points in feature space instead of input space (also known as an interdomain approximation) 2017), which takes advantage of this clustering to reduce the number of inducing points required. In practice, we find that only very few inducing points, the number of classes in the case of classifica- tion, are necessary to obtain a well performing GP and we found similarly low numbers (in the 10 to 100 range) work well for regression, which means that solving the linear system is fast and the GP has minimal overhead compared to softmax output. The fact that we can use few inducing points is not true in general for DKL (e.g. Wilson et al. (2016a) use several hundred points in input space), but requires the extra restrictions we have placed on our feature extractor to avoid the pathological feature collapse. We also find we can train with standard minibatch sizes of 128, while previous work used 5,000 on CIFAR-10 to combat gradient variance (Wilson et al.) 2016a). It is important to note that in DKL and also in this paper we are not being Bayesian about our neural network param- eters 0, so we only need a single pass through the feature extractor both when training and during inference.
-aeaillGeeillla & Figueiras- Vidal 2009 Hensman et al.)
3.1 SPECTRAL NORMALIZATION BATCH NORMALIZATION LAYERS
When applying spectral normalization the parameters of deep model, is important note that batch normalization, a crucial component of training deep ResNets, has a non-trivial Lipschitz con- stant. In particular, since batch normalization transforms the input, using scale and shift parameters and 3, according to
Fout = diag
?
(o - (xx))
+ B,
VVr((x)
Var(z) (Gouk al. 2018). Using the above equation, we can extend spectral normalization to batch normalization by dividing the weight 7 of the batch nor- malization by the (scaled) Lipschitz constant. In practice, find empirically that typical batch normalization layers in trained ResNets have relatively high Lipschitz constant, up to around 12, with 95% having a Lipschitz greater than one (see also Figure in the Appendix). This is counter to the claim of Liu et al.] (2020) that batch normalization reduces the Lipschitz value of the net- work. Following these claims, Liu al. (2020) only constrain the convolutional layers, which as we demonstrate empirically results in models which are less sensitive to changes in the input, violating the constraint introduced DUQ. Unless otherwise noted, we apply spectral normalization to both the convolutional and the batch normalization layers. For convolutional layers with 1 x 1 filters we use exact normalization, while for large filters we use an approximation, implemented originally by Behrmann et al. (2019).
it has Lipschitz constant of max;
4 RELATED WORK
Currently, the simplest and most effective way to obtain uncertainty in deep learning classification is using ensembling (Lakshminarayanan et al.) 2017). Despite its simplicity it has shown to be remarkably effective (Ovadia et al., (201)), outperforming alternatives such as MC Dropout (Gal & Ghahramani 2016) and mean-field variational inference Bayesian Neural Networks (Farquhar et al. 2019) at the expense of having to train and evaluate multiple models. An important issue with Deep Ensembles is enforcing diversity; while on large and complicated datasets it is sufficient to use a different initialization and data order, this is not sufficient on easier or smaller datasets, as we highlight in Figure 1
5

---

Under review as a conference paper at ICLR 2021
-2
(a) ResNet Softmax
(b) FFN DKL
(c) vDUQ
Figure 3: We show uncertainty results on the two moons classification dataset. Yellow indicates high confidence, while blue indicates uncertainty. In Figure 3a simple feed forward resnet with softmax output is certain everywhere except on the decision boundary. In Figure 3bb we see that simple feed forward model in combination with DKL has large areas of certainty, even away from the training data. In Figure 3c we show vDUQ with residual network as feature extractor in combination with spectral normalization, showing close ideal uncertainty on this dataset.
van Amersfoort et al. 2020) demonstrate empirically with DUQ that it is possible to quantify un- certainty deep learning using only a single deterministic model. SNGP (Liu al. 2020) extends on DUQ's ideas of a distance aware prediction function and spectral regularization, and implements them by using a deep neural network with a random fourier features (RFF) approximate GP and direct spectral normalization ization). While they obtain impressive results, is important to note that the RFF approximation makes the GP parametric, meaning there now fixed number of basis functions used in the prediction which can lead to unreliable uncertainty. In contrast, our use of variational Bayes in the form of SVGPC conserves the non-parametric property and uses infinitely many basis functions; is an approximation to the true GP posterior, whereas Liu et al. (2020)* 's random Fourier expansion leads to different model which converges expectation the true model as the number of random Fourier features tends to infinity. This is discussed in Section 6 of [Lázaro-Greiila, et al. (2010) which comment that the RFF approximation is risk overfitting, while the variational inducing point approximation is safeguarded from it as they are parameters of the variational approximation (Titsias 2009). Furthermore, the RFF approximation is only tractable for the RBF kernel and sig- nificantly more expensive for other kernels, restricting the flexibility of the model. Finally. although in principle the method of Liu et al. (200)) could be adapted to regression in a straightforward way, they do not investigate this. Our experiments demonstrate the applicability of this kind of model both on standard classification benchmarks and uncertainty sensitive regression tasks, such causal inference.
the convolutional weights (but not on the weights of batch normal-
EXPERIMENTS
5.1 REGRESSION AND IN-BETWEEN UNCERTAINTY
In Figure we show results on a ID dataset created from sinusoids. For vDUQ, we use spectral normalization, a Matérn kernel with to less smooth learned functions, which leads to quickly increasing the uncertainty when there is no data. Full experimental details are provided Appendix A
and 20 inducing points. This kernel is known lead
Deep Ensemble extrapolates incorrectly as & becomes more negative, at x = -15 the prediction far away from what seen in the training data, however the model still has low uncertainty. Meanwhile vDUQ reverts to the prior directly at the edge of the training data domain, which is the Deep Ensembles (L.akshminarayanan
ideal behavior this regression setting. suggested to let the model predict both the mean and the standard deviation. This standard deviation only measures the aleatoric uncertainty (see also Chua et al. (2018) for a discussion), which the uncertainty arising from noise inherent in the data. In this dataset there is only minimal noise, and therefore the only way to assess the uncertainty to compute the variance the mean predictions. In contrast, vDUQ's posterior variance is directly interpretable known as the total uncertainty). In the Appendix, Figure 4 we visualize sampling from the posterior, which highlights the non-smoothness of the kernel choice.
al., 2017). ) it was
the predictive uncertainty (also

---

Under review as a conference paper at ICLR 2021
Table Results on CIFAR-IO and distinguishing the test of CIFAR-IO and SVHN by uncertainty (AUROC). All results are the average and standard deviation of 5 runs.
Method
Accuracy (%)
AUROC
WRN Zaaoruykoo & Komodakis 2016)
96.2++0.03
0.932+0.018
Ensemble of 5 WRN (Lak Laakshmnarayanan al. 2017)
96.611.06
0.967
SNGP (Liu et al. 2020)
96.0100.09
0,940
DUQ van Amerstoort al. 2020
94.6+1.16
0.940+0.007
vDUQ (no Spectral Normalization)
96.2-t0.11
0.937
vDUQ
95.6+0009
0.95810.012
5.2 CLASSIFICATION ON TWO MOONS
We show results on Two Moons dataset (Pedregosa et al. 2011) for three different models: a standard softmax model, vDUQ, and variation where the spectral normalized ResNet is replaced by fully connected model (similar to Bradshaw et al. (2077). ). For experimental details see Appendix [
On this simple dataset, a multi layer neural network easily achieves 100% accuracy with sin- gle, non-linear, decision boundary going between the two moons. However these models exhibit complete certainty everywhere in the input domain as shown in Figure Bal with yellow the certain regions, and blue uncertain. The uncertainty is computed using the entropy of the class prediction: we model the problem as a two class classification problem. vDUQ (in Figure on the other hand quantifies uncertainty exactly as one would expect for the two moons dataset: certain on the training data, uncertain away from it and in between the two half moons. Figure 3hJ the importance of our contribution. When using DKL with deep network without the right constraints, the model generalizes beyond the training distribution and certain even away from the training data.
5.3 CIFAR-10 AND SVHN
In this section we look at training a large model, the Wide Residual Network, on CIFAR-10 (Krizhevsky al. 2009). For vDUQ, train the large ResNet end-to-end with the GP, this is in contrast with prior work that used a pre-trained model (Bradshaw et al. 2017). which limits the ability to apply the method new domains. We follow the experimental setup of Zagoruyko & Komodakis 2016), and use a 28 layer model with BasicBlocks and dropout. Interestingly, we can follow hyper parameter suggestions (such as dropout rate, learning rate, and optimizer) directly when training vDUQ, and no special care is necessary for the variational parameters. We remove final linear layer of the model and the resulting 640 dimensional feature vector is directly used in the GP. We train using only 10 inducing points, which are shared among the independent output GPs (although qu)) is different) and use Lipschitz factor 3. This means the model is fast, and go- ing through one epoch is just 3% slower than using softmax output. Further experimental details are discussed in Appendix A Appendix [
We provide an ablation study of the number of inducing points in
Results are shown in T vDUQ without any spectral normalization matches the accuracy of the WRN, but the uncertainty only slightly better than softmax model. With spectral normalization the accuracy drops slightly, as expected, but the uncertainty improves significantly. Note that SV- DKL (Wilson et al.) 2016a), our most direct baseline, obtains just 91% accuracy on CIFAR-10 10 using a ResNet-20 (for which the softmax accuracy is 94%, see also van Amersfoort et al. (020 ). Meanwhile, convolutional GPs (Van der Wilk al. 2017) obtain 64.6% accuracy while using 1,000 inducing points. We perform an ablation where we train vDUQ with spectral normalization only on the convolutions, similar to (Liu al., 2020), and we obtain an AUROC of 0.93 + 0.003 and accuracy matching the model without spectral normalization. This provides further evidence that the batch normalization layers undo the effect, and need to be properly normalized.
"Ohtained using the author's open source implementation, which is available at httpttttt////. gpogle/uncertainng basel ine
7

---

Under review as a conference paper at ICLR 2021
Table Comparing the performance of several causal effect inference methods. Best results in bold and note that DKLITE" results are obtained using their open source implementation (see A.3
CPENF
CEMNIST (50% deferred)
IHDP Cov. (50% deferred) random
IHDP (10% deferred) random
Method/ / Policy
random
uncertainty
uncertainty
uncertainty
BART
2.14.02
2.0.1.03
2.6.1.2
1.81.2
1.901.20
1.601. 10
BT-Learner
.271.00
.0.1.01
2.31.2
1.311.1
0.95
0.6911011
BTARNet
.18+.01
.00
2.2
1.2+.11
1.10#.03
0.76
BCFR-MMD
.32+.01
.13+.02
2.5+.2
1.7+.2
1.30::06
0.91
BDragonnet
.22.001
.022.01
2,4
1.3+.2
1,50#.05
1.05#.02
BCEVAE
.304.01
.041.01
2.5:.2
1.74.1
1.80+106
1.47
DKLITE'
2.6
1.8:555
1.744:53
1.344:41
VDU
.164.02
.01+000
1.82::35
1,23
0.90
0.53+.09
5.4 CAUSAL INFERENCE
Personalized healthcare is an exciting application of machine learning, where the efficacy of a treat- ment is predicted based on characteristics of the individual using a model trained on previously treated patients. individual's response treatment can only be known if they are member of group represented in the data and if there is prescription diversity in the group: treatment and no treatment. [Jsssnn et al. (2020) show that measures of uncertainty can identify when personalized causal-effect inference fails due to such factors. Uncertainty can then define policies for deferring treatment recommendations to an expert when there is insufficient knowledge about a person. The uncertainty estimates must be correct, tions even if their response to treatment is not known, which can result in undue stress, financial burdens, worse. In this section evaluate vDUQ on the task of inferring the personalized causal-effects binary treatment on an individual æ: via the conditional average treatment ef- feet: CATE(() ( Y==](=)] (Abrevaya et al. 2015). Intuitively, an individual would be recommended treatment if CATE(#) > O. We show that vDUQ yields uncertainty estimates that lead to safer and more data-efficient policies for withholding treatment recommendations than alternative methods significantly improving previous state-of-tle of-the--r--rer art results.
else individuals would receive treatment recommenda-
We use the semi-synthetic IHDP Hill 2011), IHDP Cov. and CEMNIST datasets to assess the uncertainty estimates of vDUQ on this causal inference task following the experimental setup in [Jesson et al. 2020). IHDP and IHDP Cov. are regression datasets with only ~550 data points, while CEMNIST is binary classification dataset with 10k points. Treatment-effect recommenda- tions are deferred to an expert either at random, or based on the uncertainty in the CATE estimate. Recommendations are deferred for 10 % of the cases in the IHDP dataset, and 50 % of the IHDP Cov. and CEMNIST datasets. We report the root expected Precision in Estimation of Heterogeneous Ef- fect (Hill 2011 (v (vCPEHE) to assess the error the remaining recommendations (lower is better). Table 2 summarizes our results and shows that vDUQ has improved performance and uncertainty estimates better suited rejection policies than other uncertainty aware methods. In particular, improve on DKLITEE (Zhang et al.] (2020), the baseline related to our method: an alternative deep kernel learning method designed specifically for CATE inference.
LIMITATIONS
Even though demonstrated to work well empirically above, a theoretical justification of the spectral normalization in combination with residual connection is still lacking. Liu et al. (2020) propose an explanation based on the fact that spectral normalization, in combination with residual architectures, induces a bi-L.pschitz constraint, which is defined as:
d (x S dy (fo(z)), fo(z)) < Kdx(F1,
From that constraint it follows that if we have two metric spaces X and Y, with a continuous function f X Y between them which is bijective, and f is bi-Lipschitz, then metric properties of X like boundedness and completeness have to be preserved by Y. and distances in X can be changed only by bounded factor in Y. In this sense, bi-L-pschhizz map between X and Y can be considered to
8

---

Under review as a conference paper at ICLR 2021
be "approximately" distance preserving. Liu et al. (2020) argue that this means that applying this penalty to a Wide ResNet causes it to be distance preserving mapping. However, the proof they provide is only valid for spectral regularisation with coefficients less than 1, whereas empirically find that exceeding has similar effectiveness in terms uncertainty and is easier to optimize. This suggests that the proof does capture the underlying mechanism of the spectral normalization. In addition, this argument is not applicable to models that do any form of downsampling as it is straightforward demonstrate the following observation (which do not claim to be novel):
Proposition 1. Let f be function from R IR" with
Then f is not bi-Lipschitz
We provide a proof in Appendix B it suffices to show that bi- bi-Lipschitz function is necessarily homeomorphism, and so cannot exist between spaces of different dimensionality.
We thus consider explanations of the effectiveness of spectral normalization and residual connec- tions based on the assumption that the bi-Lipschitz condition holds insufficient if the network uses dimensionality reduction, which most feature extraction architectures do. Some progress towards a well performing bijective deep model was made in [echrmann et al. (2019), however the iResNet significantly slower and more computationally expensive than the WRN, and were unable to achieve the performance we report here without using cimension-reducing layers. Despite these open theoretical questions, have far not found practical example where the uncertainty qual- ity was poor, and our empirical results call for further theoretical study.
7 CONCLUSION
We present vDUQ, a practical probabilistic method which allows for using deep architectures and established its efficacy in 2D, large image dataset and in causal inference for personalized medicine, obtaining or matching state of the art results each. Good uncertainty and fast inference both classification and regression make vDUQ compelling plug-in option for many applications, such as exploration in reinforcement learning, active learning, and many real world tasks. Exciting directions for future research include rigorous theory of why spectral normalization works despite downsampling, or alternatively a computationally efficient way to enforce a bi bi-Lipschitz constraint.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jason Abrevaya, Yu-Chin Hsu, and Robert P Licli. Estimating conditional average treatment effects. Journal of Business & Economic Statistics, 3354):485-505, 2015.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573-582, 2019.
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv preprint arXiv:1707.02476, 2017.
Luitzen EJ Brouwer. Beweis der invarianz des n-dimensionalen gebiets. Mathematische Annalen, 71(3):305-313, 1911.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Marc Peter Deisenroth. Manifold gaussian processes for regression. In 2016 International Joint Conference on Neural Networks (IJCNN) pp. 3338-3345. IEEE, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-4765, 2018.
Sebastian Farquhar, Michael Osbore,, and Yarin Gal. Radial bayesian neural networks: Robust variational inference in big models. arXiv preprint arXiv:1907.0086; 2019.
Andrew YK Foong, Yingzhen Li, José Miguel Hermández-Lobato, and Richard E Turner. 'in- between' uncertainty in bayesian neural networks. Workshop on Uncertainty and Robustness in Deep Learning, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackhox matrix-matrix gaussian process inference with gpu acceleration. 2018.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing lipschitz continuity. arXiv preprint 1xXv::804.00438, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian pro- cess classification. JMLR, 2015.
James Hensman, Nicolas Durrande, and Amn Solin. Variational fourier features for gaussian pro- cesses. The Journal of Machine Learning Research, 18(1):5537-5588, 2017.
Jennifer L.. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statisties, 20(1):217-240, 2011. doi: 10.11988jg5.2210008166 URL https: //doi .aah.org/lii 10. 1198 11g.//cgss..../// 2010.08162
Geoffrey E Hinton and Russ R Salakhutdinov. Using deep belief nets to learn covariance kernels for
gaussian processes. In Advances neural information processing systems, pp. 1249-1256, 2008.
törn-Henrik Jacobsen, Amold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv: 1800007088 2018
10

---

Under review as a conference paper at ICLR 2021
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. Identifying causal effect inference failure with uncertainty-aware models. arXiv preprint arXiv:2007.0016: 2020.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In Advances Neural Information Processing Systems, pp. 7026-7037, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.
Miguel Lázaro-Grees Gredilla and Anibal Figueiras- Vigual inference using inducing features. In Advances Neural Information Processing Systems, pp. 1087-1095, 2009.
Vidal. Inter-domain gaussian processes for sparse
Miguel Mázaro-Grediila, Joaquin Quinonero-Candela, Carl Edward Rasmussen, and Aníbal R Figueiras- Vidal. Sparse spectrum gaussian process regression. The Journal of Machine Learning Research, 1 11:165-1881. 2010.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedisx-Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.10108 2020.
Alexander Graeme de Garis Matthews. Scalable Gaussian process inference using variational meth- ods. PhD thesis, University of Cambridge, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Process- ing Systems, pp. 13991-14002, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Uri Shalit, Fredrik Johansson, and David Sontag. Estimating individual treatment effect: gener- alization bounds and algorithms. In International Conference on Machine Learning, pp. 3076- 3085. PMLR, 2017.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detec- tion. Conference on Uncertainty Artificial Intelligence, 2018.
Theis, A van den Oord, and M Bethge. A note on the evaluation of generative models. In International Conference Learning Representations (ICLR 2016), pp. 1-10, 2016.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics, pp. 567-574, 2009.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. International Conference on Machine Learning, 2020.

---

Under review as a conference paper at ICLR 2021
Mark Van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional gaussian pro- cesses, In Advances in Neural Information Processing Systems, pp. 2849-2858 2017.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep
kernel learning. In Advances in Neural Information Processing Systems, pp. 2586-2594, 2016a.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378, 2016b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Yao Zhang, Alexis Bellot, and Mihaela der Schaar. Learning overlapping representations for the estimation of individualized treatment effects. arXiv preprint urXiv:2001.0475: 2020.
12

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 1 2D EXPERIMENTS
We perform our 2D experiments using a simple feed forward ResNet. The first linear layer maps from the input the feature representation and does not have activation function. From there on, the model is a ResNet, x' = x + [(x), with f(() combination of a linear mapping and a relu activation function. The linear mapping has optional Spectral Normalization for which we use the implementation of Behrmann et al. (2019). We use the SGD optimizer for regression and Adam for the two moons with learning rate 0.01, and we use 4 layers with 128 features. For the toy regression, we use max Lipschitz constant 0.95, one power iteration and 20 inducing points. For the two moons, we set the noise level to 0.1 and use five power iterations, Lipschitz constant of 0.65 and four inducing points. For the Deep Ensemble in Figure D we train 10 separate models, using different initialization and data order for each and train to minimize the squared error.
A.2 WIDE RESNET EXPERIMENTS
For the WRN, we follow the experimental setup and implementation of Zagoruyko & Komodakis (211)). This means that for CIFAR-10, we use depth 28 with widen factor 10 and BasicBlocks with dropout.
We train for the prescribed 200 epochs with batch size 128, starting with learning rate 0.1 and dropping with factor of 0.2 at epoch 60, 120 and 160. We use the full training set and take the model epoch 200 (no early stopping).
For Spectral Normalization, we again use the implementation of Behrmann et al. (2019) in particular the convolutional version and also constrain batch normalization as described (Gok al. 2018). We use power iteration and use the lowest Lipschitz constant that still allows for good accuracy, which we found to be around 3 in practice. We increase the momentum of batch normalization to 0.99 (from 0.9 default) to reduce the variance of the running average estimator of the empirical feature variance, which can be a source of instability (Gouk et al. 2018).
A.3 CAUSAL EXPERIMENTS
Following Shalit et al. 2017). we use 63/27/10 train/validation/tes splits and report the PEHE eval- uated on the test set over 1000, and 20 trials for the IHDP and CEMNIST datasets, respectively. For each trial, we train for a maximum of 750 epochs and evaluate the model with the lowest ELBO evaluated over the validation set. We employ Adam optimization with a learning rate of 0.001 and batch size of 100.
The feature extractor uses a feed forward ResNet architecture with 3, 200 unit hidden layers, and ELU activations. Dropout is applied after each activation at rate of 0.I. The feature extractor takes the individual x and treatment t as input. For the CEMNIST experiment, a depth CNN input the above
Resnet is used to extract features from the image, which is then passed architecture. Spectral normalization is used on all layers of the feature extractor. We use Matérn kernel with 3. 200 inducing points, and smoothed box prior the lengthscales with range (exp(-1), exp((1))xxp1) exp())).
For the DKLITE experiments able at hhtps://bitbuckktt Srrpuckkt..r///dscchaeeee aattar/mmoorraaaaaa forrhelthhabpuul http//rc/mastte alg/dklite/ We write a custom loop over the IHDP dataset follow the above protocol. For DKLITE on CEMNIST, in initial experimentation we found that the method did not adapt well to images, so we omit the comparison from the table. We make this adaptation available at Seamymized-foo-roee
the open source code from the authors avail-
A.4 INDUCING POINT GP
We initialize the inducing points by doing k-means the feature representation of 1,000 points. We compute the initial length scale by taking the pairwise euclidean distance between feature represen- tation of the 1,000 points. We whiten the inducing points before training, suggested in Matthews
13

---

Under review as a conference paper at ICLR 2021
2017). The kernel parameters, such the length scale and output scale, different per GP. We implement it using GPyTorch (Gardner et al. 2018) and use their default values not otherwise specified.
B PROOF OF PROPOSITION
This proof makes claim to novelty, but is provided for completeness.
Proof of proposition) 7 We can show that this follows from the fact that the metric spaces I" and R" are not homeomorphic (topologically equivalent) to one another (Brouwer 1911). While this statement is extremely intuitive, proving it is surprisingly technical, we will take it as given.
First, we prove the following lemma.
Lemma 1. Let f X + Y be a bi-Lipschitz and onto function, so 1.11z1 -zllx & l11( f(z2)lly & L||a1 x2llx, and Y the image of X under f. Then f a homeomorphism between X and Y, and so X and Y are homeomorphic.
Proof. Recall that a function f is a homeomorphism f is bijective, f is continuous, and f is also continuous. We will address these in turn. To see that bijective, note that f is injective iff. VII, I2 E X, we have Iı o 22 = f(x1) + f(x2). But this follows directly from the lower Lip- schitz property of f, since if #1 + T2, then 1 121--2le T22l it follows that f(z1) * f(z2). Since f is injective (one-to-one) and onto, it is a bijection. Since any function which is Lipschitz continuous also continuous, the fact that f is continuous given. Since is bijective, the inverse function f-1 exists, and we need to show that it is continuous. We have, from the bi-Lipschitzness of f, that which implies that the inverse function is also Lipschitz, and hence also continuous. So is homeomorphism, and we are done.
so [1]()))) f(x2))llx > from which
(fx))( S [If((x)) - f(x2)lly,
We have therefore established that if a bi-Lipschitz (and onto) function exists between two spaces X and Y, then X and Y are homeomorphic. This provides a proof of our claim by contradiction: if function f Rn + _ existed and bi-Lipschitz, then we would have shown that R² and R were homeomorphic, since any bi- bi-Lipschitz function is a homeomorphism. But R and R. are not homeomorphic unless m n, so no such function can exist.
C ABLATION OF NUMBER OF INDUCING POINTS
In Ta we show the results of increasing the number of inducing points, While more inducing points generally helps, the differences are minimal.
Table 3: Accuracy on CIFAR-I0 for increasing number of inducing points (k). The results are an average runs and reported with standard deviation.
k:
Accuracy (%)
10
96.
20
96.07+0.12
50
96.17+
100
96.25+0.088
D SAMPLING FROM THE JOINT POSTERIOR
In Figure we show samples from the joint predictive posterior. This type of sampling is not possible using standard deep learning models, because there is no covariance modelled between the batch elements. Possible applications of this include batch active learning, where we want to estimate a joint posterior (Kirsch et al. 2019), but also for deep exploration RL where we can effectively sample full policy (Osband et al. 2016).
14

---

Under review as a conference paper at ICLR 2021
0.20
0.15
0.10
0.05
0.00
constant
Figure 4: The same setup in Tb but instead of visualizing the uncertainty directly, we vi- sualize samples from the joint posterior. As expected, the functions become significantly less smooth away from the data as expected using the Matérn v 2 kernel.
Figure 5: A density of the Lipschitz values in batch normalization layers, averaged across 15 WRN models that trained with Soft- max output and without Spectral Normal- ization (exactly following Zagoruyko & Ko- modakis (016)). We see that many of the constants are significantly above 1, invalidat- ing the claim that Batch Normalization re- duces the Lipschitz constant of the network.
5.0
-25
0
-5.0
-10.0
-4
-225
-15.0
-15
-10
-5
5
10
15
(a) Deep Ensembles
(b) vDUQ
Figure 6:
The same setup as in Figure
lb
but with noise sampled from N (00, 0.5) instead of
N (0, 0.05).
15


Reviews:
[{"review_id": "nSBM9tmba1t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\n##### Summary:\n\nThis paper proposes variational deterministic uncertainty quantification (vDUQ), which adopts the stochastic (sparse) variational deep kernel learning (DKL) method to enable uncertainty estimations for deep models. To avoid uncertainty collapse, the deep neural network in the GP kernel is regularized with spectral normalization, which ensures a bi-Lipschitz constraint. Experiments show that vDUQ is effective in uncertainty quantification tasks.\n\n##### Reasons for score:\n\nThe idea is clear and the paper is easy to follow. However, my major concern is about the significance of the contribution and the experimental results. See my detailed comments below.\n\n---\n##### Pros:\n\nThe idea of obtaining uncertainty estimations with DKL is interesting. Moreover, by using sparse variational inference in DKL, the entire model is just like a DNN with an extra “layer of inducing points”, requiring only a few extra parameters and computational cost, which is also desired. Overall, the paper is well-written. The figures are instructive and helpful for understanding.\n\n---\n##### Concerns:\n\nMy main concern is about the significance of the contributions. Sparse variational inference methods for DKL was previously proposed in Wilson et al. 2016a. The main contribution of this paper seems to be the idea of introducing the spectral normalization regularization to stabilize the training of DKL and avoid uncertainty collapse. Although this is an interesting idea, I think the authors did not provide clear enough explanations and rigorous analyses. \n\nIn Page 3 the authors mentioned “without the spectral regularization on the deep model, the deep network is free to map data points that are far away from the training distribution to a feature representation that’s similar to in distribution data”. This perhaps explains how uncertainty collapse happens in out-of-distribution data to some extent, but is this the primary cause of undesired uncertainties?\n\nSince the parameters of NN become the parameters of the GP prior (as mentioned in Section 2), optimizing the marginal likelihood or the ELBO w.r.t the variational parameters and the NN parameters is actually “fitting the prior to data”, which could also cause biased uncertainties [1]. Although the bi-Lipschitz property can intuitively alleviate the biases, it is not clearly explained how it works. It would be better to provide more theoretical analysis.\n\nIn Section 3.1, the authors raised a question about how informative the “distance preserving explanation” is about the motivation of using bi-Lipschitz regularization. However, a more informative explanation is not provided. Also, the last paragraph of Section 3.1 is misleading. The author mentioned “A complete, rigorous theory … is not remains an open question.” If it is addressed and has theoretical insights into the use of spectral normalization, the authors should add necessary references and explanations.\n\n[1] Salimbeni, Hugh, and Marc Deisenroth. \"Doubly stochastic variational inference for deep Gaussian processes.\" Advances in Neural Information Processing Systems. 2017.\n\n\nMinors:\n(1) In Table 1, the results of vDUQ and DUQ is outperformed by the ensemble method in terms of both accuracy and AUROC. This seems a different conclusion form the results in Amersfoort et al. 2020. It would be better to provide more discussion about it, which seems not to be expected.\n\n(2) The authors claim the vDUQ can be trained in an end-to-end fashion in Section 3. However, since the inducing points are initialized with k-means algorithms that need to look at the training data, which I think is still a (lightweight) pre-training.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aNeCIM5eszA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n1.\tIn the introduction, the author separately pointed out the issues of DUQ and DKL. However, these issues are not convincing as no citations or theoretical proof is provided in this paper. The notations in the intro are also not well-defined. X, x, x* are used without difference, which however should be clearly defined as vectors or matrices.\n2.\tThe technical contribution is very incremental. The proposed vDUQ is simply applying the inducing point GP in the DUQ to mitigate the uncertainty collapse in DKL. The ‘’inducing point variational approximation of the GP predictive distribution’’ referred as inducing point GP is not clear for me. What exactly does ‘inducing point GP’ refer to? Why the so-called inducing point GP can speed up inference in GP model? What does ‘’decouple it from dataset size’’ mean? All these important points are not clarified in the introduction.\n3.\tThe theoretical contributions are also not well-organized. The author fails to prove that the spectral normalization as a regularization scheme can be uses to mitigate uncertainty collapse. Moreover, how the spectral normalization guarantees the effectiveness of inducing point GP in vDUQ?\n4.\tI also have some concerns on the experimental results of causal inference. Why the treatment effect estimation has uncertainty settings. The authors should fully explain the uncertainty settings in causal inference, as most of the causal baselines are not proposed for uncertainty settings.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EdUrRBJMYFl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Variational deterministic uncertainty quantification\n\nSummary:\nThe paper proposes a method for out-of-distribution detection by combining deep kernel learning and Gaussian processes. Using neural networks as a kernel for the GP as well as inducing point approximation alleviates the scalability issues of GP. The idea itself has merits, however, the presentation and experiments are not convincing.\n\nStrengths: The idea of using deep kernels within GP is a good solution that allows benefiting from both the expressiveness of the kernels and uncertainty estimates for GP. Additionally, using the uncertainty estimates for causal inference is a nice application. \n\nWeaknesses: Although the approach is interesting it needs to be further developed and evaluated in multiple setups. I find it limiting that it relies on the residual connection, making it unsuitable for other NN architectures, which means it will apply to only a limited number of tasks.\n\n\nThe presentation of the method should be better structured. I appreciate the background on deep kernels and how it helps to overcome the limits of GP, however, there is a lack of presentation of the method itself. A description, algorithmic listing or even an equation for the uncertainty score proposed is missing in the current version of the text.\n\nIn the introduction vUQD is presented as favorable wrt UQD due to its rigorous probabilistic interpretation, however, this was never further analyzed in the text. Also, seems that the method is concerned only with the epistemic uncertainty in the data? In general, the whole presentation of related work and positioning of this paper in the uncertainty literature is not clear. What source of uncertainty does the method address? There is much to be elaborated on this topic and I believe the discussion on this will significantly improve the paper.\n\nThe discussion on spectral-normalization and bi-Lipschitz in 3.1\nPlease clarify it or explain it better, in the current writing it is contradicting the proposed method:\n“A complete, rigorous theory of why the spectral normalization as used in this and previous work is a useful regularization scheme is not remains an open question”\n\nExperiments:\n\nToy examples:\nFigure 1 - on regression, I do not find this example motivating, first, why choosing noiseless data? Second, why is the vUQD increasing in reasons where there is data (such as the peaks?) Why does it compare only to deep ensembles?\nFigure 2 - Why choosing a toy example where a linear classifier works in the original space? \n\nWhat is the sensitivity to the number of inducing points for the GP? An ablation study at least for the toy data sets can help. \n\nWhy were standard datasets such as MNIST and fashion MNIST not included?\nThe empirical evaluation should be extended with more baselines and datasets.\n\n\nMinor:\nThe manuscript needs proofreading, language errors increase increasingly towards the conclusion. \n\n------------- Update after reading authors response -------------\n\nI thank the authors for their detailed responses, they have answered most of my concerns and I raise my score to 5. I am still not convinced about the method covering both the aleatoric and epistemic uncertainties, without any theoretical or intuitive justification, and without any discussion/clarification on that part. If indeed this is the case, then additional experiments should be included, for example for a regression task, the standard UCI datasets [1].\n\n[1] Hernandez-Lobato, J M and Adams, R P. Probabilistic ´\nbackpropagation for scalable learning of bayesian neural networks. In ICML-15, 2015.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8FF6w4XXhkD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a single deep deterministic model harnessed with spectral normalization to improve uncertainty in regression and classification tasks. The method relies on deep kernel learning + inducing point approximation for GP, and spectral regularization of the deep model to avoid uncertainty collapse in OOD.\n\nThe main contribution of this paper is methodological. The paper has extensive simulations, and demonstrates the utility of the proposed approach in a wide-range of applications for both regression and classification. Having said that, the proposed approach can be seen as a modification to [Liu et.al, 2020], with different approximation and different regularization. In that sense, the novelty of the paper could be seen as modest, and a comparison with [Liu et.al, 2020] highlighting the differences in practice is missing.\n\nMore comments:\n\n* What is the main advantage of this approach w.r.t [Liu et.al 2020]? How do these compare in terms of uncertainty estimation and computational speed? This should be included. The authors discuss [Liu et.al, 2020] in related work, highlighting the important difference that [Liu et.al, 2020] is a parametric model, but it is similar in that both formulate it as a GP and regularize for distance-awareness. A comparison with [Liu et.al, 2020] would strengthen this paper considerably.\n\n* Figure 1 could be improved: it only shows deep ensembles as baseline, how about the other approaches discussed in the paper? Moreover, it is unclear whether vDUQ provides better in-between uncertainty compared to Deep Ensembles (similar width, but deep ensembles interpolation is more smooth.\n\n* A deeper focus on the normalization, i.e., theoretical or empirical properties of spectral normalization and comparison with other normalization schemes would make the paper more interesting.\n\n* The paper has some strong/categorical sentences with which we do not agree: e.g., first intro paragraph: \"there is no single method that works on large datasets ...\" [Liu et.al, 2020] would be such method for example.\n\n* Simulation results are convincing in terms of utility, as the authors demonstrate that the proposed approach works in high-dimensional big datasets, and meaningful applications such as causal inference for healthcare. Yet, the experiments miss the point of elucidating how much spectral normalization compared to other normalization schemes.\n\n* Could the authors include an ablation study showing the impact of the number of inducing points in the approximation? The authors mention as a strength that a low number of inducing points is good enough, so showing evidence for that would strengthen the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that the paper represent thorough work but also is closely related to existing literature. (All referees point to other non-overlapping literature so it is a crowded field the authors have entered.) The amount of novelty (needed) can always be discussed but given the referees unanimous opinion and knowledgable input it is better for this work to be rejected for this conference. Using this input can make this work a good paper for submission elsewhere. 
Decision: Reject