Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: 
1.	In the introduction, the author separately pointed out the issues of DUQ and DKL. However, these issues are not convincing as no citations or theoretical proof is provided in this paper. The notations in the intro are also not well-defined. X, x, x* are used without difference, which however should be clearly defined as vectors or matrices.
2.	The technical contribution is very incremental. The proposed vDUQ is simply applying the inducing point GP in the DUQ to mitigate the uncertainty collapse in DKL. The ‘’inducing point variational approximation of the GP predictive distribution’’ referred as inducing point GP is not clear for me. What exactly does ‘inducing point GP’ refer to? Why the so-called inducing point GP can speed up inference in GP model? What does ‘’decouple it from dataset size’’ mean? All these important points are not clarified in the introduction.
3.	The theoretical contributions are also not well-organized. The author fails to prove that the spectral normalization as a regularization scheme can be uses to mitigate uncertainty collapse. Moreover, how the spectral normalization guarantees the effectiveness of inducing point GP in vDUQ?
4.	I also have some concerns on the experimental results of causal inference. Why the treatment effect estimation has uncertainty settings. The authors should fully explain the uncertainty settings in causal inference, as most of the causal baselines are not proposed for uncertainty settings.

review_title: This paper proposed a new vDUQ (variational deterministic uncertainty quantification) that combines the inducing point GP with Deep Kernel Learning so as to obtain predictive uncertainty in deep learning for both classification and regression problems. This paper 1.	This paper is not well-written and fails to meet the ICLR standard:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a single deep deterministic model harnessed with spectral normalization to improve uncertainty in regression and classification tasks. The method relies on deep kernel learning + inducing point approximation for GP, and spectral regularization of the deep model to avoid uncertainty collapse in OOD.

The main contribution of this paper is methodological. The paper has extensive simulations, and demonstrates the utility of the proposed approach in a wide-range of applications for both regression and classification. Having said that, the proposed approach can be seen as a modification to [Liu et.al, 2020], with different approximation and different regularization. In that sense, the novelty of the paper could be seen as modest, and a comparison with [Liu et.al, 2020] highlighting the differences in practice is missing.

More comments:

* What is the main advantage of this approach w.r.t [Liu et.al 2020]? How do these compare in terms of uncertainty estimation and computational speed? This should be included. The authors discuss [Liu et.al, 2020] in related work, highlighting the important difference that [Liu et.al, 2020] is a parametric model, but it is similar in that both formulate it as a GP and regularize for distance-awareness. A comparison with [Liu et.al, 2020] would strengthen this paper considerably.

* Figure 1 could be improved: it only shows deep ensembles as baseline, how about the other approaches discussed in the paper? Moreover, it is unclear whether vDUQ provides better in-between uncertainty compared to Deep Ensembles (similar width, but deep ensembles interpolation is more smooth.

* A deeper focus on the normalization, i.e., theoretical or empirical properties of spectral normalization and comparison with other normalization schemes would make the paper more interesting.

* The paper has some strong/categorical sentences with which we do not agree: e.g., first intro paragraph: "there is no single method that works on large datasets ..." [Liu et.al, 2020] would be such method for example.

* Simulation results are convincing in terms of utility, as the authors demonstrate that the proposed approach works in high-dimensional big datasets, and meaningful applications such as causal inference for healthcare. Yet, the experiments miss the point of elucidating how much spectral normalization compared to other normalization schemes.

* Could the authors include an ablation study showing the impact of the number of inducing points in the approximation? The authors mention as a strength that a low number of inducing points is good enough, so showing evidence for that would strengthen the paper.
review_title: Extensive simulations, modest novelty compared to [Liu et.al, 2020], a key baseline that is missing

final_score: rating: 5: Marginally below acceptance threshold
review_content: ---
##### Summary:

This paper proposes variational deterministic uncertainty quantification (vDUQ), which adopts the stochastic (sparse) variational deep kernel learning (DKL) method to enable uncertainty estimations for deep models. To avoid uncertainty collapse, the deep neural network in the GP kernel is regularized with spectral normalization, which ensures a bi-Lipschitz constraint. Experiments show that vDUQ is effective in uncertainty quantification tasks.

##### Reasons for score:

The idea is clear and the paper is easy to follow. However, my major concern is about the significance of the contribution and the experimental results. See my detailed comments below.

---
##### Pros:

The idea of obtaining uncertainty estimations with DKL is interesting. Moreover, by using sparse variational inference in DKL, the entire model is just like a DNN with an extra “layer of inducing points”, requiring only a few extra parameters and computational cost, which is also desired. Overall, the paper is well-written. The figures are instructive and helpful for understanding.

---
##### Concerns:

My main concern is about the significance of the contributions. Sparse variational inference methods for DKL was previously proposed in Wilson et al. 2016a. The main contribution of this paper seems to be the idea of introducing the spectral normalization regularization to stabilize the training of DKL and avoid uncertainty collapse. Although this is an interesting idea, I think the authors did not provide clear enough explanations and rigorous analyses. 

In Page 3 the authors mentioned “without the spectral regularization on the deep model, the deep network is free to map data points that are far away from the training distribution to a feature representation that’s similar to in distribution data”. This perhaps explains how uncertainty collapse happens in out-of-distribution data to some extent, but is this the primary cause of undesired uncertainties?

Since the parameters of NN become the parameters of the GP prior (as mentioned in Section 2), optimizing the marginal likelihood or the ELBO w.r.t the variational parameters and the NN parameters is actually “fitting the prior to data”, which could also cause biased uncertainties [1]. Although the bi-Lipschitz property can intuitively alleviate the biases, it is not clearly explained how it works. It would be better to provide more theoretical analysis.

In Section 3.1, the authors raised a question about how informative the “distance preserving explanation” is about the motivation of using bi-Lipschitz regularization. However, a more informative explanation is not provided. Also, the last paragraph of Section 3.1 is misleading. The author mentioned “A complete, rigorous theory … is not remains an open question.” If it is addressed and has theoretical insights into the use of spectral normalization, the authors should add necessary references and explanations.

[1] Salimbeni, Hugh, and Marc Deisenroth. "Doubly stochastic variational inference for deep Gaussian processes." Advances in Neural Information Processing Systems. 2017.


Minors:
(1) In Table 1, the results of vDUQ and DUQ is outperformed by the ensemble method in terms of both accuracy and AUROC. This seems a different conclusion form the results in Amersfoort et al. 2020. It would be better to provide more discussion about it, which seems not to be expected.

(2) The authors claim the vDUQ can be trained in an end-to-end fashion in Section 3. However, since the inducing points are initialized with k-means algorithms that need to look at the training data, which I think is still a (lightweight) pre-training.


review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: Variational deterministic uncertainty quantification

Summary:
The paper proposes a method for out-of-distribution detection by combining deep kernel learning and Gaussian processes. Using neural networks as a kernel for the GP as well as inducing point approximation alleviates the scalability issues of GP. The idea itself has merits, however, the presentation and experiments are not convincing.

Strengths: The idea of using deep kernels within GP is a good solution that allows benefiting from both the expressiveness of the kernels and uncertainty estimates for GP. Additionally, using the uncertainty estimates for causal inference is a nice application. 

Weaknesses: Although the approach is interesting it needs to be further developed and evaluated in multiple setups. I find it limiting that it relies on the residual connection, making it unsuitable for other NN architectures, which means it will apply to only a limited number of tasks.


The presentation of the method should be better structured. I appreciate the background on deep kernels and how it helps to overcome the limits of GP, however, there is a lack of presentation of the method itself. A description, algorithmic listing or even an equation for the uncertainty score proposed is missing in the current version of the text.

In the introduction vUQD is presented as favorable wrt UQD due to its rigorous probabilistic interpretation, however, this was never further analyzed in the text. Also, seems that the method is concerned only with the epistemic uncertainty in the data? In general, the whole presentation of related work and positioning of this paper in the uncertainty literature is not clear. What source of uncertainty does the method address? There is much to be elaborated on this topic and I believe the discussion on this will significantly improve the paper.

The discussion on spectral-normalization and bi-Lipschitz in 3.1
Please clarify it or explain it better, in the current writing it is contradicting the proposed method:
“A complete, rigorous theory of why the spectral normalization as used in this and previous work is a useful regularization scheme is not remains an open question”

Experiments:

Toy examples:
Figure 1 - on regression, I do not find this example motivating, first, why choosing noiseless data? Second, why is the vUQD increasing in reasons where there is data (such as the peaks?) Why does it compare only to deep ensembles?
Figure 2 - Why choosing a toy example where a linear classifier works in the original space? 

What is the sensitivity to the number of inducing points for the GP? An ablation study at least for the toy data sets can help. 

Why were standard datasets such as MNIST and fashion MNIST not included?
The empirical evaluation should be extended with more baselines and datasets.


Minor:
The manuscript needs proofreading, language errors increase increasingly towards the conclusion. 

------------- Update after reading authors response -------------

I thank the authors for their detailed responses, they have answered most of my concerns and I raise my score to 5. I am still not convinced about the method covering both the aleatoric and epistemic uncertainties, without any theoretical or intuitive justification, and without any discussion/clarification on that part. If indeed this is the case, then additional experiments should be included, for example for a regression task, the standard UCI datasets [1].

[1] Hernandez-Lobato, J M and Adams, R P. Probabilistic ´
backpropagation for scalable learning of bayesian neural networks. In ICML-15, 2015.
review_title: Interesting idea that needs polishing in terms of presentation and empirical evaluation 

Meta-review: The reviewers all agreed that the paper represent thorough work but also is closely related to existing literature. (All referees point to other non-overlapping literature so it is a crowded field the authors have entered.) The amount of novelty (needed) can always be discussed but given the referees unanimous opinion and knowledgable input it is better for this work to be rejected for this conference. Using this input can make this work a good paper for submission elsewhere. 
Decision: Reject