Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new model HetSNGP that captures distance-aware model uncertainties as well as heteroscedastic data uncertainties. HetSNGP combines these two types of uncertainty in a beneficial way, allowing for out-of-distribution detection and generalisation as well as in-distribution performance and calibration on a variety of benchmark datasets.
main_review: Strengths:
This paper is well-motivated. Uncertainty estimation in deep learning has lately been a hot topic for improving reliability and resilience in safety-critical applications. This paper combines the complementary benefits of these two types of uncertainty modeling and achieves a good calibration performance.

Concerns:
1.	The proposed method Is a combination of well-known techniques. The novelty of this paper Is limited.
2.	The proposed method Is a little bit too heuristic. There are many assumptions which are not sufficiently justified. This may limit the scope of application of the method.
3.	The experiment results are weak. The proposed method cannot surpass baseline consistently.
4.	This paper is hard to understand. 

summary_of_the_review: Overall, I think this is a well-motivated work but the novelty and contributes are limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new method for model and data uncertainty estimation in deep neural networks combining the heteroscedastic method (Collier et al. 2020) and the Spectral-Normalised Gaussian Process (SNGP) method (Liu et al. 2020). It is shown that the two methods are complementary and their combination outperforms state-of-the-art methods on Out-Of-Distribution (OOD) detection on common image classification benchmarks. 

main_review: The paper is well written and the scientific contributions are clearly formulated and experimentally validated.
The experimental results look quite convincing and show the benefits of this combined uncertainty model.

There are many technical details that are quite hard to understand and that seem to originate from other papers from the literature, notably in Section 3.3 (computational approximations). 
This makes the paper difficult to follow.
Also, it seems that the experimental evaluation is missing a comparison with some more existing methods for uncertainty estimation and OOD (like the ones mentioned in Section 5, "Related Work").

Overall, I still think the paper is interesting and of good quality.

summary_of_the_review: The proposed approach is scientifically sound, the presentation is of good quality. 
But the experimental results may lack some SOTA comparison.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method that can jointly learn the model uncertainty for out-of-distribution detection and data uncertainty for in-distribution calibration. The effectiveness of their method has been validated on different out-of-distribution datasets.
main_review: strengths
+ How to build a model can jointly mode the model and data uncertainty is an interesting problem.
+ A method is proposed that has a good calibration performance. It can be used for out-of-distribution detection. 

weakness
+ The technical contribution of this paper may not be enough. Specifically, algorithms in this paper seem to be similar to SNGP [1]. I think that the major difference between the two methods comes from the different data generative processes.
+ The assumption used in this paper might be strong. The authors propose a hierarchical model of two latent random variables which are sampled from two Gaussian distributions. It is better to add more explanations that how realistic this assumption is. I think that the proposed method could be more useful if this assumption can be mitigated. Additionally, different approximations are made to ensure the tractability of distributions. Although some of them are commonly used, it is better to add more discussions to show that these approximations will not lead to large biases. 
+ The experimental advantage of the proposed method is not large. Table 2, Table 3, and Table 4 show that the performance of the proposed method and the baseline method SNGP are quite similar.  
The baselines considered are not that inclusive. I understand this problem is novel and there hasn't been fruitful literature. But I think it is better to add more calibration methods as baselines.

[1]. Liu, Jeremiah Zhe, et al. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness." arXiv preprint arXiv:2006.10108 (2020).
summary_of_the_review: Overall, I think this work is well motivated. But I have some concerns for both technical and empirical novelties.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the combination between model uncertainty and data uncertainty based on spectral-normalized Gaussian process. Theoretical results show that heteroscedastic SNGP allows for joint modeling of model and data uncertainties. The paper then proposes an approximate inference scheme for efficient model training. Experiment results support these claims on synthetic datasets and typical classification benchmarks. 
main_review: The first contribution, that jointly model two different types of uncertainty, as far as I know,  make the first step for noisy label. The second contribution, amounts to large-scale matrix computation approximation, makes the whole method be accessible. However，the contribution of the whole paper can not meet the requirement of ICRL as (1) limited novelty. The proposed method is a limited improvement of SNGP. (2) The computation approximation all come from the existing techniques [1,2,3]. (3) For joint uncertainty estimation setting, it is not very novel and easily to be solved. We can use methods in [1,3] to handle the data uncertainty, then use the method in [2] to handle model uncertainty.

(1)	Experimental results. Compared to the baseline model SNGP, the proposed method uses more complex method but achieve slight performance promotion even performance reduction. Only using the ensemble strategy can bring stable performance promotion (Table 4).  More discussion is needed.
(2)	For computation approximation, four types of approximations are used, the performance and time yield should be discussed in the experimental section. For practicable, the author can verified the algorithm (w/wo approximation) on small datasets.
(3)	In experimental section, the evaluation metrics are not clear. 

[1] Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. Correlated
input-dependent label noise in large-scale image classification. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 1551–1560, 2021.
[2] Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via
distance awareness. arXiv preprint arXiv:2006.10108, 2020
[3] Mark Collier, Basil Mustafa, Efi Kokiopoulou, Rodolphe Jenatton, and Jesse Berent. A simple
probabilistic method for deep classification under input-dependent label noise. arXiv preprint
arXiv:2003.06778, 2020.

summary_of_the_review: The jointly model data and model uncertainty seem some novel. However, the contribution is incremental when compared to SNGP. The experimental analysis is not sufficient and strong. Overall, I tend to reject this paper.
review_title: null

Meta-review: This paper studies the combination between model uncertainty and data uncertainty based on the spectral-normalized Gaussian process. Empirical results show the effectiveness of the proposed method. Overall, the paper is well-motivated and well-written. However, there are several concerns about the paper. (1) The novelty is marginal. The contribution of combining SNGP and heteroscedastic models into a single model may not be enough. (2) More analyses and insights are needed on why the mentioned two types of uncertainty are complementary. (3) More recent state-of-the-art methods on classification with noisy labels are suggested to be included to interest the readers. There are diverse scores. However, no one wants to champion the paper. We believe that the paper will be a strong one by addressing the concerns.
Decision: Reject