Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper investigates the aleatoric uncertainty and epistemic uncertainty in machine learning. The evaluation was performed on benchmark regression tasks. The comparison with other state-of-the-art methods was provided. The evaluation of the robustness against out of distribution and adversarially perturbed test data was performed.

Strength:
1. Experiments were complete. Analyses were provided with useful information.
2. A model with smaller number of parameters was proposed.
3. Computation efficiency was improved.

Weakness:
1. Total evidence and model evidence were defined. The derivation of these evidences should be clarified.
2. Theoretical justification for related methods could be improved.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed deep evidential regression, a method for training neural networks to not only estimate the output but also the associated evidence in support of that output. The main idea follows the evidential deep learning work proposed in (Sensoy et al., 2018) extending it from the classification regime to the regression regime, by placing evidential priors over the Gaussian likelihood function and performing the type-II maximum likelihood estimation similar to the empirical Bayes method [1,2]. The authors demonstrated that the both the epistemic and aleatoric uncertainties could be estimated in one forward pass under the proposed framework without resorting to multiple passes and showed favorable uncertainty comparing to existing methods. Robustness against out of distribution and adversarially perturbed data is illustrated as well.

On the technical side, the novelty is incremental. The extension from the classification regime to the regression regime, from the conjugate Dirichet prior to the conjugate Normal-Inverse-Gamma prior, is quite straightforward. Besides, the presentation of the paper could be largely improved. It is not easy to follow the derivation in Section 3. The discussion of concepts and problem definitions look fragmented and incoherent. Even though the presentation largely follows (Sensoy et al., 2018) and uses terms from theory of evidence, the derivation actually is more aligned with the prior network [3] under the Bayesian framework which is missing from the references. It is really confusing that the authors talked about the variational inference when conjugate prior is used, and it is unclear how the variational distributions are used in Section 3.2 or how the "I don't know" loss term relates to the KL-divergence between the variational distribution and the prior in Section 3.3. This term was manually added as additional regularization to "prefer the evidence to shrink to zero for a sample if it cannot be correctly classified" in (Sensoy et al., 2018), and a different regularization was used to encourage distributional uncertainty in [3]. I hope that the authors could spend more efforts clarifying their ideas, especially the derivations in Section 3.2 and 3.3.

On the other hand, there is no referring to the input x in the entire derivation and problem formulation in Section 3. It took me a while to realize that the formulation in (4) actually defines the generation for a particular input, not for all the inputs. That is, the model is trying to model heteroscedastic uncertainty, not the homoscedastic counterpart. It could be better to call out the dependence on the input explicitly. 

On the quantitative side, the baseline models considered in Section 4 are mainly concerned with epistemic uncertainty estimation. So it would be good to explicitly discuss which uncertainty estimation was compared with. This work estimates both aleatoric and epistemic uncertainties, so a better comparison is to models that estimate both quantities (Kendall & Gal, 2017)[4] which has been shown to give better output estimation comparing to epistemic uncertainty estimation only (Kendall & Gal, 2017).

Other comments:
- What is the \pi in equation (8)?
- The "I don't know" loss introduced in Section 3.3 used L-p norm. What is the originality of the L-p norm here? In practice, which p value should be used? In the experiments, which p value was used?
- The RMSE results of the depth estimation presented in Table 2 are orders of magnitude smaller than those from existing work, for example Table 2(b) in (Kendall & Gal, 2017). Was a different RMSE computation used in this work?
- From the caption in Table 2, it seems that only 5 samples were used in MC-dropout, which is considerably smaller than those used in existing work (Kendall & Gal, 2017).

[1] D.J.C. MacKay. Hyperparameters: optimize, or integrate out? Maximum Entropy and Bayesian Methods, Springer 1996.
[2] B. Efron. Two modeling strategies for empirical Bayes estimation. Statistical Science, 2014.
[3] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. NeurIPS 2018.
[4] Y. Kwon, J.-H. Won, B.J. Kim, and M.C. Paik. Uncertainty quantification using Bayesian neural networks in classification: application to ischemic stroke lesion segmentation. MIDL 2018.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel approach to estimate the confidence of predictions in a regression setting. The approach starts from the standard modelling assuming iid samples from a Gaussian distribution with unknown mean and variances and places evidential priors (relying on the Dempster-Shafer Theory of Evidence [1] /subjective logic [2]) on those quantities to model uncertainty in a deterministic fashion, i.e. without relying on sampling as most previous approaches. This opens the door to online applications with fully integrated uncertainty estimates. 
This is a very relevant topic in deep learning, as deep learning methods are increasingly deployed in safety-critical domains, and I think that this works deserves its place at ICLR.

Pros:
1.	Novel approach to regression (a similar work has been published at NeurIPS last year for classification [3]), but the extension of the work to regression is important.
2.	The experimental results show consistent improvement in performance over a wide base of benchmarks, scales to large vision problems and behaves robustly against adversarial examples.
3.	The presentation of the paper is overall nice, and the Figures are very useful to the general comprehension of the article.
Cons:
1.	The theory of evidence, which is not widely known in the ML community, is not clearly introduced. 
I think that the authors should consider adding a section similar to Section 3 of Sensoy et al. [3] should be considered. Currently, the only step explaining the evidential approach that I found was in section 3.1, in a very small paragraph (between “the mean of […] to \lambda + 2\alpha.”). I believe that the article would greatly benefit from a more thorough introduction of concepts linked to the theory of evidence.
2.	The authors briefly mention that KL is not well defined between some NIG distributions (p.5) and propose a custom evidence regularizer, but there’s very little insight given on how this connects to/departs from the ELBO approach. 

Other comments/questions:
1.	(p.1)  I’m not sure to fully understand what’s meant by higher-order/lower-order distributions, could you clarify?
2.	(p.3) In section 3.1, the term in the total evidence \phi_j is not defined.
3.	(p.3) Could you comment on the implications of assuming that the estimated distribution can be factorized? 
4.	(p.4) Could you comment on the difference that there is between NLL_ML and NLL_SOS from a modelling perspective?
5.	(p.4) The ELBO loss (6) is unclearly defined, and not connected to the direct context. I would suggest moving this to the section 3.3, where the prior p(\theta) used in eq. (6) is actually defined.
6.	(p.4) In equation (6), p_m(y|\theta) isn’t defined, and q(\theta|y) is already parameterized on y if I understand that q(\theta)=p(t\heta|y1,…,yN). Making the conditioning explicit in equation (6) might make the connection to the ELBO clearer. 
7.	(p.7) I’m not sure to understand how the calibration of the predictive uncertainty can be tested by the ROC curves if both the uncertainty and estimates error are normalized. Could you also define more clearly what you mean by an “error at a given pixel”? 
8.	Spelling & typos:
-	(p.4) There are several typos in equation (8), where tau should be replaced with 1/\sigma^2. 
-	(p.8) In the last sentence, there is “ntwork” instead of network.
-	(p.9) There is a typo in the name of Jøsang in the references. 
-	(p.10) In equation (13), due to the change of variable, there should be a 
-(1/\tau^2) added;  
-	(p.10) In equation (14), the \exp(-\lambda*\pi*(…)) should be replaced with \exp(-\lambda*\tau*(…)). 

[1] Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Saiedeh N Razavi. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1):28–44, 2013. 
[2] Audun Jøsang. Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018. 
[3] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. "Evidential deep learning to quantify classification uncertainty." Advances in Neural Information Processing Systems. 2018.

review_title: Official Blind Review #1

Meta-review: This paper presents a method for providing uncertainty for deep learning regressors through assigning a notion of evidence to the predictions.  This is done by putting priors on the parameters of the Gaussian outputs of the model and estimating these via an empirical Bayes-like optimization.  The reviewers in general found the methodology sensible although incremental in light of Sensoy et al. and Malinin & Gales but found the experiments thorough.  A comment on the paper pointed out that the approach was very similar to something presented in the thesis of Malinin (it seems unfair to expect the authors to have been aware of this, but the thesis should be cited and not just the paper which is a different contribution).  In discussion, one reviewer raised their score from weak reject to weak accept but the highest scoring reviewer explicitly was not willing to champion the paper and raise their score to accept.  Thus the recommendation here is to reject.  Taking the reviewer feedback into account, incorporating the proposed changes and adding more careful treatment of related work would make this a much stronger submission to a future conference.
Decision: Reject