Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for uncertainty estimation of a classifier network called “KLoSNet”. The KLoS metric is designed to measure both the uncertainty between classes and the probability of out-of-domain samples. For a given sample, the estimated KLoS value is predicted from a secondary network that is trained to predict the true KLoS score of the first since ground truth class information is not available during test time.
The method is evaluated on a standard set of benchmarks and shows good performance with respect to recent work.
main_review: The paper shows good results and an extensive evaluation on various datasets and two (smaller) network architectures. Additionally, a toy dataset showcases the differences between typical measures and the proposed KLoS well.
There are three main weaknesses:

#### W1 - Baseline Comparisons
To my understanding from the appendix, KLoSNet is trained in three stages. First a standard classifier is trained based on (Joo et al., 2020) which corresponds to Eq 6. Then, the weights are frozen and a 5 layer MLP is trained to predict KLoS* from the final layer of the original network. Finally, the whole network is finetuned.
This procedure makes it difficult to compare the performance in Tab. 1, as the baselines do not receive the extra training, which potentially conflates the performance gain through longer training with the benefits of the proposed method.

#### W2 - Joint Metric
KLoS is designed as a joint metric for classification confidence and OOD detection. This requires combining the two concepts into one score by implicitly weighing them. It is not clear that this step is meaningful or desired. It might be beneficial to know whether a prediction is uncertain because of class uncertainty or because the sample is far from the distribution. 

#### W3 - Ablation
While some hyperparameters such as the oversampling rate $\kappa$ are ablated, some important experiments are missing: 

The choice of the prototype Dirichlet distribution $\gamma_{y}$ is potentially an important hyper parameter. It would be interesting to understand the influence of the choice of precision $\alpha$ as this controls the ‘smoothing’ of the actual target.

The proposed training scheme relies on initialization of KLoSNet with the original classifier. Is this important to learn a proper measure or does this simply speed up convergence?

#### W4 -  Minor Concerns
Secon 4.1 states that the Mahalanobis metric ``may have trouble detecting misclassifications’’, however the results in Tab.1 do not seem to confirm this explanation. 

summary_of_the_review: The design of the evaluation makes it difficult to understand exactly where the performance improvement comes from (W1) as the proposed method seems to be trained ~50% longer than the baselines. Additionally, I am not fully convinced that a joint metric for OOD and misclassification is desired (W2). Finally, some ablation experiments are needed to understand the effect of certain design choices (W3).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an uncertainty measurement KLoS by computing the KL divergence between the Dirichlet distribution from the Evidential Neural Networks and a class-wise prototype concentrated on the predicted class. An extra network KLoSNet is further trained to align with the evidential training objective. Empirical studies are performed to show the efficacy of both OOD uncertainty estimation and ID misclassification detection. Results are shown on the task of (1) synthetic dataset with three Gaussian-distributed classes and (2) CIFAR datasets. 


main_review: +ves: 

+ The idea of using a class-wise divergence measure based on evidential models is interesting. 

+ The paper has a nice presentation overall. The figures are effective in conveying the key idea and intuition. 

+ The results section is well structured. It's nice to see both OOD detection and ID misclassification performance. 

Concerns: 

- Authors highlight that the uncertainty estimation should consider both class confusion and lack of evidence. However, this may not be always desirable for OOD uncertainty estimation. Take an extreme example, in the classification task of "bird" vs "car", given images with "A: bird sitting on the car" and "B: computer", the model would have more class confusion in the former case. Yet A is an in-distribution image and B is an OOD image. 

- The above concern brings to my next question: have the authors investigated the efficacy of the two individual terms in Equation (6) for OOD uncertainty estimation?

- "KLoS also penalizes samples having a different precision α_0. " I understand that lower precision should be penalized but why a higher amount of evidence is not preferred?

- The experimental results are based on relatively small model capacity (ResNet-18 at the largest). It is unclear how well the method scale for common capacity (say ResNet-101). This is particularly concerning, as Mahalanobis (the best baseline considered) is relatively disadvantaged in this small-capacity setting where the representation might not be as optimal as larger models. Even in this setting, the results are not conclusively in favor of the proposed method, and only is marginally better than Mahalanobis. With that being said, the results would have been more complete if a consistent trend is shown at a larger model capacity. 

- Missing  OOD detection baselines Gram Matrix [1],  Generalized ODIN [2]. 

- Lastly, it'd be great if the authors can comment on the low performance of common baselines in Table 1. For example, the reviewer uses a standard ResNet-18 trained on CIFAR-100, using the same learning rate and schedule as detailed on Page 16. The OOD uncertainty estimation performance (on LSUN) is not matchable to Table 1. For example, using MCP yields AUROC 75.73, using ODIN and Mahalanobis yields 82.13 and 82.98 respectively. In contrast, the performance for the baselines methods seems low across the board (~70-75%). 


References

[1] C. Sastry, S. Oore. Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices. ICML 2020.

[2] Y. Hsu, Y. Shen, H. Jin, Z. Kira. Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data. CVPR 2020. 

summary_of_the_review: In summary, this paper presents an interesting idea however the experimental evaluations are not sufficiently convincing in the current state. I would like to hear the authors address my concerns raised above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper suggests training an auxiliary DNN to predict the uncertainty associated with a given example. The auxiliary model uses the predictions of a model fit to the predict the concentration parameters of a Dirichlet (an "evidential neural network") to measure the KL divergence between the predicted posterior and one based on the ground truth labels. (The auxiliary model is necessary because the ground truth labels are not available at test time.) The primary claim is that "[...] leveraging the second-order uncertainty representation that evidential models provide, KLoS captures both class confusion and lack of evidence in a single score." In a series of experiments, it is found that the proposed model is competitive with or outperforms selected alternate models across several benchmarks for OOD detection and misclassification detection.
main_review: The \lambda term seems to be an important hyper-parameter but it was unclear (unless I missed it!) how it was tuned or how sensitive it is.

It would have been helpful to frame the proposal in terms of standard notions of epistemic (model) and aleatoric (irreducible) uncertainty. If the proposed measure is meant to capture epistemic uncertainty, would we expect this value to be greater on average on held-out data, for smaller training datasets? If so, it'd be nice to show that via experiments training with different amounts of data and showing this value exhibits the desired trends.

Overall, while the experiments suggest that the proposed method is doing something reasonable, it's not clear to me that it's doing something reasonable for the purported reasons. Besides the other suggestions above, some additional model ablations may be helpful to demonstrate the benefits of the proposed approach, such as auxiliary regression of simplified versions of the proposed objective.

Separate from my concerns about the justification for the method, I also found there to be a lack of references to related recent work and I think the paper would be more convincing with consideration of stronger baselines. Experimental comparisons aren't strictly necessary in all cases, but at minimum further discussion would be help position this work relative to other deterministic approaches, such as:

- The Mahalanobis baseline is competitive with the proposed approach, but more recent, improved versions of Mahalanobis have been proposed for OOD: https://arxiv.org/pdf/2106.09022.pdf
- Deep deterministic uncertainty (DDU): https://arxiv.org/pdf/2102.11582.pdf
- MIR, see: https://arxiv.org/pdf/2107.00649.pdf
- Deterministic uncertainty quantification (DUQ): https://arxiv.org/pdf/2003.02037.pdf


summary_of_the_review: Overall, this paper proposes an interesting and to my knowledge novel approach to uncertainty quantification. However, I believe it has two main weaknesses: (1) there's a lack of discussion and experimental comparisons to recent work in uncertainty estimation and (2) there isn't enough introspection of the proposed approach to confirm it's working for the purported reasons.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the problem of doing out-of-domain and misclassification detection using the evidential learning framework, which suggests modelling the second-order uncertainties of class assignment probabilities with a Dirichlet-Multinomial hierarchical model. The model maps an input observation, be it an image, to the concentration parameters of the Dirichlet prior on the class probabilities by a neural net with an arbitrary architecture. On top of this existing framework (Sensoy et al., 2018, Malinin & Gales 2018), the paper proposes to use a KL divergence based decision criterion, called "KL on Simplex (KLoS)",  which fits a reverse KL divergence between the model output and a spike on the predicted class label in order to smoothen the predictions.
main_review: The paper provides nicely drawn illustrative figures that explain the impact of first-order and second-order uncertainties on uncertainty quantification profiles of classification models.

The paper suffers from multiple fundamental weaknesses detailed below:

i) The paper lacks a concrete hypothesis. Which particular problem of the "existing" evidential learning problem is being targeted? Figure 1 only motivates the evidential learning framework, but does not point to any weakness of it. The claim in Page 2 that the evidential models require auxiliary OOD data holds for the Prior Networks approach (Malinin & Gales, 2018, 2019) but not for EDL (Sensoy et al., 2018). As I understand, the only novelty claim is to use Eq 5 as an OOD and misclassification detection criterion instead of the well-established mutual information or entropy criteria. However, I am having hard time to decipher what the claimed limitation of the mutual information criterion is. All the argumentation in the paper is valid only against MCP, which is obvious for the uncertainty quantification community, but what is the problem with mutual information, which in essense has mechanisms for handling both in-domain and out-of-domain uncertainties? For example, the claim:

"To capture uncertainties due to class confusion and lack of evidence, an effective measure should account for the sharpness of the Dirichlet distribution and its location on the simplex." 

holds also for the mutual information criterion.

ii) The only section in the paper where there is a claim about a methodological novelty is Section 2.2. I am having hard time to make sense of the criterion proposed in Eq 5. It first calculates the Dirichlet density corresponding to an input pattern that captures both first and second-order uncertainties, then makes a heuristic choice of comparing it to a spiky Dirichlet distribution. Why should it be better than decomposing the uncertainty of the predictive distribution according to the law of total variance or any other decomposition criterion that has an information-theoretic meaning? I also do not understand why divergence from the spiky (winner-take-all) version of the own prediction of a model should correlate with the probability of an input sample coming from a different domain.

iii) The paper does not report experimental results that give a decisive outcome about the central claims of the paper. As I understand, all models in Table 1 are trained using Eq 6 first, which corresponds to the Reverse KL design of Malinin et al., 2019, just without OOD data. It is then blurry what exactly happens. Is then only KLoSNet trained further on Eq 8? If so, the comparison is unfair, as the proposed solution gains more training resources than its counterparts. If all other models are also trained further with Eq 8, the comparison is still unfair, because they are trained further with a loss that is designed for another OOD criterion than they are designed for. A decisive comparison could have been as follows:

a) Take EDL loss "literally". Train it for X epochs.

b) Take Reverse KL loss "literally". Train it for X epochs.

c) Take Eq 8 and train KLoSNet "without" any pretraining for X epochs. 

Evaluate all three models with respect to both mutual information and KLoS scores. Now compare which score and which model gives best OOD performance and draw the scientific claim accordingly.

The results in favor of KLoSNet reported in Table 1 may have something to do with post-training KLoSNet with respect to a criterion that is later used as a performance score. This would of course make both the trained model and the chosen criterion advantageous over others, but it would be the same if any other criterion was chosen.
summary_of_the_review: Overall, the paper lacks a clear hypothesis statement (Item i)), it does not propose a novel solution (Item ii)), and it does not conduct a set of experiments that give conclusive results in favor of a solid scientific hypothesis (Item iii)). Hence my initial grade is a clear reject.
review_title: null

Meta-review: This submission received 4 diverging ratings: 6, 5, 5, 3. On the positive side, reviewers appreciated the central idea and a quality manuscript. At the same time, they have raised important concerns around unfair comparisons with baselines, experiments not fully supporting the claims and lack of comparisons with some prior methods. After discussions with the authors most reviewers stayed with their original ratings. 
The AC agrees that the weaknesses in this case outweigh the strengths. The final recommendation is to reject.
Decision: Reject