Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper performs isometrization of the distances in the IsoMax loss and replaces entropic score with the minimum distance score for  entropic out-of-distribution detection solution.
main_review: The method is simple and show effective in experiments.
The authors also provide insights into why their methods work.
limitations_and_societal_impact: As the overall method is simple and incremental, the novelty is limited.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose some marginal modifications to the IsoMax loss (i.e. normalising the logits and the prototypes, multiplying the distance between the logits and the prototypes by a non-negative scalar, using a different uncertainty score based on the distance from the prototypes). 

Positive aspects:
- the formulation makes sense, and the distance score is simple but reasonable

Unclear points:
- why is it necessary to initialize the prototypes randomly? Other papers like [1] show one can select the prototypes to make the training procedure stable.
- why should multiplying by a fixed scalar |d_s| (learned only at training time, but not depending on the input) have any effect on the performance?
Weaknesses:
- Novelty is marginal
- Experiments are performed with only one seed (need to repeat the experiments with at least 5 different seeds) 
- Does not compare against baselines with similar concepts [1, 2], and makes unfair comparisons with methods that leverage more information/have accuracy drops; recent literature has shown there are many ways to preserve accuracy and improve OOD detection [3,4,5] that would make fair comparison baselines
- Uncertainty estimation is more than just out of distribution detection: it would be good for the authors to show calibration performance, and evaluate accuracy and calibration on corrupted datasets (e.g. CIFAR-10/100-C) [6].
- The clustering properties of the proposed method are not analysed in depth
- Given the few and statistically unreliable experiments presented, at least showing the method could scale to ImageNet would have been interesting

[1] Class Anchor Clustering: a Loss for Distance-based Open Set Recognition, Dimity Miller, Niko Sünderhauf, Michael Milford, Feras Dayoub
[2] Uncertainty Estimation Using a Single Deep Deterministic Neural Network, Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal
[3]Training independent subnetworks for robust prediction, Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, Dustin Tran
[4] BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, Yeming Wen, Dustin Tran, Jimmy Ba
[5] Fast Predictive Uncertainty for Classification with Bayesian Deep Networks, Marius Hobbhahn, Agustinus Kristiadi, Philipp Hennig
[6] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, Dan Hendrycks, Thomas Dietterich
main_review: Originality: the paper is incremental and does not introduce particularly innovative concepts or methodologies. It just slightly modifies the IsoMax technique. 

Clarity: the paper is clearly written and organised. The extensive presence of allegedly unfair comparison does not make much sense to me, especially considering the experiments have been performed without averaging over (at least) 5 seeds, and many times the differences in these cases marginal or the method is underperforming. There is also no reason to break the presentation of the result for one type of experiments in so many tables. 

Significance: The paper does not show statistically significant results, and the results seem to be mixed in many cases, also based on which metric is used. The choice of the baselines is questionable and neglects most of the recent literature (I cited just a few of the most recent techniques). 

Quality: the paper is well written, but the experiments are not statistically significant nor extensive enough for acceptance. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: This work improves "IsoMax loss" by using normalized feature vectors (and a learned scalar multiplier) when training a CNN for classification. The paper situation the new loss function in the Out-of-Distribution literature an focuses on fast inference time without needing input preprocessing.  The results generally show the IsoMax_I loss out performs similar methods.
main_review: Originality: 
The main proposal of this work is to improve IsoMax loss by normalizing it's prediction vectors and multiplying them by a single learned parameter. This is a very minor change over IsoMax. Various distance metrics have been used for OOD, and the proposed S_{MinDistance} is very straight-forward as it's the exact metric being optimized. 

Clarity: This paper well describes the OOD problem and in what situations the new proposed loss function is suitable for use.  The title is disingenuous as the method is a change to the final layer of a neural net, while the loss function is still LogLoss (or CrossEntropy ).

Quality: The evaluation procedure is generally okay for the proposed method. There is a significant amount of text/format overlap with the prior work IsoMax Loss.

Unfortunately, comparison to state-of-the-art is lacking. Specifically, Generalized-Odin [11] often outperforms the proposed method even without input preprocessing (see table 7). Additionally, Generalized-Odin is also a similar change to the final layer of a neural net. 

It's not clear in what situation IsoMax_I should be used instead of Generalized-Odin.

Post rebuttal update:

While I appreciate the authors very detailed response, my primary issues have not been addressed: novelty and comparison to SOTA. The proposed work (a change to the final layer of a CNN) is not of enough novelty to be of interest to the broader community. Some valid critiques against my recommended SOTA (generalized-odin) were given, but failed to address the primary difference in OOD score (using a "fair" comparison). Disappointingly, the results discussed in the comments are cherry-picked by architecture. I've run G-odin myself and get better results than reported in this paper (without input preprocessing/hyper-param tuning). Lastly, there's significantly more datasets used in OOD papers in the last year or 2, and it's not clear if cifar10/cifar100 results generalize to harder datasets.

For these reasons I will keep my initial score.


limitations_and_societal_impact: The paper is missing the broader impact section, but there is discussion throughout on reducing energy use.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work proposes a method to improve the entropic out-of-distribution detection method which comprises the IsoMax loss for training and the entropic score for out-of-distribution detection. The IsoMax method defines the probability (also called entropic score, ES) based on the softmax of the distance from the feature map f(x) to the learnable prototype associated with each class \mu_j, i.e. p(y=k|x) = exp(-|f(x)-\mu_k|)/\sum_j exp(-|f(x)-\mu_j|). The authors observed that since the feature map f(x) are unnormalized, examples with low norms are unjustifiably favored to be considered OOD examples. So they propose to use the normalized feature map f’(x)=f(x)/|f(x)| to fix it. The method is called IsoMax_I. They also propose to use the minimum distance score min_j |f’(x)-\mu'_j| as the OOD score instead of using the maximum probability max p(y|x).
main_review: - Originality: the work proposes a simple but effective fix to the original IsoMax loss.

- Quality: the paper is written well and easy to follow. There are a few comments:
1. The experiments compare the performance among the regular softmax loss, IsoMax+ES and IsoMax_I+MDS (the proposed). It would be better to show the performance of IsoMax+MDS and IsoMax_I+ES, so that the contribution made by the two proposed components, IsoMax_I and MDS, can be disentangled. Similarly, in Figure 1, it would be great to show similar histograms for IsoMax + MDS and IsoMaxI+ES. 
2. The proposed IsoMax_I loss involves a hyper-parameter called distance scale d_s. How do you tune d_s? Do you need extra validation data for tuning?

- Clarity: in general the paper's text, figures and tables are presented clearly. There are a few comments: 
1. The authors mentioned that Mahalanobis method (Table 1) requires hyperparameter tuning and requires adversarial samples (line 183). In my understanding, the vanilla Mahalanobis method fits class conditional Gaussian distributions based on the feature map, which does not involve hyperparameters or adversarial samples.
2. In Table 5, it is unknown which OOD datasets are the results for.

- Significance: the results are important. 


***Update: Thank the authors for your response to my questions and your efforts on providing more results. Overall the method is simple and effective. I increase my score from 5 to 6. Please include the updated figures and results into the final version. Thank you. ***
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The extensive discussion here seems to have resulted in some substantial improvements to your paper, but there remain some concerns about the thoroughness of the experimental results (in particular with respect to the choice of architecture and of datasets) and clarity of the discussion. Some reviewers complain that the changes in your proposal are too simple; I want to emphasize that I do _not_ think this is a good reason to reject a paper (and indeed have fought to accept "too simple" papers in the past), but that "simple" methods do tend to come with a higher bar for clarity and thoroughness of the presentation. The changes brought up in this discussion were also of somewhat significant scope. Thus, it seems best if you integrate the results of the discussion here into a revision to be reviewed at another venue.
Decision: Reject