Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper presents an approach for calibration of neural networks. The proposed approach is based on the analysis of the softmax logits and decomposing it into two components that depends on the norm of the feature vector (norm component), and the other that depends on the similarity (similarity component) with the weight vector corresponding to a class. The paper then derives learnable parameters that could control each of the components and thus can be used for calibration of the predicted probabilities. The paper claims that for out-of-distribution (OOD) scenarios, this calibration scheme outperforms previously proposed approaches. The empirical results are reported for in-distribution (IND) on CIFAR-10 and CIFAR-100 and their corrupted versions of these for OOD, for various different models. 
main_review: Originality: 
The idea of geometric decomposition is both interesting and insightful and as far as I know, original. The derivation that leads to trainable parameters that can control the norm and the similarity components is original and seems to be quite generally applicable. This is also validated from the experiments. 

Quality: 
+ The paper brings new insights over previous approaches [10,11,12] by the proposed geometric decomposition. 
+ The experimental results are thoroughly done and presented, validating the claims made. 

Clarity: 
The paper is very well written and easy to follow. The approach is well motivated and the related work description seems to be complete (AFAIK). 

Significance: 
I believe the geometric decomposition of the norm component and the similarity component, and its parameterization is an important step towards making predicted probabilities more meaningful. The ideas in this paper could potentially inspire more research as well. 

However, I hope the authors can address the following comment in the rebuttal. 
- Line 148 states an assumption that \cos|\phi_y| is close to zero. This may be true after sufficient training of a high-capacity model for datasets like CIFAR and other similar ones. However, what happens when this assumption is violated? This may happen for certain classes when there is a high amount of imbalance. Kindly discuss the impact on the performance of the proposed approach. 


Update after rebuttal. 
My concern has been addressed. I retain the rating. I think it is a pretty good paper. 
limitations_and_societal_impact: The limitations and societal impact are pointed to. Perhaps potential failure scenarios of the said method could be pointed to, so that follow up works may explore that as well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of model calibration in the presence of distribution shifts for vision classification tasks. The authors propose a sensitivity decomposition on the last softmax layer of the model. The resulting calibration approach demonstrates improved performance compared to existing calibration methods under distribution shifts.
main_review: Strengths:
1. The paper takes a geometric perspective to study model calibration. The finding that the insensitivity of the norm accounts for the poor calibration under distribution shifts is interesting.
2. The approximate decomposition is elegant and easy to implement. It does not impose additional computational burden.
3. The proposed calibration approach is simple. It achieves good performance without increasing training and inference complexity significantly.


Major comments:
1. Do you have intuition on why the decomposition should be additive besides its simplicity? What happens if a product term is included in the decomposition?
2. I found the interpretation of $\lVert \Delta x \rVert_2$ confusing. For instance, in the first bullet on the top of page 5, when $\alpha$ is fixed, it is clear that a larger $\beta$ corresponds to a larger $\mathcal{C}_x$, but why a smaller $\lVert \Delta x \rVert_2$? Moreover, in Figure 1c, all OOD data have small values of $\lVert \Delta x \rVert_2$ (this is also implicitly assumed in Eq (9)), why can’t they have large values? If a model outputs moderate values on the training set but large values on the test set, this is also a sign of distribution shifts, right?
3. In the proposed calibration method in Sec 3.4, only $\beta$ is tuned on the validation set. Why don't you also tune $\alpha$?
4. I would like to see standard errors in Table 3-5.
5. In Sec 4.2, the findings are for corruptions with Gaussian noise. Do the findings hold under other types of corruptions?
6. It is unclear to me what the metric AUROC is measuring and how it is related to the OOD detection performance. I would like to see more explanation.

Minor comments:
1. The sentence “While both … calibrated to accuracy” on line 156-158 is unclear to me.
2. On line 161, “\Delta \phi_j” —> “\Delta \phi_y”.


-----
Updates: they answered most of my questions, so I decided to raise my score to 7.

limitations_and_societal_impact: See comments above for suggestions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes Geometric Sensitivity Decomposition (GSD) as the new calibration method in the face of out-of-distribution (OOD) data. The paper identifies that the insensitive norm is responsible for bad calibration under distribution shift. Based on GSD, the paper proposes a simple and quick training and inference scheme. The paper achieves SOTA in calibration metrics in the face of corruptions while having arguably the simplest calibration method to implement.
main_review: **[Update after the rebuttal] I am satisfied with the extra mile that the authors went to show the proposed method's effectiveness against the ensemble models. The provided reason behind why it was not a fair comparison before and how it is now fair comparison is sound. Updated my rating accordingly.**

Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

**The method is new. It is clear how this work differs form previous contributions. Related works are adequately cited.**

Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

**Theoretical support is good. Experimental results seem weak given that many metrics are not SOTA compared to prior arts. It is difficult to figure out the pattern when this method yields better results (i.e. Table 3, Table 2); I think it is a work in progress. Yes, honest in reporting all the numbers.**

Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

**It is clearly written. Well organized.**

Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

**The theoretical idea and its simplicity are promising. The empirical results are not so much as it is presented today.**
limitations_and_societal_impact: Good.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of neural calibration from a novel geometric perspective and finds that the insensitive norm is responsible for bad calibration under distribution shift. Based on this insight, this paper proposes Geometric Sensitivity Decomposition (GSD), which decomposes the norm and angular similarity into an instance-dependent and instance-independent component. Subsequently, this paper proposes a training and inference scheme, to encourage the norm to reflect distribution changes. The proposed method is easy to implement and achieves SOTA results in calibration metrics in the face of corruptions.
main_review: Originality: This work studies the problem of neural calibration under distribution shift from a novel geometric perspective. This work leverages properties of the norm of the input embedding to strengthen calibration, which clearly differs from previous works. However, this work refers to [10][11][12] (references of the paper), the difference between them is not made clear.
Quality: Overall, this work is technically sound and the claims are well supported by experimental results. The finding that the insensitive norm is responsible for bad calibration under distribution shift is key to this work, we hope the authors can give more theoretical analysis or experimental results (e.g., different datasets and base models) to support and strength this point. Meanwhile, some definitions are confusing and should be further explained: why Eq.2 uses “+” while Eq.3 uses “-“? why use such a kind of non-linear function in Eq.9?
Clarity: This paper is well organized. Some writing details need to be worked on:
1.hard-to-understand sentences: line140, “…making the angles larger”; line206, ”…=µx – σx”
2. misquotation:Line255, “Tab9”
3. a table like Tab. 10 may be better to compare “Extended Training Time”.
Significance: Studying the problem of calibration under distribution shift from a geometric perspective has some implications for other researchers or practitioners. This paper achieves promising performance on standard calibration metrics in the face of out-of-distribution (OOD) data and corruption. This problem and method have positive impact. Since I am not very familiar with the neural calibration literature, I am not sure the novelty of this paper reaches the bar of "accept".
The possible limitations and future works of this paper needs to be discussed.
limitations_and_societal_impact: Nil
review_title: null

Meta-review: The authors study the problem of model calibration in the presence of distribution shifts for computer vision. The authors propose a sensitivity decomposition on the last softmax layer of the model. This calibration approach demonstrates improved performance compared to existing ones.

All reviewers recommended to accept. 

Accept.
Decision: Accept (Spotlight)