Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Update: the authors went out of their way to address my concerns about the absence of the unbalanced class setting: they added a new datasets (SVHN), new results (table 4) and updated some of their explanations. All these additions seem satisfactory. I was also pleased with the feedback about computational cost (R3). I improved my rating. 
While I agree with the concerns of reviewer 4 (those I could understand), they would apply to every publication I have read about calibration, and I think the authors addressed these concerns to the best of our current knowledge.

This paper proposes an information maximization binning scheme for calibration. Starting with a good introduction, a clear progression leads to the core algorithm described by theorem 1. Limits of previous histogram-based approaches, both in terms of performance or reliability of metric, are clearly demonstrated with clear figures and proper references.
While using information measures to drive histogram binning has been done, I assume that the current classification setting where one maximizes the MI between the logit and the class is novel (the authors do not give pointer to previous work here, only mentioning the Info Bottleneck without references).

Theorem 1 leads to an alternative minimization algorithm with analytical steps. I did not check the convergence behavior proof but Figure 3 is convincing enough. I did not fully understand the information bottleneck limit.

Experiments show first that the information binning strategy is far superior than equal-mass or size binning. Table 2 and 3 then shows how it improves on most scaling algorithms used for calibration. One detail I am not comfortable with: the ECE_{1/K} hack, as it looks like a last-minute addition to give even stronger gains to the I-MAX method. A more principled introduction would be better (see below).

This would be an excellent paper except for the following, which casts doubts whether all the steps of the method generalize to an unbalanced multiclass setting. It is probably possible to fix or explain before publication.
This paper relies on a very unnatural and unfortunate state-of-affair in ML: classes are equally distributed on the test data. The phrasing does even consider any other possibility, and some of the algorithms seem to be quite specific to this setup, requiring significant changes in the “real world” case where test classes are not equally distributed.

At the end of section 3.2, the authors propose an algorithm to merge {S_k} across K classes based on the observation that they have similar distribution. Rather than a proof, they run a simulation on ImageNet (Sec A.2) that shows it is better than binning each S_k separately. While the experiment is elegant, it probably strongly relies on the fact that each S_k has the same 1:K-1 split. What would happen if the classes follow a more realistic Zipf law, as observed in real NLP classification tasks? I would assume that the merging process could still be applicable, but applied to groups of {S_k} with similar class-0/class-1 distributions.

In section 4.1, the trick to remove from the measure of the ECE classes where the predicted probability is less than 1/K also depends on a uniform 1/K prior. It should also be adapted to a non-uniform prior.

review_title: Elegant algorithm with great results, but why is it restricted to a uniform class prior?

final_score: rating: 7: Good paper, accept
review_content: Update after the rebuttal: The authors have answered my concerns. I believe the paper should be accepted and would be a nice contribution to the current research.

===============================================================



The paper proposes a novel approach for post-hoc calibration of outputs of the neural networks to estimate uncertainty of its prediction. The paper considers the histogram binning approach (in contrast to scaling approaches existing in the literature) and utilises the information theory in building bins.

Strong points:
* The work is very well placed in the context of the existing literature identifying the current gaps
* Theoretically sound motivation of the approach
* Extensive empirical evaluation

Weak points:
* Some discussion of the cost of the proposed method is lacking - i.e. how much in terms of computational time and memory this new calibration method is? 
* The methods are compared with respect to accuracy and Expected Calibration Error (ECE) only. It has been shown that ECE is not a good metric for comparing different methods (see, e.g. Ashukha, A., Lyzhov, A., Molchanov, D. and Vetrov, D., 2020. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. ICLR 2020). 

I am recommending acceptance of the paper, though addressing the weak points above would largely improve the paper. The reasons for this decision is that strong points outweigh weak points: the proposed idea is interesting, it is shown that it is promising in practice (subject to not very good metrics) and the paper is mostly well written and easy to follow.

Questions to authors: 
Could you please address raised weak points?

Additional feedback (not necessarily important for evaluation, but could help to improve the paper):
1. The part on shared class-wise binning is rather rushed in the main paper and it is not very clear. It is also rather independent contribution from the main I-Max calibration contribution. It would be better to somehow put them under one umbrella
2. Section 2. “Bayesian DNNs, e.g. (Blundell et al., 2015) and their approximations (Gal
& Ghahramani, 2016)” – a very arguable statement, I would suggest rephrasing it. Since Blundell et al. proposed variational inference which is also an approximation, and Gal & Ghahramani work is not an approximation of Blundell et al.’ model
3. Section 4.2. “Namely, matrix scaling w. L_2” dot after w is read as a full stop which is confusing
4. After eq.(5). "So, we can solve the problem by iteratively and alternately updating ${g_m}$ and ${\phi_m}$ based on (A12)." - it seems eq. 5 and A12 are the same and it would be more convenient to refer to eq. 5 in the text right after it.
5. I am a little bit missing the overall procedure of the proposed calibration. I.e. all details are there (especially if refer to appendix), but after reading the main paper, there is no feeling that I can now go and implement it for my problem. Maybe a pseudocode can help, or just step-by-step guidance

Minor:
Section 3, first paragraph: “Sec. 3.2)” – redundant bracket


review_title: Interesting idea with extensive experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper highlights the issues with the scaling method and histogram binning i.e., underestimate calibration error in scaling methods and failing to preserve classification accuracy, and sample-inefficiency in HB. They use the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. They claim that their approach mitigates potential loss in ranking performance and allows simultaneous improvement of ranking and calibration performance by disentangling the optimization of bin edges and representatives. They also propose a shared class-wise (sCW) strategy that fits a single calibrator on the merged training sets of all K class-wise problems to improve the  sample efficiency.

The paper is well written and the authors provide enough motivation and intuition of why maximizing the mutual information between labels and quantized logits would help multi-class calibration. There are some concerns and issues that I think needs to be addressed. 

1- One approach in estimating uncertainty in classification is to choose a model and a regularized loss function to inherently learn a good representation. For example using confidence as a term for regularization in neural networks is proposed in Regularizing neural networks by penalizing confident output distributions (ICLR 2017) that penalizes low-entropy output distributions. I think it is worth comparing the results with such existing work and discussing the advantages and disadvantages since a similar concept has been used one while training the model and this paper as a post-hoc calibration. 

2- It is interesting that a single calibrator on the merged training sets of all K class-wise problems (sCW) performs well. As it is mentioned in the paper, it introduces bias due to having samples drawn from the other classes. In HB, increasing its number of evaluation bins reduces the bias, but in (sCW) such bias can not be controlled. Moreover, Figure A2 shows it achieves smaller JSDs which is not expected. Is there any reason for that? What would happen if the number of bins is increased?


3- Based on the experimental results, it seems I-Max performs better than other binning approaches. However, compared to the scaling methods it seems GP (Wenger et al. 2020) performs better at NLL/Brier than the I-Max variants. 

4- Even though the paper shows combining I-Max with GP improves the ECE, it is not clear how the issues of each approach will be handled. For example, the ECE might be underestimated.  

review_title: The paper provides enough motivation and intuition of why maximizing the mutual information between labels and quantized logits would help multi-class calibration, but there are some concerns that needs to be addressed. 

Meta-review: The paper proposes to maximizing the mutual information to optimize the bin for multiclass calibration. The idea, technique, and presentation are good. The paper solves some multiclass calibration  issues. The author should revise the paper according the reviewer's comments before publish.
Decision: Accept (Poster)