Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Pros: originality, clarity, technical correctness.
Cons: experiments need some clarifications.

Calibration typically relies heavily on binning the data, both for the calibration itself  (Histogram Binning) and how to measure its quality (ECE). Thus both operations suffer from sampling issues that are a cause for both bias and variance. The idea to rather perform the analysis using cumulative distributions would seem obvious, as it has been tried on so many other problems, but I have not seen it used for calibration. They do it for both calibration and its measure:
-	The use of the Kolmogorov-Smirnov test to measure calibration between the target and the output distributions.
-	Spline fitting of the cumulative distribution to compute its derivative
As the implementation details are far from obvious , there are several original contributions, especially in implementing the splines.

I found the description both very clear and concise, switching between intuition and equations.
The only part I found confusing is the second paragraph of section 4.2 (“One method of calibration..”). Fortunately, the next paragraph gives a very simple intuitive explanation by just stating how it is implemented.

Experiments are very comprehensive and show improvements over Temp scaling and other methods. However, there are also some results that contradict previously reported experiments and need to be clarified:
-	Besides Temp scaling, the main other methods are borrowed from Kull et al, so one could expect some consistency. However ECE results from Table 6 look very different from Table 3 in Kull et al. I assume these are different types of ECE: confidence vs. class-wise? In Table 1, KS result for the ODIR methods of Kull et Al are much worse than Temp scaling, in particular for CIFAR-100 and Imagenet. This contradicts results reported in Table 2 (this paper) and Table 3 (Kull et al) and should be explained.

-	I am no expert in image classification,  but the 70% accuracy reported for CIFAR-100 seems way below current numbers, which have exceeded 80% since 2017, in particular for the proposed architectures (for instance Wide Resnet or DenseNet) https://benchmarks.ai/cifar-100.  However these numbers seem to be consistent with what is reported by Kull et al (Table 18 in https://arxiv.org/pdf/1910.12656.pdf), so I assume the issue comes from borrowing their architecture and scores. While this should not impact comparative results, it would have been more satisfactory to use baseline architectures that match the state-of-the-art.

Additional experiments or discussions on the following would greatly help:
-	As the spline method is not always better than Temp scaling, how do they compare from a computational viewpoint? How long does the binary search over thousands of calibration examples take compared to the DNN feed-forward?
-	From Tables 1,2 and 6, KS error and ECE rank methods quite differently. One reason one should trust KS more is that it does not depend on binning choices, and rely on a time-proven test. But could one come up with an experiment that shows that KS is provably more reliable? 

Post rebuttal: I have read the authors responses with to my 4 questions, and appreciate how detailed and honest they are. They satisfy my concerns.

review_title: Elegant calibration method, beautifully described. Experiments may need more explaining.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors present a binning-free calibration measure from a Kolmogorov-Smirnov-based test. Besides, the cumulative probability distribution is estimated using a spline-based fitting from percentiles. The approach allows correcting the probability estimation from trained deep learning models. The paper is clear and well-founded. 

review_title: The paper presents an interesting contribution with potential applicability in the deep learning community.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a post re-calibration spline-based approach for the re-calibration of multiclass predictions.  Experiment results on image datasets are provided and evaluated according to the Kolmogorov–Smirnov (KS) statistic. 

**Strengths**:
 - The proposed binning-free calibration measure is a good idea and widely used in binary classification problems

- As expected, experimental results demonstrate that optimizing calibration with a spline-based approach results in better calibration

**Weaknesses**:

*Inconsistent notation*:
-  The authors should stick to either $x $ or $X$
-  The definition of the KS statistic is inconsistent with standard formulations Eq (8). Also, the paper claims to obtain the maximum value at $\sigma=1$; this is a strong assumption.

*Weak experiments*:
- Why KS statistic and not Wasserstein or calibration slope?
-  Eq (11) assumes $f_k(x_i)$ are distinct, which is not always the case.
- Calibration and accuracy are orthogonal concerns, where the proposed post re-calibration approach is prone to overfit on calibration at the expense of accuracy (and this is the case, as experimental results show a loss in accuracy)
-  Demonstrating an approach that accounts for the calibration-accuracy tradeoff is crucial
- Extending the proposed approach to regression problems would strengthen the submission
- A  qualitative discussion on:
1)  Why Temp. Scaling is better calibrated in some instances
2)  Why calibration results of proposed solution (and alternatives) vary across different network architectures for the same dataset
3) Why the proposed approach drops in performance between top-1 and top-2 predictions
- The proposed re-calibration approach may not scale well with large datasets; a computational complexity comparison against alternatives is crucial
review_title: An ok submission but incremental. 

final_score: rating: 7: Good paper, accept
review_content: The paper presents a post-hoc calibration method for deep neural net classification. The method proposes to first reduces the well-known ECE score to a special case of the Kolmogorov-Smirnov (KS) test, and this way solves the dependency of ECE on the limiting binning assumption. The method proposes next to recalibrate the classification probabilities by fitting a cubic spline to the KS test score.

Strengths:
 * The proposed approach is a clearly novel solution to an essential problem for the safety-critical use of deep learning.
 * The paper presents the material in a very clear and neat way, following an excellent scientific writing practice.
 * Both the KS test based treatment of network calibration and the elegant way to improve the calibration by cubic spline fitting are interesting and likely to attract the attention of the deep learning uncertainty community.
 * The paper presents an exhaustive set of experiments that report performances of quite deep state-of-the-art network architectures on four different benchmark tasks. It also provides comparisons against a decent number of state-of-the-art calibration methods. The results demonstrate the benefits of the proposed method.

(Minor) Weaknesses: The paper can be improved with a few small adjustments in the presentation:

 * The so-called "python semantics" notation in Sec 2 only introduces complexity and does not bring any concrete value to the story line. What is wrong with denoting the r-th top score simply as f^{(n)}. Why do we need the minus? We can simply assume that the first element of the sorted list is the top score.

 * Why do we need Proposition 4.1 and its proof (sketch and full) in this paper? It only echoes the standard relationship between a PDF and CDF. The former is the derivative of the latter.

 * Although the results in Tables 1 and 2 are groundbreaking in favor of the proposed method, they are calibration scores based on the KS score. That is only one way of measuring the calibratedness of a network, and furthermore that is the score the proposed method is maximizing. It would be interesting to see how well the proposed method generalizes across other calibration scores that may highlight complementary aspects of the uncertainty treatment capabilities of the predictor, such as the Brier score.

Overall, this is a solid piece of work that qualifies to appear at the ICLR'21 proceedings.
review_title: Solid work on an important topic

Meta-review: Many papers have been written on calibrating neural networks recently.  This paper presents a definition of calibration that is more robust than the popular ECE measure while also being more discerning than the Brier score.  Then it proposes a practical spline-based method of post-editing the output softmax scores to make them more calibrated.  The method is shown to be better than existing methods both on their measure and established measure (thanks to reviewer's questions on that.).
The paper should be of much interest to the community.
Decision: Accept (Poster)