Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on proposing the local calibration error (LCE) to span the gap between average and individual reliability, which measures the average reliability of a set of similar predictions, proposing a metric to measure calibration locally around a given prediction. Experiments are conducted and the results seem good, beating most baselines in the experiments chosen.
main_review: Strengths:
    1. The motivation of this work is very clear.
    2. The article is very clear and easy to understand.
    3. A large number of experiments were carried out to prove authors' claim.
Weakness:
    1. Why choose the binning scheme over predicted model confidences? How about taking all samples into account, or other schemes? 
    2. For a unique sample, the neighborhood the authors defined maybe change when the size of its bin is changed, does it make sense? I think you can refer to [1].
    3. Is the promising experimental result due to the superiority of the method, or is it an accidental result from a specific kernel function and feature map?
    4. Since performance is sensitive to many choices, as discussed in Section 3.2 ``the MLCE is very sensitive to \gamma", the authors set \gamma as 0.2 and show the good performance, but how can I choose the best one in real-word applications? So clear and strong ablation study is necessary.
   5.  The difference between the confidence and the accuracy for one sample is not reasonable when using it in Eq. 4. Since the accuracy is not a good metric for only one sample, although the authors propose using neighbors (what about the case the prediction of the current sample is wrong?)
   6. Using kernel to introduce locality is not new.

Other concerns:
    1. The style of Table 3 is different from other tables, is there anything special the authors want to emphasize? Although the article is clear on the whole, there will still be misunderstandings on this point.
    2. In Fig. 3 and Fig. 4, there are no error bars, while the other same type figures (Fig. 5, Fig. 6 and Fig. 7) have reported that. 

[1] Karandikar, A. , et al. "Soft Calibration Objectives for Neural Networks." (2021).
summary_of_the_review: The paper proposes a new metric for calibration, and propose a post-calibration algorithm for this metric. Overall, the novelty is a little weak and the improvements are not quite clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new metric for calibration in classification where calibration is measured over localities in the input space, and the localities are determined with a kernel over the feature space. A recalibration algorithm (LoRe) is additionally proposed, which aims to recalibrate the class predictions w.r.t. local calibration error (LCE). The experiments show that 1) LoRe achieves better local calibration (measured by MLCE, i.e. maximum LCE) than baseline recalibration methods on ImageNet, 2) LoRe achieves better group-wise calibration (measured by worst group-wise maximum calibration error) across 3 datasets with identifiable groups, and 3) achieves competitive performance in a simple decision making task where confident incorrect predictions incur high costs. 
main_review: Strengths:
- This paper is generally well written and clear. It is easy to follow and the experiments support the argument made in this paper.
- The notion of local calibration seems novel to me. Although I have not scoured papers in this area, conditioning calibration error not just on confidence, but also feature similarity in the classification setting seems new to me.

Weaknesses:
- While the experimental results look positive, I believe they are not surprising because, as far as I can tell, LoRe uses the same feature map and kernel as the core evaluation metrics (LCE, MLCE). What happens empirically when different feature maps and kernels are used for applying LoRe and evaluating with LCE, MLCE?
- Further, it seems as though computing the proposed metric (LCE, MLCE) depends on 1) a specific feature map, 2) a specific method for further reducing dimensions, 3) a specific kernel function, and 4) specific settings of the hyperparameters for the kernel. It is hard to be convinced that this is a robust metric when calculation depends on so many choices, i.e. I can imagine different rankings for the methods if the evaluation metric was computed with different choices of the above. 
- Continuing from the above point, is there a reason why the authors chose a Laplacian kernel? Have the authors experimented with other kernels? How about different feature maps?
- Choosing a suitable $\gamma$ seems quite arbitrary. Last line of Section 3.2, the authors simply state, "0.2 is a good intermediate point" without any justification. In practice, how should $\gamma$ be chosen?
- How much does the LCE metric diverge from classwise-ECE? If I understand correctly, LCE is meant to measure ECE in localities over the input feature space. Classwise-ECE, in a certain way, provides a proxy of this, because ECE is measured for each true class label, and inputs features should be well clustered within each class label (assuming meaningful features). In fact, if the features were separable by class, I believe LCE should be identical to classwise-ECE with an appropriate kernel and hyperparameter.
- Continuing from the point above, I believe it would have been more meaningful to 1) compare against methods that target classwise-ECE (i.e. perform recalibration by class), and 2) to additionally report classwise-ECE values in the experiments.
- Is LoRe an efficient algorithm during test-time? It seems as though Equation 6 needs to be computed for each test point x, where x_i indexes all points in the recalibration dataset $\mathcal{D}$. This would be in contrast with other post-hoc methods which, e.g. learn a lookup table or tune a scalar during the recalibration phase, and can predict during test-time with a single forward pass of the classifier.

Minor comments:
- What is $d$ in the denominator of the Laplacian kernel definition (Section 3.2)?
- In Figure 2, is the vertical dashed line simply ECE? It is referred to as "average calibration error" in text in Section 3.3.
- Are the experiments with ImageNet run with just 1 seed?
summary_of_the_review: While the concepts introduced and methods proposed are interesting, I do not believe the proposed metric is necessarily robust, and am not convinced of its advantages over an established metric (classwise-ECE, which also has not been reported in this work).
I believe elaborating (theoretically or experimentally) on these points could make a much stronger case for this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new measure of calibration called Local Calibration.
While conventional calibration measures are only defined with probabilistic outputs, the proposed local calibration measure further incorporates the feature space by considering the neighbouring region with a kernel.
The authors also propose a calibration method according to the definition and experimentally demonstrate the advantages.
main_review: In general, the idea of evaluating a model on both the feature space and output space is quite interesting. It provides a potential direction to extend the current probability calibration framework.

On the positive side, while the proposed LCE straightforwardly applies the kernel method, the authors put some reasonable effort into investigating the proposed measure's theoretical properties. While the proof of Theorem 1 relies on some existing work on kernel methods, it might provide some valuable references to other researchers interested in similar problems.

On the negative side, one main concern with this paper is the ambiguity of the proposed definition of Local Calibration. On the one hand, we have an existing definition of a perfectly calibrated model, which is solely defined in terms of the probabilistic output space. On the other hand, we also have the definition of a Bayes-optimal model, which is defined only on the feature space. For both cases, we can define a classifier that is optimal for either measure. Furthermore, we know that being a Bayes-optimal model is a sufficient but not necessary condition for being a perfectly calibrated model.
However, while the proposed local calibration tries to incorporate both output space and feature space, it is unclear what is an optimal classifier under the definition due to the introduced kernel space. The author did not clearly show the requirements for a classifier to be perfectly local calibrated and its relationship to the Bayes-optimal model / perfectly calibrated models. It hence could be problematic when we want to treat local calibration as a measure to optimise.  

Also, while the authors used NLL and BS for their experiments, they did not discuss much the framework of Proper Scoring Rules. PSRs can be used to evaluate the overall quality of a probabilistic forecaster in terms of both refinement and calibration. A classifier with minimal refinement loss and calibration loss is Bayes-optimal and optimal both on the feature and probabilistic output spaces. It is therefore closely related to the local calibration concept proposed in this paper. I would therefore encourage the authors to consider potential links to PSRs further so that the local calibration concept can be clearly linked to calibrated / Bayes-optimal classifiers.

A minor suggestion is also to consider links to existing kernel-based probabilistic classifiers (relevant vector machines / Gaussian Processes), where the probability output is directly optimised with a given kernel space.
summary_of_the_review: This paper provides an interesting method to improve probabilistic outputs with the proposed definition of local calibration. However, the local calibration concept is not linked with the existing definition of calibrated / Bayes-optimal classifiers, so it might not stand firm as a measure to optimise. Some further links to exciting kernel-based classifiers will also bring better insights to the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel calibration metric, the local calibration error (LCE) that measures the average miscalibration degree of local predictions, which complements the existing global ECE metric. In addition, the authors also propose a novel local recalibration method called LoRe to minimize LCE.
main_review: Strength: the paper is well written and the proposed LoRE method seems simple yet effective. The proposed local calibration concept is novel. 

Weakness:

1. Lack of motivation for the proposed local calibration concept. The authors indeed mentioned about fairness and causal inference as two potential interesting applications, and conducted fairness related tasks in the experimental section. But the connection is not clear to me. Furthermore, the authors should  provide some comments on the pros and cons of LCE over existing fariness related group-wise calibration measures (ref i).  

2. The dimensional reduction procedure. There should be some detailed discussion on how to choose the kernel bandwidth (based on the empirical observation of the transition behavior?). 

3. Experiments. While the provided experiments are highly useful, I think both 5.2 and 5.3 need improvement. Specifically, in 5.2, include some popular baseline datasets such as CIFAR 10/100 ; in 5.3, the authors might want to compare against existing group-wise calibration measures & recalibration methods.

4. Some discussions and/or comparisons about highly related and recent work are missing, such as [i] for fairness and calibration and [ii, iii] for recently proposed kernel-based calibration measures.

[i].Pleiss, Geoff, et al. On Fairness and Calibration. NeurIPS, 2017.
[ii]. Zhang et al. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. ICML, 2020. 
[iii] Gupta et al. Calibration of neural networks using splines. ICLR, 2021. 
summary_of_the_review: As discussed above, my primary concerns for this paper is on the motivation of local calibration, as well as some possible improvement over the experimental section (such as comparison with existing approaches). I hope that those concerns can be addressed in the rebuttal process.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new metric for measuring model reliability called local calibration, which allows for some insight into which regions of input space a model may be less well-calibrated in. They provide a post-hoc adjustment method for improving local calibration and empirically show that it is effective, without hurting overall measures too much.

main_review: Strengths:

- local calibration is a sensible idea and it's good to see it worked out like this
- local/individualized notions of calibration are an important problem
- same thing for the proposed method: it's simple and seems like the thing you would try in this situation
- the experiments show fairly consistently that this method provides gains in improving local calibration metrics

Weaknesses:

- it's not clear to me why we are filtering for prediction bin as well as weighting along input space - my intuition around what "local" calibration should mean would involve reliability among all similar inputs, rather than just those that also have similar outputs. The analogy to global calibration metrics is not immediately obvious to me and merits more discussion
- the choices around representations which are operated on seem under-discussed. For instance, my guess from reading this paper is that t-SNE is quite important to the practical results of this method - would be good to hear if that's true or not, or how other approaches compare results-wise. If t-SNE is essential to the application of the method, it indicate the method may not be that robust
- Additionally, I'm curious about the application of t-SNE here - how is a test point placed in a t-SNE representation space which was fit on a validation set? As far as I know, t-SNE representations are usually not amortized
- 2-d pictures in Fig 2 are very nice but I find the right side confusing - I need some clarification on how to read these CDFs. Also, you state that the regions of over/under-confidence are clustered in input space, but it's not clear how we would tell if this is more than we'd expect given random variation or not
- I'm skeptical that MLCE is the only metric worth looking at here - I'm also curious about something like average LCE
- Table 2: how is this correlation calculated - over what sample of models?
- Table 4: I'd like to see standard deviations/confidence intervals here. This type of metric is similar to an average LCE, which is good, but the wins seem uneven (which is fine). Also curious to here why some metrics show different results than others


Other thoughts:

- would be interesting to think about customizable kernels to inject notions of similarity we care about (e.g. the improper group-wise kernel mentioned in section 4)
- Fig 3: tell me what these legend abbreviations mean
- as bandwidth goes to \infty, can you discuss what other methods it becomes similar to?
- I would be interested to see exploration of which regions are uncalibrated for models

summary_of_the_review: This is a sensible idea (metric and method) applied to an important problem. The exposition is fairly clear although some design choices are under-discussed. The experiments show that the method works as intended, although they could benefit from some deeper analysis.
review_title: null

Meta-review: The reviewers all generally find the paper both well-motivated in addressing an important challenge as well as well-written. However, there's quite a bit of hesitation around whether the proposed metric is convincing enough as an approach to measure local calibration.

Reviewer 76PS and 784d's concerns around the choice of feature map and associated hyperparameters remain unaddressed, and I agree with their concern. There is no clear understanding of what constitutes a "good" feature map, which makes the metric quite difficult to use whether as a benchmark of ML methods or for general application.  I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue.
Decision: Reject