Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The authors present an approach for testing calibration in conditional probability estimation models. They build on a line of work in the kernel estimation literature assessing whether the conditional distributions are well calibrated (i.e. P(Y | f(X)) = f(X), where f is some predictive model). They develop an MMD kernel estimator and expand on practical choices of kernels that are computationally tractable. They then derive an asymptotic null distribution for calibrated models, enabling control over the error rate when labeling a model uncalibrated. A few simulation studies are done with neural networks to show the applicability of the method.

Review:
This is an excellently written paper. The intro and first few chapters are a joy to read and really explain the problem well. There is a lot of nuance to calibration, so I really appreciated the precision and clarity in the exposition.

The idea itself also seems quite elegant. Generalizing a previously published kernel approach from only discrete distributions to handle a more general class of problems may seem like a small conceptual step. However, I think the authors did a good job explaining the challenges of this extension. The resulting estimators are now applicable to many more problems than the existing work.

Note I am not an expert in kernel learning, so I have not evaluated the proofs for correctness.

My main issue comes with the lack of empirical studies. The toy problem is not terribly interesting and does not reveal any particular insight. It leads me to believe that maybe this is not that useful of a method, since the authors did not have anywhere that they could apply it to and derive meaningful insights or uses. The comment in the conclusion about the differentiability of their kernels is interesting and I think incorporating this into the training procedure could potentially show some very clear pragmatic use of this method.

Overall, I like the paper. It is clearly written and presents what I think is an interesting and novel idea. 
review_title: Well-written and an elegant idea

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper addresses probabilistic data driven model calibration, i.e. aligning predicted target probabilities with actual ones. 

This problem has been extensively studied for classification tasks but solutions for regression tasks have limitations as illustrated by Fig.1. The authors intend to fill this gap and introduce a general kernel-based calibration framework that subsumes other ones previously defined for classification. The contribution of the authors is thus clearly stated and positioned w.r.t. prior arts. 

The authors start by proposing an alternative definition of calibration (Def.2) in order to cast the problem into integral probability metrics. It is this re-definition of the problem that allow them to encompass prior arts as special cases. 
For a number of practical and theoretical reasons, the authors focus on a special case of this framework which involves the computation of the MMD as a metric. 

Based on the MMD literature but also relying on the structure of their problem (where the auxiliary variable can be marginalized out), the authors provide several consistent estimator with known rates in dataset size.

The validity of the proposed estimates is assessed through convincing numerical experiments that involve a calibration test.

I honestly do not have much critic to address to this work which seems to have reached a level of maturity perfectly adapted for publication in ICLR. The only damper is that the proposed methodology allows to detect miscalibration not yet to cure it. However, the authors seem to have some ideas on that too as mentioned in their conclusion.

review_title: A very nice piece of work

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors define (Definition 2) a generalized form of calibration error for any model with probabilistic output and any output space (binary classification, multiclass classification, real-valued regression, ordinal regression, structured prediction, etc.).

The main novelty of Definition 2 seems to be the introduction of a dummy variable $Z_X$ which is sampled from the predicted distribution $P_X$ over the target space, in order to compute an integral probability metric that compares the joint distribution of $(P_X, Y)$ to the joint distribution of $(P_X, Z_X)$. The authors note that in some cases, $Z_X$ can be integrated out of the definition either analytically or using numerical integration. It is not clear, from my limited background knowledge in the related work, why it should be helpful to introduce a dummy variable and then integrate it out.

At first reading, Definition 2 seems too general to be useful in practice, as it requires the choice of a space of functions $\mathcal{F}$. The authors argue (with details in the appendices, which are not provided to me at this time) that it generalizes several previous definitions of calibration error including the maximum mean discrepancy, the total variation distance, the Kantorovich distance, and the Dudley metric.

Section 3 went beyond my area of expertise and beyond my comprehension; I feel unqualified to provide an informed review of this section. I think some reference to kernels or RKHSs should be made in the title or the abstract.

The experiment in Section 5.2 demonstrates the utility of the proposed SKCE calibration metric, alongside more common metrics like negative log-likelihood (NLL) and mean squared error (MSE). It appears that the SKCE curves (for both training and test data) have a very similar shape to the NLL curves, so it's not clear what benefit SKCE provides above and beyond the more common and easily-computed NLL. I would have liked to see more convincing experimental evidence of the marginal benefit of this approach beyond common calibration metrics.

Regarding the significance of the work, I can add that in practice, I find that relatively few ML users are concerned with the calibration of their models, and these are entirely restricted to problems of classification (almost always binary classification) or quantile regression. The novelty of this work seems to lie mostly in its applicability beyond these types problems: the authors write, "A key contribution of this paper is a new framework that is applicable to multivariate regression, as well as... discrete ordinal or more complex (e.g., graph-structured) [output]," so I venture a guess that the intended audience for this work is relatively small.

**Minor comments**

In equation (2), I believe the RHS should be $\max P_X$ instead of $\arg \max P_X$.

Section 2, paragraph 2, you wrote "instead of the discrepancy between the conditional distributions $\mathbb{P}(Y | X)$ and $P_X$." Did you mean to write $\mathbb{P}(Y | P_X)$ instead of $\mathbb{P}(Y | X)$? That would make more sense to me, since the sentence would refer to comparing the LHS and RHS of Equation (1).


review_title: Experiments not convincing enough

Meta-review: This is a well written paper addressing a challenging problem with an original approach.  While one reviewer claims there is not a strong call for calibration of regression tasks, this may well be because methods don't exist.  Certainly, calibration is a critical tool for classification.

The major failing of the paper, however, is the empirical evaluation.  Given that no prior work exists, it is arguably OK to not do this, but one could easily reject the paper on this issue alone, as AnonReviewer4 was inclined to do.  One reviewer, however, thought highly of the paper, which bumped up its average score, more than I think it should have got (due to the poor experimental work).

The abstract could be improved by mentioning the use of kernels, the nature of this solution is a substantial part of the paper.
Decision: Accept (Poster)