Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose an approach for calibrated predictions under domain shift scenarios. The approach, that leverages (unlabeled) test samples allows for unsupervised post-processing calibration, even for off-the-shelf models for which the training data is not available. Experiments compare the proposed approach with existing calibration methods in shifted domains.

Equation (5) is confusing. If I understand correctly, the authors are simply making the point that q(x,y=k) can be written in terms of q(x,y\neq k) by weighting by the ratio of conditionals, which are available.

Sensitivity to noisy labels. The experiment is reasonable and the results are convincing, however, the authors do not justify why accurate (manual) labels on the target set are not feasible in many applications. The authors could point to a few examples for context.

The authors assume that q_s(y) = q_t(y), which seems restrictive in practice. Though it does not impact my opinion of the proposed approach, it seems narrow to think of a practical situation where the space of covariates is changing but the class composition remains unchanged. This is vaguely addressed in Section 6. Perhaps it can be elaborated further.

I enjoyed reading the paper, the proposed reinterpretation of NLL in terms of a weighted average and its approximation based on weights that do not depend on the labels but the (assumed known) labels marginal is interesting and seems to yield good results.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors present an algorithm for postprocessing neural networks to ensure calibration under domain shift.
Calibration under domain shift is an interesting challenge that has been receiving increasing attention and tackling this in an unsupervised manner is an interesting approach. However, I have 2 major concerns regarding the approach presented by the authors.

What makes calibration under domain shift useful and appealing is that the model is then robust against any changes in the test distribution that can occur during the life cycle of a model. These often include erroneous/samples (corresponding to truly OOD samples), but also gradual domain shift, where the test distribution continuously moves away from the training distribution (e.g. due to a continuous drift in user behaviour/change in customer base) or unforeseen changes. My first major concern is regarding the requirements for UTS, which render this approach not very useful in many of these practical  applications: UTS first requires knowledge of and access to the test distribution; in addition it assumes that the distribution of the labels remains unaffected under domain shift. These assumptions are violated in the practical applications described above, in particular those where a gradual, continuous domain shift occurs - in this case, access to the test distribution is difficult since it changes continuously. On this note I also would have liked to see some analysis on how performance depends on the number of samples that are available from the test set, since in practice this might be substantially smaller than the full test set used.
Furthermore, I find the assumption that the distribution of labels remains unchanged problematic (q_s(y) = q_t(y) and even q_s(y|x)=q_t(y|x)): once sufficiently out-of-domain, labels become meaningless and predictions for truly OOD samples should have maximum entropy. Even for small domain shifts in practical applications it is not clear why q_s(y|x)=q_t(y|x) should hold and it would have been useful to see a discussion and some robustness analysis on this.
Finally,  the algorithm requires re-calibration whenever the test distribution changes, which in practice is  often not clear (and part of the reason why dealing with predictions under domain shift is so challenging). 

In addition to doubts on practical applicability, my second major concern is regarding the depth of the evaluation.
First, while the authors present some comparisons to probabilistic methods, I am missing a crucial comparison to Evidential Deep Learning (Sensoy et al, NeurIPS 2018), which results in far superior performance than deep ensembles, SVI or dropout. Importantly, the comparisons to probabilistic approaches presented by the authors are very limited. The big advantage of those approaches is that, once trained, no further recalibration is necessary and well calibrated predictions can be made for any level of domain shift, whereas UTS requires a recalibration step for very level of domain shift. That is why I think it is crucial to not only show one arbitrarily picked level of domain shift for each dataset/perturbation, but calibration across all levels of domain shift, as for TS and TS-Target; since no recalibration is required for those probabilistic approaches 
 this is very straight-forward and would be very informative - especially since e.g Figure 5 shows that UTS has only very minor advantages over TS in many settings. 
I appreciate that the authors report some performance in terms of ECE in the supplement, but I think it would be very informative to report performance in terms of ECE for all domain-shift experiments: The Brier score conflates accuracy with calibration (see eg the 2 component decomposition), whereas ECE directly quantifies calibration and is hence easier to interpret and arguably the more meaningful measure when quantifying calibration. 

Minor:  I find the manuscript lacks clarity. Aspects such as the definition of calibration as well as implications and interpretation of Proposition 1 should be described in more detail in the manuscript. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I've read the rebuttal and unfortunately, I'd like to keep my score as is. I still think the assumption made in the paper is too limiting for most practical settings. 

#########################

The paper proposes an unsupervised calibration method in a domain adaptation setting. The approach is based on the well known temperature scaling and does not require labels for the calibration set. The problem of calibration under domain shift is an important problem in areas where uncertainty estimation is useful; the paper tackles this problem and relaxes the assumption of knowing the input distribution. The method does not rely on the labels in the calibration set but has a major limitation of knowing the task distribution which may not be true in many practical settings where uncertainty estimation is relevant (such as medical diagnostics).

This assumption may not be a negative point for the paper as any domain adaptation problem needs at least some minimal assumptions; however,  the limits of the proposed method should be studied with respect to this assumption. For instance, in the experiments how robust are the experiments with respect to the assumption of a known q(y)? In the practical applications of the method in medical domain and self driving cars, q(y) is only known up to some approximation; so understanding the robustness of the method w.r.t. to this assumption is critical in real applications. 

Also with the recent attention to calibration and uncertainty estimation in DL; I believe the acceptance bar for papers in this area has risen. Unfortunately, most papers in this area rely on completely synthetic experiments which makes their impact limited. I understand that ground truth uncertainty may not be available in some of these domains; however, other indirect metrics such as missclassification detection can be used. There are also medical datasets available (e.g. Diabetic Retinopathy) that can be used for evaluation. 

To summarize, the paper addresses an important problem of calibration under domain shift but it needs some more empirical work to show the real advantage and limitations of the proposed method in a practical setting.
review_title: Official Blind Review #2

Meta-review: The paper proposes a method called unsupervised temperature scaling (UTS) for improving calibration under domain shift.

The reviewers agree that this is an interesting research question, but raised concerns about clarity of the text, depth of the empirical evaluation, and validity of some of the assumptions. While the author rebuttal addressed some of these concerns, the reviewers felt that the current version of the paper is not ready for publication.

I encourage the authors to revise and resubmit to a different venue.
Decision: Reject