Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a simple method for calibration that is robust to distributional shifts in the data. The proposed method extends the temperature scaling approach by learning to predict a suitable temperature value on an unseen domain. 
The authors show empirically that their approach (MD-TS) can outperform other approaches (including vanilla temp scaling) on both in- and out-of-distribution test sets. 
Along with this, the authors also provide a theoretical justification for their proposed method. 

Calibration methods generally fail to remain robust on out-of-distribution examples and this paper works towards solving an important problem

strengths_and_weaknesses: Strengths:
- The proposed method is both simple and effective at calibrating several models over multiple domains (an important problem) (+ originality, +quality + significance)
- The paper provides an interesting theoretical justification for their proposed approach which gives a better intuition for regarding calibration over multiple domains (+originality,  + quality, +significance)
- The analysis seems comprehensive; the authors also provide ablations for their approach
- The paper is well explained and easy to follow (+clarity)

Weaknesses:
- I felt that the chosen baselines were not very strong. [1] show that temperature scaling by itself isn't very robust on out-of-distr. examples. Perhaps the authors could compare against some of the methods shown in [1] to be more robust such as deep ensembles and MC dropout. 


[1] Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Dillon J, Lakshminarayanan B, Snoek J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems. 2019;32.


Overall, I feel this is a worthy contribution and vote for acceptance. Willing to increase my score if there's a more comprehensive comparison to previous methods. 
questions: Why was the temperature prediction model chosen to be linear? 
Prior work has experimented with methods to learn to predict a temperature value (https://arxiv.org/pdf/1903.00802.pdf): they use a 2 layer MLP to predict the temperature. 

The ECE metric can be very sensitive when the number of examples in the test set are small and are not distributed over all bins. In such cases it might be interesting to also compare on other more standard metrics (apart from MAE) like the Brier Score.

limitations: The authors do not have a section detailing potential societal impacts, but they talk about how calibration can be useful in critical applications in the introduction.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a type of temperature scaling that is more robust on out-of-distribution dataset. The work first learns K temperatures for K in-domain datasets. Then a linear regressor is learned to predict the K temperatures given feature embeddings, hence learning to predict a suitable temperature given a sample. The linear regressor is then applied to out-of-distribution samples, calibrating a model’s output. The paper verifies the idea on three datasets and illustrates improved calibration results by the proposed method.
strengths_and_weaknesses: Strength
-	This paper asks an important question: How should one choose temperature for predictions from out-of-distribution inputs. 
-	The empirical result shows improved calibration score with the proposed method over the baselines.

Weakness
-	The assumption in the Theorem 5.2 is strong. The assumption starts by assuming that Out of distribution is similar to a mixture of in-domain distributions. This assumption, however, is too strong without proper citations or empirical results. If this assumption holds, then simply learning a universal temperature with data collected from the in-domains may be enough. 
questions: Questions
1.	In Definition 2.2, is the number of pooled data identical in each domain? If not, taking the average should not be enough. It should reflect the number of examples in each domain as well. Hence, taking expectation over K domains should be modified.

limitations: -	The concept of the proposed method is tested on image domain. It would be interesting to see the method on other domains, such as NLG (translation).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose a multi-domain temperature scaling method for distribution calibration with distribution shifts. The heterogeneity of temperature is used to improve the robustness. Numerical experiments and theoretical results are presented to validate the method.
strengths_and_weaknesses: strengths:

1. well-motivated
2. clear presentation and good writing
3. extensive results to support

weakness

1. more rigorious  experimental results (more baselines and efficiency, detailed in the limitations)
2. more introduction to highlight the motivation
questions: Like TS, MD-TS should also adopt a validation dataset, right? Can we do multi-domain without the validation dataset?
limitations: 1. The baselines only include TS and max-softmax. Can you compare with the more recent method, such as node, sde-net and others?
2. How is the optimization goes? Is MD-TS still efficient, compared with TS?
3. In the introduction, only an example of multi-domain uncertainty? I suggest a more detailed elaboration on this motivation, since it’s the main contribution and claim.
review_title: null

Meta-review: metareview: Reviewers find the paper original, useful, thorough in its numerics (in the revision), and clearly written.
confidence: Certain
Decision: Accept