Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an uncertainty ( density ratio) of source-target prediction to adapt for the target domain. The methods also validated in the robust semi-supervised tasks.
main_review: The paper proposes an uncertainty ( density ratio) of source-target prediction to adapt for the target domain. The novelty is somewhat limited; other works also tackled the distributional shift using the confidence and prediction of domain classifiers.[d][f]. The paper is lacking in some of these aspects: 
Clarifications: 
What is the $\sum$ in Eq 3
How to present work different/related adversarial domain adaptation methods; this discussion will clarify the paper further.  using the Bayes theorem, the Differentiable density ratio estimation can be treated as the discriminator's prediction.( discussed in sec 2.3) 
Do source and target predictors share some parameters? 
In Eq 6and  7, how to obtain y for unsupervised domain adaptation case. Is it a class label or domain label?
Performance: 
The performance is not compared with recent state-of-the-art methods in domain adaption[a][b][c][d][f] in real workd dataset such as VisDA.
Related comparison:
The paper missed some of the useful references that tackle the domain adaptation using uncertainty.[d][e][f]. I suggest authors to compare their methods with these related works in terms of performance.


[a] FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation, CVPR 2021
[b]d-SNE: Domain Adaptation using Stochastic Neighborhood Embedding, CVPR, 2019
[c]Self-adaptive Re-weighted Adversarial Domain Adaptation, IJCAI 2020
[d]Unsupervised Domain Adaptation via Calibrating Uncertainties, CVPR workshop 2019
[e]Model Uncertainty for Unsupervised Domain Adaptation ICIP 2019
[f] Attending to Discriminative Certainty for Domain Adaptation, CVPR 2019
summary_of_the_review: The paper is lacking in terms of performance on domain adaptation tasks as compared to state-of-the-art methods. Some more clarification about the method will increase the readability of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problem of uncertainty calibration under distribution shift. By using a domain classifier in the distributional robust learning (DRL) framework, the authors estimate the density ratio between the source and target domain to achieve well-calibrated predictions under domain shift. A regularized DRL framework is further proposed to promote smoothed model prediction and improve the calibration. Experiments on Office31, Office-Home, and VisDA-2017 demonstrate the superiority of DRL over empirical risk minimization (ERM) and the temperature scaling method measured by expected calibration error (ECE), Brier Score, and reliability plots.
main_review: Strengths:
+ The authors incorporate the density ratio estimation method into the existing distributional robust learning (DRL) framework and successfully achieve a better calibration under domain shift.
+ They further integrate their method as a plug-in module in downstream applications such as unsupervised domain adaption and semi-supervised learning, leading to significant improvements. This part is new to me and riveting. I appreciate the authors for not only making the predictions more calibrated but also achieving higher performance.
+ Competitive performance on various tasks including Office31, Office-Home, and VisDA-2017 demonstrate the superiority of DRL over empirical risk minimization (ERM) and the temperature scaling method measured by expected calibration error (ECE), Brier Score and reliability plots.
+ This paper is well-written and easy to follow.

Weaknesses:
+ My first major concern is the novelty of this paper. There are mainly two technical parts of this paper: the density ratio estimation method and distributional robust learning (DRL). However, density ratio estimation by a domain classifier is common practice in domain adaptation and transfer learning. Moreover, this technique has already been successfully tailored into calibration under domain shift [1][2]. As for the distributional robust learning (DRL) framework, it has been successfully proposed in Liu & Ziebart, 2014 as stated by the authors.
The differences between these related works with the proposed method should be fully discussed to make the contribution enough for this top-tier conference.
+ Another major concern is the performance of FixMatch on CIFAR10. The reported number (91.60) is much lower than that reported in the original paper of FixMatch, which is about 95.7?
+ What is the definition of human selection frequencies (HSF)? Why should the estimated weight match HSF? I highly encourage the authors to deeply analysis this topic to bring more insights.
+ Why Brier Score of Temperature Scaling (TS) is missing in Figure 4?

[1] S. Park, O. Bastani, J. Weimer, and I. Lee. Calibrated prediction with covariate shift via unsupervised domain adaptation. 2020.

[2] X.Wang, M. Long, J. Wang, and M. Jordan. Transferable Calibration with Lower Bias and Variance in Domain Adaptation. In NeurIPS, 2020

summary_of_the_review: A uncertainty calibration method under domain shift with a distributionally robust learning framework that show competitive performance on comprehensive experiments but lacks enough novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In the paper, the authors propose a framework for learning calibrated uncertainties under domain shift based on a distributionally robust learning (DRL) framework. As a plug-in module, the proposed method benefits downstream tasks of unsupervised domain adaptation and semi-supervised learning in experiments. 

main_review: 1. The paper studies an interesting and important research question: calibrate uncertainties under domain shift. And the authors proposed a novel method based on DRL and density ratio estimation. The proposed method can be seen as a plug-in tool for existing methods. 
2. The paper may overclaim in the abstract that the authors "consider the case where the source distribution differs significantly from the target distribution". The proposed method has a covariate shift assumption. Then how could the two distributions differ SIGNIFICANTLY? Does the proposed method still work when the support of the source and target domain are (largely) different?
3. Density ratio estimation is known to be hard when the input data is complex/in high-dimension. I have some doubts about how well the density ratio is estimated in the proposed method. I think further investigation in the learned density ratios is necessary to understand how the proposed method works. Figure 3 shows a coarse study on this point but I'm not convinced by this simple result. The discussions and ablation studies may not enough to show this point. Also, the threshold of low/high HSF seems quite artificial. 
4. The proposed method is based on a strong covariate shift assumption, which is very likely to be violated in real-world applications. The authors show that self-training may help to relax this assumption. But how about the proposed method without self-training? Self-training is already a strong additive in weakly-supervised learning for achieving good performance. Perhaps it is the self-training part in DRST that makes the proposed method work? I hope to see more results for the proposed method without self-training. 
summary_of_the_review: The research question is important and useful but the writing of the paper may overclaim the usage of the proposed method. Also, it may be hard to conclude how the proposed method works under the current evidence.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes scaled uncertainty prediction in the context of unsupervised domain adaptation. The problem in unsupervised domain adaptation is to obtain predictions in a target domain where no (or few) labels are available. A secondary problem then is the calibration of predictive uncertainties in the target domain. This paper proposes a model of uncertainty proportional to the (log) of density ratios between domains. Results include ECE and Brier scores on two common Domain Adaptation datasets. 
main_review: Strengths:

* The proposed use of density ratios makes an attempt to quantify the domain shift. This is an improvement over other works which take domain shift for a given. 
* Uncertainty is evaluated using both ECE and Brier score. (Usually papers choose only one of the two.) 
* Section 2.2 derives use of density ratios to scale (temperature of) uncertainty from theory in Distributionally Robust Learning, which is a novel combination of views. Subsequently, Figure 3 shows that the density ratios correspond to human interpretation of domain differences. This correspondence validates the design choice for density ratio. 

Weaknesses:

* “Intuitively, if the test sample is highly unlikely in the training distribution, then the resulting confidence levels should be lowered”. I don’t agree with this intuition. Let’s say the training distribution is of synthetic renderings and the test distribution of real-life pictures, then confidence could still be high when the same object is depicted. What would be a counter-argument to this statement and how does that influence the results?

* The human selection scores seem to be a major motivation of this paper, but the figure misses error bars. When the ratio estimates change from 2.2 to 2.6, what is the noise in these numbers? (In other words, how statistically significant is this observation?)

* The calibration and training of density ratios is not well explained. Densities are known to be hard to interpret in high dimensions [3]. Could the relevance of these density ratios be shown via some experiment?

* Question: From figure 4 it seems that DRL generally shows signs of underconfidence (C-A negative), while TS and VADA signs of overconfidence (C-A positive). Could this effect be explained?


Minor comments,
  * Section 3: why use 5 bins for ECE? A lower number of bins introduces more bias, especially for high accuracy models (where the last bin is most occupied) [4]. This makes the ECE results less reliable.
  * Section 2.2: This section has many connections to the research in GANs, where a similar density ratio is estimated [1,2].

NB: These are meant as suggestions and do not influence the review. Use of papers on arXiv is not part of the review procedure. 

[1] Goodfellow et al. "Generative adversarial nets." NeurIPS 2014

[2] Arjovsky et al. Wasserstein generative adversarial networks ICML 2017

[3] Nalisnick, et al. "Detecting out-of-distribution inputs to deep generative models using a test for typicality." arXiv 2019

[4] Roelofs, et al. "Mitigating bias in calibration error estimation." arXiv 2020

summary_of_the_review: This paper derives a new scaling for predictive uncertainty in unsupervised domain adaptation. Motivated by Distributionally Robust Learning theory, the scaling, density ratios, are verified against human frequency scores. Results show better calibration under similar (slightly better) predictive performance.
review_title: null

Meta-review: This paper investigates the problem of uncertainty calibration under distribution shift. Based on a distributionally robust learning (DRL) framework, the paper estimates the density ratio between the source and target domains to achieve well-calibrated predictions under domain shift. 
As a plug-in module, the proposed method benefits downstream tasks of unsupervised domain adaptation and semi-supervised learning in experiments on Office31, Office-Home, and VisDA-2017, demonstrating the superiority over empirical risk minimization (ERM) and the temperature scaling method measured by expected calibration error (ECE), Brier Score, and reliability plots.

After extensive interactions and discussions on the paper, the final scores were 6/5/5/5. AC considered the paper itself, and all reviews, author responses, and discussions, and reject the paper from the following concerns:
+ *Overclaimed Novelty*: This paper is mainly based on the well-established competitive distributionally robust learning (DRL) framework. The contribution of the newly-proposed regularization form that can further promote smoothed prediction and improve the calibration performance is relatively trivial. The designs of the resulting predictive form and the learning using new gradients, mentioned by the authors in the rebuttal, need further exploration and elaboration to verify its contributions.
+ *Lack of Clarifications*: Some key points mentioned by several reviewers are still not clear. For example, how well the density ratio is estimated, especially in high dimensions? Further, a positive correlation between HSF and density ratio is not enough to prove the main argument of the paper.
+ *Some statements are not well-supported*: For example, the statement that "the harder the examples are, the farther away the examples are from the source domain", claimed by the author in the rebuttal, is untenable.

In summary, this paper studies a promising research direction of uncertainty estimation, but the work cannot be accepted before addressing the reviewers' comments. I suggest the authors to substantially revise their work by incorporating all rebuttal material as well as addressing the remaining concerns.
Decision: Reject