Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary**
The paper proposes to use the distributionally robust learning (DRL) for unsupervised domain adaptation. First, the authors demonstrate how differentiable density ratio estimation can be done for source and target domains in an end-to-end manner. Following this, the authors demonstrate how confidence estimation (reliant on DRL) can be utilized for self-training based approaches for unsupervised domain adaptation. In terms of results, the authors show that the proposed approach — Distributionally Robust Self-Training (DRSL) — can provide competitive performance on t he VisDA 2017 benchmark. Furthermore, DRSL is able to achieve more calibrated probabilities (useful for more calibrated uncertainty measures) with competitive predictive performance. Finally, the authors provide some intuition as to why DRSL results in increasingly aligned conditional source and target distributions due to increased focus on learning shape features.


**Strengths**
- With the exception of a few points raised in the weaknesses section, the paper is generally well-written and easy to follow. 
- When initialized with ASG, the proposed approach shows roughly ~1% improvement compared to baselines and prior approaches on the VisDA benchmark. This improvement is coupled with increased sample-efficiency (significant within 95% standard error), more calibrated probabilities and monotonic reductions in the measure distribution gap. This is perhaps one of the strongest positive points in support of the proposed approach.
- The presented analysis about the kind of features DRST increasingly focuses on (although not fleshed out more extensively) is interesting and can likely provide more insights into how to improve on DRST in future work. 

**Weakesses**
- One of the minor weaknesses with the current draft is readability. I found it slightly hard to digest the proposed approach in the form it is presented in the current draft. For instance, it was unclear as to how gradient expressions for the classification network from the expected target loss for DRL from equation 8 were aligned with the proposed reduction in equation (6) — specifically, what data are the gradients estimated on? In context of DRST, is this on labeled source + pseudo labeled target per-epoch from an earlier DRST epoch? While sections 2.1 and 2.2 in themselves are more or less straightforward to follow, putting them together in context of DRST seems slightly hard for a reader. Improving this in terms of clarity would definitely benefit the paper.
- While the experiments attempting to understand the kind of features DRST focuses compared to baselines and other approaches using Grad-CAM are useful and definitely provide some insight, the paper would definitely benefit if this claim were backed by quantitative metrics of some kind. Namely, (1) to understand if DRST indeed focuses on shape features, what is the mean IoU of region highlighted by grad-cam (w.r.t. the predicted class) with (say) the segmentation mask of concerned object in the image and (2) since Grad-CAM is being relied on as a tool to obtain saliency maps, observed saliency maps should also be provided with measures of fidelity — how reliant are Grad-CAM maps in terms of providing feature importance visualizations in context of the models and dataset involved (see [A] for adopted benchmarks and metrics, namely the Remove and Re-train (ROAR) metric). Since focus on shape features if proposed as a cause for declining source and target conditional distributions, including measures from the aforementioned points would strengthen this claim.

[A] - A Benchmark for Interpretability Methods in Deep Neural Networks
review_title: Promising approach to improve self-training for unsupervised domain adaptation

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary:
This paper introduces distributional robust learning (DRL) for the unsupervised domain adaptation task. The proposed DRL based approach is claimed to better measure model prediction during self-training with an auxiliary discriminator. This paper also recognizes an optimization issue of conventional DRL and proposes an end-to-end solution via density ratio estimation. 
 
However, I have the following concerns,

1.The generalization error (both theoretically and empirically) of the gradient approximation is unclear. It is necessary to analyze how effective and under what conditions the proposed approximation can work for the expected target loss optimization.

2.It needs elaboration why the density ratios can be directly replaced as discriminator predictions, which seems not straight-forward and is the main difference to the conventional DRL. 

3.There are some related work not covered in the paper. It is quite surprising that Ben-Tal et al. (2013) is not discussed, also a few later application work (Shafieezadeh-Abadeh et al. 2015; Sinha, Namkoong and Duchi 2018; Hu et al. 2018). More importantly, some existing study already touched high-dimension data and self-training (19NIPS). A thorough discussion of how this work makes unique contributions is required.

4.The baseline results on Office31 and Office-Home datasets are missing. Besides, since it claims to better measure prediction uncertainty, a direct comparison of prediction confidence with the baselines could better justify the superiority of the proposed approach.


Ben-Tal et al. "Robust solutions of optimization problems affected by uncertain probabilities," Management Science, 2013
Shafieezadeh-Abadeh et al., “Distributionally robust logistic regression,” in NeurIPS, 2015
Sinha, Namkoong and Duchi, “Certifiable distributional robustness with principled adversarial training,” in ICLR, 2018.
Hu et al., “Does distributionally robust supervised learning give
robust classifiers?” in ICML, 2018
Najaf et al., "Robustness to Adversarial Perturbations in Learning from Incomplete Data," NeurIPS, 2019


review_title: submission 2815 review

final_score: rating: 6: Marginally above acceptance threshold
review_content: I find the paper to be well motivated. Self-labeling has proven to be a useful approach for unsupervised domain adaptation. And since wrong pseudo-labels in the target domain result catastrophic failure in early iterations, it makes sense to calibrate the production of pseudo-labels through the use of uncertainty estimation. This is done through the framework of distributionally robust learning. 

I find three central concepts in the method: First, the formulation of the problem as a min-max game between the adversary and the predictor (appears in Eq.1 and Eq.3). Second, the relaxation of the predictor into a formula involving a ratio of densities (Eq.2, Eq.4 and Eq.5) and third the estimation of the gradients of the target loss through the use of source data. While the individual components seem well motivated and clear, I struggle to see the connection between them. For instance, how does figure 1(a) related to figure 1(b)? What is the intuition of using a ratio of densities as a good uncertainty estimation? How can source data be used to approximate gradients on the target? A clearer explanation of the connection could be useful or at least a sketch of the proofs be provided in the main text. 

In addition, while Alg.1 is shown to be a relaxation of the original formulation, Alg.2 seems to be an adhoc attempt to combine the DRL framework with self-training. Why is this the correct formulation to use? At the very least, an experimental comparison to other possible approaches can be used - for instance, one can apply a standard UDA approach instead of Alg.1 and choose, based on standard confidence measures, the top confident labels for a subsequent iterations of Alg.2.

The use of three different datasets though a variety of experiments demonstrates the usefulness of the approach.  However, some experimental evaluation is missing in my view. First, an ablation study for the different components of both Alg.1. and Alg.2 would be usefull. For instance, in Alg.1, viewing the density ratios as an uncertainty estimation, what is the effect of using other approximations for the uncertainty, either by the adjusting this ratio, or using existing uncertainty estimation methods. How does a different choice for the flexibility of the adversary affect UDA results (Eq.3)  For Alg.2 what is the effect of using different pseudo-label portion p? 

Further, the paper claims to better captures the shape features, but this is not shown.  One could use the evaluation shown in [1], for instance, to demonstrate this. 

Overall, on the positive side,  I found the problem well motivated and important. The individual components of Alg. 1 are well explained and shown theoretically. The overall approach well evaluated on 3 datasets and some of the components evaluated.
On the negative side, clarity of the paper could be improved, particularly with regards the the connection between the main components. Additional experiments, and an in particular an ablation study regarding Alg1. and Alg.2 would provide a much better understanding of the method. 

[1] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR 2019. 

review_title: Well motivated idea and interesting formulation, but lacking clarity around the main components and incomplete evaluation. 

Meta-review: The paper proposes a rather complex algorithm for unsupervised doamin adaptation.
While the paper provides detailed explanation, some motivation and some experimental resulst,
it does not provide any theoretical guarantees for its performance. More concerning, since domain adaptation
can only succeed when there is a close relationship between the source and target tasks, and only with algorithms 
that take that relationship into account, any scientific proposal for domain adaptation should include a clear
discussion of the assumptions driving the proposed algorithms and of the circumstances under which the proposed approach 
may or may not work. This is missing in the current submission. 

More specifically, a similar ocncern was voiced by Reviewer 3 
Namely ".The generalization error (both theoretically and empirically) of the gradient approximation is unclear. It is necessary to analyze how effective and under what conditions the proposed approximation can work for the expected target loss optimization." Thsi point was not addressed in teh authors' rebuttal.

Anotehr key concerning point that was also brought up by reviewer 3 read:
"It needs elaboration why the density ratios can be directly replaced as discriminator predictions, which seems not straight-forward and is the main difference to the conventional DRL." In response the authors cite the paper by Bickel et al 2007 but it falls short of addressing the well know fact that density ratio cannot be reliably estimated from samples of bounded size. The authors should have explained specific assumptions that can make this step of their algorithm og through.
Decision: Reject