Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a novel approach for partial domain adaptation (PDA) based on the adversarial reweighting of source samples. The motivation of this paper is quite clear that the source private sample may cause negative transfer in such a partial domain adaptation setting. Reweighting is a common practice to address such challenges. In this paper, the authors propose an improved version that the sample weights can be updated with the help of adversarial learning based on a discriminator. It is a smart design that shows promising results.
main_review: Strengths:
+ As mentioned above, the motivation of this paper is quite clear and convincing.

+ The proposed adversarial re-weighting technique is simple but seems effective.

+ The whole paper is organized logically, and the discussion of related works is comprehensive enough to cover most of the literature in this field. 


Weaknesses:
- Although the motivation is clear, the whole paper is built upon the well-explored techniques such as sample re-weighting and adversarial training, making me feel like a combination. I am doubtful about the originality of this paper for the standard of NeurIPS.  

- The proposed method seems too simple, and there is no significant improvement (comparing Baseline and AR) on some datasets like DomainNet.

Question:
1. I doubt the fair comparison between the proposed method and other baseline methods. It seems the naïve version (Baseline) outperforms other PDA methods by a large margin.

In all, I recommend the score 5 due to the weaknesses and questions. I will consider upgrading the score if the authors provide a strong rebuttal.

limitations_and_societal_impact: Yes. The authors provide adequate discussion of limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a partial domain adaptation method, through an adversarial reweighting approach. Instead of learning a domain-invariant feature space, the proposed method reduces the domain shift through a source data re-weighting method.
main_review: In general, the paper is well written and easy to follow. The proposed method has some novelty. It identifies some of the limitations of previous PDA methods and proposes a re-weighting model for re-weight the source data to achieve distribution alignment, rather than learning a domain-invariant feature space, to alliterate negative transfer in PDA. However, some of the statements are not very convincing, the novelty is limited, and the experiments are not fair and complete enough. The details can be found in the limitation part.
limitations_and_societal_impact: - It claims that previous PDA methods cannot work well on challenging benchmarks, like VisDA-2017 and DomainNet. It would be better to give more details on what aspects are these benchmarks more challenging, and compared to which benchmarks?

- The main cause of mistakenly assigning high weights to the source-only-class data for the previous methods is not very convincing. What are the key insights of the causes? It is good that Section 2 demonstrates the limitations of the previous methods through some experimental findings. However, it lacks more in-depth analysis, and thus the justifications are not convincing enough.

- It claims that previous PDA methods heuristically setting the weights of the source data, which is not precise. For example, the discriminator-based methods [5, 46] also automatically learn the source data weights through adversarial training rather than heuristically setting them.

- How/why does the proposed method can deal with the limitations of previous PDA methods? The proposed method assume that by re-weighting the source domain data, the domain distribution can be mitigated. However, this assumption is very strong. The domain shift between the source and target domains may not be able to be reduced by simply re-weighting the source domain data, especially for the so-called challenging benchmarks with large domain shift, which motivates the subsequent UDA methods that reduce the domain shift by learning a domain-invariant feature space.

- It argues that the proposed adversarial learning-based re-weighting method is different from previous kernel MMD-based methods and optimal transport model. However, to my understanding, they are all very commonly used criteria for aligning different distributions in domain adaptation, the only difference is the chosen metric/divergence for measuring the distribution shift. Thus, the novelty of the re-weighting method is limited.

- The comparisons to the previous method in the experiments part are not fair and complete. First, the author adopts different network architecture compared to previous PDA methods. For example, a state-of-the-art spherical network is adopted for the classifier C. Second, some comparison results are missing. For example, the results on ImageNet-Caltech, DomainNet, and VisDA-17 of some previous PDA methods that perform better than the proposed method on Office-31 are missing, e.g. RTNet[6] and DPDAN[43]. Thus, it is hard to judge the superior of the proposed method over these methods.

- The analysis of the experimental results is not sufficient, especially on why does the proposed method only perform comparably to other PDA methods on Office-31 dataset?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new method called Adversarial Reweighting (AR) for partial domain adaptation (PDA) where the target label space is a subset of the source label space. AR aims to estimate the weight of source instances via an adversarial training manner and Wasserstein distance and then exploits the weight to train a simple objective (re-weighted cross-entropy on source + entropy minimization on target). Results on several PDA benchmarks verify the effectiveness of AR.
main_review: Pros:
Different from previous PDA methods relying on class-level weights, the proposed AR method explores Wasserstein distance and learns the instance-level weight for each target sample, which is interesting and new in this field.
Promising results on several PDA benchmarks, especially 78.3 on the Office-Home dataset.

Cons:
the results of [BA^3US, ECCV2020] are not fairly compared since a weak variant BAA is compared throughout this paper. It seems that BA^3US performs better for Office and some other tasks like I->C.
it is also very confusing that the baseline method shown in each table achieves much higher results than previous PDA methods.
In figure3, the parameters sound sensitive, how do the authors determine these parameters for an unsupervised domain adaptation task? In fig.3(b), the authors need to compare AR with other PDA methods like BA^3US to study the effectiveness of AR.

minor comments:
if the Wasserstein distance is replaced by a vanillna JS distance, how AR performs for these PDA tasks?

[BA^3US, ECCV2020] Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng. A balanced and uncertainty-aware approach for partial domain adaptation. In ECCV, 2020.
limitations_and_societal_impact: See my comments above.
review_title: null

Meta-review: This paper points out that the existing partial domain adaptation method has negative transfer when some source-only data are mistakenly assigned high weights. The authors thus propose an adversarial reweighting approach to minimize the distance between reweighted source and target domains. The proposed method outperforms existing methods on several domain adaptation datasets.

However, there are several concerns. 1) The motivation of the paper is not strong enough. The claims in the paper are not convincing. For example, the paper claims that existing PDA methods do not work well on the challenging datasets, such as visDA2017, but it does not explain why such datasets are challenging. Also, the analysis of the drawbacks of existing methods is not thorough enough. For example, the discriminator-based methods [5,46] also automatically learn the source domain weights using adversarial learning rather than heuristically setting them. 2) The proposed adversarial learning method differs from existing methods only in distribution divergence measures. The novelty of the proposed method is limited. 3) The performance of the proposed method does not clearly outperform existing methods by a large margin. Also, the comparison might not be fair because the authors aopt a different network architecture compared to previous PDA methods. The authors have added a long list of new results in the rebuttal, which means the manuscript needs significant modifications of the experimental section.

Overall, the paper will have some practical values if the experimental results are further strengthened. However, given the above concerns, especially those on the novelty of the paper, we think the paper does not meet the acceptance requirements of NeurIPS.

Decision: Accept (Poster)