Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the unsupervised domain adaption (UDA) problem. Particularly, the paper proposes to impose neighbor class consistency on target features to preserve intrinsic discriminative nature of target data and presents an entropy-based weighting scheme to improve robustness against the potential noisy neighbor supervision. The motivation of the paper is clear and the method is well presented. Extensive experiments show the effectiveness of the proposed method. However, the paper suffers some problems, such as
1. The idea of neighborhood consistency in UDA is not very significant. It lies on the assumption that target features from source pre-trained model are highly intrinsic discriminative. However, the distribution discrepancy between source domain and target domain may be very large, such as unsupervised cross-dataset person re-identification. It is not sure whether such assumption/observation is still satisfied.
2. From Table 3-5, NC-SP outperforms the state-of-the-art methods with only a small margin. From Table 3, SC, FR , SP make a great contribution to the performance. It is suggested to add some comparisons to Table 3-5 that only the proposed VNC or ENC is used without other tricks.
3. It is suggested to evaluate the influence of hyper-parameter \lambda in Eq. (10) and (11)
4.  It is suggested to evaluate neighbor size K on more datasets, in order validate the association of K with target dataset and number of classes.
review_title: The idea of neighbor consistency heavily lies on the assumption that target features from source pre-trained model are highly intrinsic discriminative. It is a controversial assumption. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles Unsupervised Domain Adaptation. The authors focus on the intrinsic discriminative feature for target samples. The proposed method, Neighborhood Class Consistency among target samples and augmented ones, is proposed as a set of multiple losses to calculate the consistency from several aspects. The experimental results show that the proposed method achieves state-of-the-art performance using the same backbone network.

**Pros**
- The consistent approach making use of neighborhood structure in the target domain is well motivated and easy to understand.
- The proposed method achieves SoTA performance on several benchmark datasets.

**Cons**
- There is a related paper by Gu et al. (2020), although it is not referred to in this paper. Gu et al. also focus on a pseudo labeling approach and discriminative distributions in the target domain. Additionally, the experimental results show that the performance is very close to that in this paper.  Although the detailed method is different, the authors should have compared them qualitatively and quantitatively.

Gu et al., Spherical Space Domain Adaptation with Robust Pseudo-label Loss. In CVPR, 2020.

- Section 4.2 says that the authors use different values for the hyperparameter, such as $\lambda$, for each dataset. How do they tune them? Since the method is for UDA, there cannot be a validation set in the target domain. Therefore, different values of $\lambda$ seem unnatural. Alternatively, the authors should report the performance with different hyperparameter values not only for the number of neighbor samples in Fig. 3 (b) but also for $\lambda$ and triplet margin $m$.　

**Minor comments**
- In the main text, there are some points the authors should insert a white space before a bracket and a citation.
- In the last paragraph of Section 1, an abbreviation "NC" is used without an explanation. What does it stand for?
- How often do we need to update the memory bank $\mathbb{V}_t$ while a model is trained? Frequent updates of the memory would be time-consuming.
-The format of References is messy. Some papers, such as CyCADA, should not be arXiv preprints but published ones. Some proceeding names are capitalized while the others are not. Some conference names are abbreviated while the others are not.
-The blank Appendix section should be deleted.

**Overall rating**

Although there should be more descriptions about the existing method and experimental results, the reviewer is leaning toward acceptance. The rating can be upgraded if the authors could solve the cons above.

**Additional comment after rebuttal**

Unfortunately, the reviewer would like to downgrade my first score. The remaining concerns are about R2-A1 and R2-A2.

In R2-A1, the authors add some discussions about experimental results and explanations about Gu et al. (2020). The second discussion about generating pseudo labels would be the most considerable theoretical difference between the proposed method and Gu et al. (2020). The authors state that the proposed method addresses the problem by incorrect pseudo labeling of Gu et al. (2020); however, they fail to show quantitative nor qualitative discussions that support the statement. The third discussion seems just showing the proposed method achieves similar performance to that of Gu et al. (2020). As discussed in the fourth comment, the authors could add another result showing the proposed method is complementary to Gu et al. (2020).

In R2-A2, the authors honestly state that the hyperparameters are tuned according to the test data evaluation. However, it should be avoided to evaluate unsupervised domain adaptation methods since there are no labeled data in the target domain in a real setting. Moreover, Figure 4 (a) shows that the proposed method is sensitive to lambda on Office31 dataset. When lambda=0.5 on all dataset, the proposed method achieves SOTA on VisDA as shown in Table 5, but seems to fail on Office31.
review_title: A consistent and effective approach making use of neighbor samples

final_score: rating: 6: Marginally above acceptance threshold
review_content: Based on the observation that the target features from source pre-trained model are highly intrinsic discriminative and have a high probability of sharing the same label with their neighbors, a simple but effective method to impose Neighbor Class Consistency on target features is proposed. The experimental results are promising.
[1] Compared with VNC, the improvement of ENC is very limited. Whether the authors have repeated the experiments in several times. It is better to show the variance of the results in Table 2, which may be more suitable to demonstrate the effectiveness of ENC.
[2] In Eq.(9), it forces that the anchor samples should be closed to the neighbor samples by comparing with the distance between the anchor sample and the augmented sample. What is the advantage of such a strategy compared with triplet loss. It is also better to show the comparison in the experiments.
[3] For the neighbors, when k is equal to 1, the proposed method achieved the best performance. Hence, the neighbors with the minimum distance could not guarantee that they share the same labels. How does the proposed method estimate the confidence.


review_title: In this paper, a simple but effective method to impose Neighbor Class Consistency on target features and to learn the discriminative features is proposed. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed neighbor class consistency regularization together with an entropy-based weighting factor to tackle the problem of unsupervised domain adaptation. Another self class consistency regularization was further introduced to help training. The difference between "neighbor class" and "self class" is the positive pair selection, where "neighbor class" uses k-nearest neighbors as positive pairs, and "self class" uses an augmented version of the anchor itself.

The paper is generally well-written and easy to follow. Sufficient experiments are conducted. However, I think the quality of the paper is not publishable now due to the following weaknesses.

Major weaknesses:

1. The core contribution of this paper is the class consistency loss (Eq. (5)&(7)) between the anchor and its neighbors, however, similar ideas have been investigated in existing works, e.g. [A, B]. But unfortunately, this paper even did not include them in the related works.



2. I think the work lacks novelty. The memory bank in Sec 3.3 (1) is similar to [C, D] and the data augmentation rules in Sec 3.4 are similar to [E]. The consistency loss is popular in knowledge distillation tasks. In my point of view, the paper heavily borrowed techniques from existing works.  The author should at least acknowledge the relations between the paper and these works.



3. It's somewhat counter-intuitive that the paper required the similarity between the anchor and its augmented version to be closer than the similarity between the anchor and its nearest neighbor (Eq. (9)). A situation may exist: the augmentation is very heavy while the nearest neighbor shows very similar visual features with the anchor.
The author has shown the effectiveness of Eq. (9) in Table 2, namely "FR". However, it's not convincing enough, I recommend to evaluate "ENC + SC + SP (NC-SP)", i.e. removing "FR" from the whole model.


Minor weaknesses:

4. The paper claimed its state-of-the-art performances on various benchmarks. But it missed some previous methods, e.g. [F]. The paper achieved even inferior performance than [F] on the VisDA17 benchmark, i.e. 86.2 in this paper v.s. 87.2 in [F].

5. As I mentioned in weakness-3, it's better to use "minus" rather than "plus" in ablation studies.

6. More hyper-parameter analysis would help improve paper quality. For example, as described in Sec 4.2, the author adopted different $\lambda$ values for different datasets. It's better to show how much does the value of $\lambda$ impact the final performance.



[A] Regularizing Class-wise Predictions via Self-knowledge Distillation. CVPR 2020.

[B] Invariance Matters: Exemplar Memory for Domain Adaptive Person Re-identification. CVPR 2019.

[C] Unsupervised Feature Learning via Non-Parametric Instance Discrimination. CVPR 2018.

[D] Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020.

[E] A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.

[F] Contrastive Adaptation Network for Unsupervised Domain Adaptation. CVPR 2019.


=========================================================================================================

Update:

Thanks for the authors' response. But unfortunately, the most significant concern has not been addressed well, which is regarding of the paper's novelty. I thought the contributions of this work are incremental, and the authors' rebuttal did not convince me well. Btw, the authors claimed that [A] used KL-divergence while their work used mutual information maximization, however, minimizing KL-divergence is actually one kind of mutual information maximization. What's more, the authors even did not clearly point out which kind of mutual information maximization they used in either the manuscript or the rebuttal. Thanks again for the authors' efforts, but I choose to maintain my original score.
review_title: Official Blind Review #1

Meta-review: This paper proposes a method (via a novel objective) for feature alignment between source and target tasks in an unsupervised domain adaptation scenario.

Pro
- the proposed approach is sensible in many realistic scenarios of distribution shift
- the submission provides an extensive empirical evaluation establishing state of the art results on several benchmark tasks

Con
- there is no thorough discussion of the the underlying assumptions (when should we expect them to hold? for shich types of tasks? which types of shifts? can they generally be reliably tested? from which type of data? unlabeled?)
- one reviewer raised concerns over novelty, which should be more clearly addressed before publication
- two reviewers raised concerns over use of target data for hyper-parameter selection, which seem valid; these should be fixed or clearly explained (and implications of this be discussed) in subsequent versions of this work

I agree with concerns of the reviewers (the last two points), and would therefore not recommend this work for publication in the current state.
Decision: Reject