Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents a contrastive learning framework for Semi-Supervised Domain Adaptation. A basic assumption of few labeled target samples is made according to the Semi-Supervised setting. The key idea is to apply an instance-level and inter-domain contrastive alignment, using labeled source samples, few labeled target samples, and pseudo-labeled target samples. For instance-level contrastive loss, strong augmentations are used as positive pairs. For inter-domain alignment, cluster centroids are aligned across the two domains. The proposed method shows consistent improvement over multiple datasets.
main_review: Strengths
=========

a. The approach is simple. It is interesting to see that the two contrastive learning losses collaboratively improve the performance.

b. The results look promising.

c. The manuscript is easy to read


Weaknesses / Scope of improvement
==============================

a. Comparison against prior arts: 

* The proposed contrastive losses are inspired by prior works [24, 26], however no comparison is made against these methods in the experiments (or did I miss it?).

* Since the proposed method uses strong augmentation, which can be seen as additional data, it is expected that the performance would improve. However, the results reported for the baselines are taken from their papers (L210). Considering that the baseline papers use weak augmentations (e.g. MME [42] uses horizontal flipping and random cropping), in my opinion, this comparison is unfair. The baselines should be evaluated using strong augmentations as shown in Table 4 (e.g., consider RA + Grayscale augmentation for competing methods).


b. Quality of pseudo-label / few-shot samples: According to Eq. 4, all the target samples are assigned a pseudo-label which is the model’s predicted output. This would also contain misclassified target samples. However, there is no discussion on how to mitigate the noise in pseudo labels. Moreover, there is no discussion on the performance when the one-shot labeled target samples are outliers.

c. L283 (“Instance Contrastive Learning improves pseudo-label accuracy”): It is unclear whether this is truly the case, since the individual modules don’t yield high performance on their own (L242). Moreover, there is no discussion regarding how the combination of the two losses gives significant improvement. It would be interesting to study why this is the case.

d. Supplementary L11-12 (“results in the main paper are achieved through one-time training”): Could the authors clarify if multi-run statistics are reported in the main paper? For instance, it seems that the result presented in Table 1 of the paper (Office-Home - Alexnet - CLDA - Mean = 57.4%), is around 1% less than that mentioned in the Supplementary Table 4 (Mean = 58.31%). Moreover, this discrepancy suggests that the results do not truly indicate the average-case performance of this method. The authors are suggested to present multi-run statistics for all experiments, since the performance depends upon the chosen one-shot / few-shot samples. 


Additional clarifications
===================

a. Algorithm 1 (Supplementary): The source centroids are updated using the exponential average as shown in Eq. 3. However, the target centroids are not updated likewise. Could the authors clarify the intuition behind this choice?

b. How is the validation set constructed (Fig. 3) and used? Does this require labeled target samples beyond the few-shot labeled samples used for training? Is the validation set required for every experiment?

c. It is a bit surprising that such a simple method is able to outperform other methods by a significant margin. Could the authors confirm if the test set and the few-shot labeled samples are identical across all the compared methods?

d. L207 (“Complete details of all the experiments are included in the Appendix.”): Unfortunately, I could not find any relevant detail about the experiments in the Supplementary. Please update the supplementary to include details about the aforementioned points.


Minor comments
===============

a. Page 9: “Figure 4” -> “Table 4”

b. L53: “Summary” -> “summary”

c. Please maintain consistency in references (e.g. [1] is cited as NeurIPS while [4] cited as NIPS; [32] is published in ICML 2015, but cited as arXiv etc.).

d. A convergence plot could be useful in determining the stability of the training.


Considering both the strengths and weaknesses, I would suggest a revision at this stage. I would be happy to increase the score if my major concerns are addressed (see Weaknesses and Additional Clarifications).

--------

Post Rebuttal Comments: My major concerns were addressed in the authors' responses. Therefore, I am increasing the score.
limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method for semi-supervised domain adaptation. In their methodology, contrastive prototypical alignment is used for the source and target examples (pseudo labels are computed for unlabeled target examples) and contrastive loss is used for the unlabeled examples from the target domain. The hope is that by using the contrastive loss on the unlabeled examples, the target examples would form clusters and the classification boundaries would lie on the low density data regions. 
main_review: Strengths:
-------------
- The paper is very well written. Easy to read and follow. 
- Everything including datasets and implementation details are well described
- Experiments on multiple datasets and  ablation studies are also well conducted.
- They achieve improved performance in most experiments over the baselines. 

Weakness:
--------------
- The novelty is very limited. The method proposed in 3.2 Inter domain contrastive alignment is very close to other protortypical alignment methods in unsupervised and semi-supervised domain adaptation. 
- Its is not well understood how contrastive learning on unlabeled examples help. The authors say it clusters the target examples better and point to improved overall performance as evidence for it. We know that clusters formed by self-supervised learning methods do not necessarily correspond to classes of interest.  I do believe in this case, a theoretical justification and an empirical evaluation of the type of formed clusters is needed. 

limitations_and_societal_impact: Limitations are discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is about semi-supervised domain adaptation.  The inputs are labeled source samples, as well as labeled and unlabeled target domain samples.  The task is to train a model that performs well on the target domain.  The authors propose a framework with two main components to learn a domain-agnostic representation.

- Inter-domain contrastive alignment: This module reduces the discrepancy between classes centroids of the same class from source and target domain, while increasing the distance between class centroids of different classes from both source and target domains. Class centroids are computed as an exponential moving average via equations (2) and (3).  A contrastive loss is used to pull and push the centroids.
- Instance contrastive alignment: The goal of this module is to pull the classifier boundary into a low-density region, which is done by regularizing the classifier on the unlabeled target data.  Image augmentation is used to create perturbed images of unlabeled target data and a contrastive loss is employed to pull the representation of the same instance (perturbed and unperturbed) together while pushing it away from other samples.

Experiments on Office-Home, DomainNet and Office31 show promising results.
main_review: ### Clarity

- It seems from lines 141ff that class centers for unlabeled target samples are computed via pseudo labels, which are then pushed closer to the class centers of the labeled data. But in the paragraph around Equation 4, I did not find anything about the labeled target images?! Shouldn't those also contribute to the class center and pushed towards the source domain class centers?

- Line 251ff: How is the instance contrastive alignment actually different to FixMatch?

- In Section 3.3, how do you back-propagate through the exponential moving average of the class centers. Doesn't this create a computational graph that spans over all previous training iterations?


### Experiments

- In line 193, the experimental setup is defined as to use the same **number** of labeled samples in the target domain. But are the exact same samples used as in prior work for the labeled target set or just the same number? I would assume the final performance (strongly) depends on **which** samples are chose; not all subsets of size N from the target domain dataset are equal!

- In Section 4.3, did you also compare with a sort of upper bound where all source and target data is labeled?

- Line 217: Does "3-shot setting" mean that 3 samples are labeled? I'm only asking because this nomenclature is often used for few-shot learning but not for semi-supervised learning.


### References

- There is a related work that I think could be discussed as related (concurrent) work [A].

- I think references to some relevant work on class-aware domain adaptation are missing, like [B,C]. These works use a class-specific discriminator in an adversarial domain adaptation scheme, where class information for unlabeled target data is also obtained via pseudo labels.

- In line 133, I think there could be a reference to "using the output space for adaptation": [D], although this paper does use an adversarial approach for domain adaptation.

- I think the sentence in lines 162ff ("This idea ...") should get a reference.

#### References
- [A] Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning. Liu et al. arXiv'21
- [B] Ssf- dan: Separated semantic feature based domain adaptation network for semantic segmentation. Du et al. ICCV'19
- [C] Domain Adaptive Semantic Segmentation Using Weak Labels. Paul et al. ECCV'20
- [D] Learning to Adapt Structured Output Space for Semantic Segmentation, Tsai et al, CVPR'18



### Minor comments and typos

- Type in caption Figure 1: "... samples from both domains. (b) ..."
- Line 45: success ... success ?
- Line 53: "In summary, ..." & "... we propose ..."
- Line 90: "[24,26] use ..."
- Line 97: "... of limited labeled target ..."
- Line 118: The notation would be more consistent if the labeled source data is denoted as $\mathcal{D}_{ls}$
- Line 124: Remove the dot after $\mathcal{F}(\cdot)$
- Line 132: "[6,16] have ..."
- Line 159: "... is the strong augmentation function." More details are required at this point, e.g, a definition, a citation or a reference to the implementation details of the paper or something.
- Line 179: Use "eqref" to refer to equations.
- Line 183: "... Office-home [52] and Office31 [41]."
- Line 210: "... Maeta-MME [30] and ENT [14] ..."
- Line 240: "... accuracy drops from 72.4\% ..."
- Line 245: "... contrastive loss affects the ..."

### Post-rebuttal update

After having read all other reviews and the authors' responses, I have decided to keep my initial rating and recommend accepting the submission. I appreciate the authors' effort in providing all additional explanations and experiments.
limitations_and_societal_impact: - Limitations are okay, albeit being rather generic.
- Is there a possibility that the domain transfer/alignment method introduces additional biases, e.g., because the target data (although mostly unlabeled) is biased?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a contrastive learning method for semi-supervised domain adaptation (CLDA), where few labelled samples from target domain are available. The CLDA consists in inter-domain contrastive alignment, which decreases distance between same classes from source and target domains, and instance contrastive alignment based on strongly augmented unlabeled target images to reduce intra-domain discrepancy. The experiments are reported on three commonly used domain adaptation datasets. 


main_review: Strengths

1. The proposed method is simple and makes sense. It is interesting to see that the contrastive learning is effectively applied to semi-supervised domain adaptation problem. 

2. The ablation studies are thorough and performance on three benchmarks are good. 

Weaknesses

1. The inter-domain contrast alignment depends on the pseudo labels of the target samples. It is well known that some of the pseudo labels are inaccurate, which may be hurtful for the centroid estimation of target categories. Some recent studies [34,33, R1] have investigated different ways for mining high-confident pseudo labels and I am wondering how the proposed method will improve by considering this situation.  

2. The instance contrastive alignment only makes use of target samples. It will  also be interesting to consider the contrastive loss for the source samples, particularly as a regularizing term for the supervised, cross-entropy loss. 

3. Some recent works on semi-supervised domain adaptation are missing, e.g., [R1-R3]. It is necessary for the authors to cite these advances and compare with them. In particular, connection with and difference from R3 should be discussed and clarified.

 
[R1] Cross-Domain Adaptive Clustering for Semi-Supervised Domain Adaptation. CVPR, 2021.
[R2] Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation. CVPR, 2021.
[R3] Prototypical Cross-domain Self-supervised Learning for Few-shot
Unsupervised Domain Adaptation. CVPR, 2021.

----------------------------------------------------------------
After rebuttal
The authors' responses basically addressed my concerns and I have also read the comments of other reviewers. I keep my initial rating and recommend acceptance of this paper. 

limitations_and_societal_impact: The limitation of the proposed method regarding inaccuracy of pweduo labels is not favorably addressed. See Main Review for my suggestion.
review_title: null

Meta-review: 
 Overall, the reviewers found the proposed method simple but interesting, the paper well-written, and the experiments/ablations thorough. The reviewers raised a number of concerns including comparisons to related works (and concurrent work) and fairness of comparisons [aZTZ, bD47, MSYR], pseudo-labeling noise [aZTZ, bD47], as well as smaller clarifications and analysis. The authors provided a thorough rebuttal with significant additional experiments, including even comparisons to the concurrent CVPR works. The reviewers have been overall satisfied with the responses and some have decided to increase their scores. 

As a result, I recommend acceptance and strongly encourage the authors to incorporate the many experimental results and clarifications into the revised paper, including discussions on the experimental setup and noise resilience. 

Decision: Accept (Poster)