Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of unsupervised domain adaptation (UDA), where most state-of-the-art methods aim to learn a joint embedding space for samples from the (labeled) source domain and (unlabeled) samples from the target domain. 

In contrast to this, the authors propose to use contrastive learning pre-training techniques to first learn an encoder using target/source/additional data: Intuitively, this objective pushes positive samples together, and further away from other samples. The notion of "positive set" is defined as all samples which are obtained from data augmentations of a same sample natural image.
A classifier is then learned on top of these features to solve the classification task on the labeled source dataset.

Using standard contrastive learning techniques, the authors show that they can achieve similar accuracies as some SoTA baselines on three UDA datasets. The remaining of the paper analyses the features learned by this techniques, and in particular, how they do not learn to align the two domains as is the common intuition in UDA. This analysis relies on a "connectivity graph" defined for contrastive learning techniques, which, in the current setting, can be used to connect samples from the source and target domain, to determine how likely they are to be in the same positive set.
main_review: The main strength and contribution of the paper is the analysis of the source-domain connectivity, which was interesting to read. It was well illustrated through a toy example, and then applied to some real-life domain adaptation datasets. The whole analysis seems sound and well-though.
 However the main two weakness are **(i)** there is a lack of clarity on the data augmentation used for contrastive learning; these seem to be key to define the whole notion of connectivity so it is a bit surprising they're not discussed in more details; And **(ii)** it's not clear how significant the reported work is and what it could lead to; In particular, one of the claims of the paper is that contrastive pretraining can reach similar accuracies as other UDA techniques while not aiming to align the domains as adversarial UDA techniques do. However there is not much comparison to these adversarial techniques in the second part of the paper; For instance, Figure 5 suggest that, while domain embeddings can be easily separated with contrastive pretraining, there is still some kind of "domain connectivity" which is really important for final accuracy, which seems to actually match the "domain alignment" goal from adversarial UDA techniques.


**Some more precisions on parts of the paper I found a bit unclear:**
  * (i) **Unclear definition of the positive and negative sets for contrastive learning.** It is briefly mentioned in Section 3.2 that the positive sets in contrastive learned are defined from multi-crop data augmentations, however it's not really well detailed, and in particular it's not clear how this impacts final accuracy (e.g. how much does the features quality degrade with simpler data augmentations ? how much tuning does it require ?)
  * (ii) **Choices of datasets and baselines.** The experiments of Table 1 only seem to have one "fair" baseline (DirT/Sentry) because ERM is trained on the source data online. It would have been interesting to experiment on datasets such as Visda2017 or Office-31 that seem to be more commonly used in general and might have more benchmark results available. 
  * (iii) **Impact of data augmentation**: This links to points (i) and (ii): It's not clear to me how much the data augmentation strategy impacts the model, and whether it could also improve the baselines: Are Sentry and ERM baselines also trained with the same data augmentation as the contrastive learning techniques ?
* (iv) **Readability of table 2** Table 2 and Section 4.1 would be stronger with more details on the classifiers used. For instance, if I understood correctly, the authors train a classifier to separate the source and target domain on either (i) the input images or (ii) the pretrained contrastive features. Because classifier (ii) as higher accuracy than (i), they conclude that the contrastive features 

**Small typos/unclear notations:**
  * **Table 1, middle, second row**: the highest accuracy (Sentry) is not bold
  * **Definition 1** Shouldn't the definition of connectivity include some kind of expectation over drawn samples $x$ and $x'$ ?
summary_of_the_review: The main strength and contribution of the paper is the analysis of the source-domain connectivity, which was interesting to read. It was well illustrated through a toy example, and then applied to some real-life domain adaptation datasets. The whole analysis seems sound and well-though.
 However the main two weakness are **(i)** there is a lack of clarity on the data augmentation used for contrastive learning; these seem to be key to define the whole notion of connectivity so it is a bit surprising they're not discussed in more details; And **(ii)** it's not clear how significant the reported work is and what it could lead to; In particular, one of the claims of the paper is that contrastive pretraining can reach similar accuracies as other UDA techniques while not aiming to align the domains as adversarial UDA techniques do. However there is not much comparison to these adversarial techniques in the second part of the paper; For instance, Figure 5 suggest that, while domain embeddings can be easily separated with contrastive pretraining, there is still some kind of "domain connectivity" which is really important for final accuracy, which seems to actually match the "domain alignment" goal from adversarial UDA techniques.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates why contrastive learning method benefits unsupervied domain adaptation.  In particular, the authors use SwAV, a recently proposed self-supervised learning method to pretrain a model on both targeted and source datasets. The authors found that by using such a model for UDA tasks surprisingly nice results are achieved. The authors analyze the reasons behind using a connectivity model.
main_review: - The paper is mainly about a finding that pretraining with SwAV ( a recently proposed self-supervised method) can benefit unsupervised domain adaptation tasks. This is done by pretraining on both source and target models.
- Overall, the findings are interesting. But it would be more convincing if the authors report the results of initializing the backbone used in standard UDA tasks with pretrained self-supervised model (SwAV) in this case. So this would make comparisons with other approaches more fair. 
-  It would be more convincing if the authors experiment with other self-supervised models like MOCO, SIMCLR. I am curious whether the same conclusion would hold.
- The presentation could be further improved---there are lots of typos...
summary_of_the_review: Overall, I think the observations in this paper are interesting. But I think more experiments would make the paper stronger.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to study the contrastive pre-training on domain adaptation tasks. The key contribution is to design a connectivity model for better data selection and data augmentations of pre-training. To verify the benefit of contrastive pre-training to domain adaptation, experiments are performed on a variety of benchmark tasks including BREEDS, DomainNet and CIFAR-10.
main_review: Strong points of the paper: 
1.	The goal of applying contrastive pre-training on unsupervised domain adaptation is a critical topic to study. 
2.	The paper proposes to use the connectivity model to study contrastive learning. 
3.	Experiments are comprehensive with a variety of benchmark tasks.

Weak points of the paper: 
1.	This paper does not propose a new method, but proposes a connectivity measure based on the analysis of existing methods. Despite it maybe practical to improve fine-tuning with connectivity measure, the technical contribution is rather limited.
2.	The main experiments in this article are based on existing methods. It would be good to show the results of using domain adaptation methods based on the new connectivity measure. Even a simple solution on the toy task can verify the practicality of the proposed measure to a certain extent.

summary_of_the_review: Although this paper gives an interesting new idea to solve the problem of domain adaptation from connectivity, it does not extend the idea to a specific method, nor does it conduct sufficient experimental verification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies how contrastive learning (CL) behaves in the unsupervised domain adaptation (UDA) setting. In particular, it finds that CL brings a different mechanism for UDA compared to traditional adversarial DA methods, where the features learned are far apart between the source and target domains. It further develops a conceptual model for explaining the success of contrastive learning under domain shifts, and empirically demonstrates through experiments on several benchmark datasets.
main_review: ## Strengths
+ The topic of the study is of great interest, and to me is pretty novel and not explored in the literature.
+ The observations are intriguing: contrastive learning based methods connect disparate domains in a way that is different from traditional adversarial DA perspective. The findings could have implications on when / how contrastive learning works when data are from different domains.
+ The evaluation and presentation is clear and thorough. The empirical results are well aligned with the proposed connectivity model, which could be promising for understanding the behavior of CL.
+ The writing is clear and easy to follow; the paper is well structured.

## Main Weaknesses
1. __Ambiguous shift assumptions throughout paper__ 
- Throughout the paper, the authors never explicitly define what is the __shift assumption__ across domains. Is the study conducted under the covariate shift assumption (i.e., $p_i(x) \neq p_j(x)$ and $p_i(y|x) = p_j(y|x)$)? If so, what will be the results under other assumptions (e.g., label shift)? Will this affect the main conclusion?

2. __Investigations limited to visual recognition tasks__
- First, the title and body texts seem to hold without specifying the data modality. However, all experiments are limited to image data for visual recognition tasks. Nevertheless, CL has been successful beyond image data, for example, texts [1], audios [2] and time series [3]. The authors only experimented on image data, which might not be enough for a rigorous conclusion.
- The limited experiments only on image data also lead to doubts on whether some claims are true or not. For example, the paper claims "Contrastive pre-training is a strong domain adaptation method". This might be true for image data, as it is interpretable by human, and people can design effective augmentation methods (e.g., those in SimCLR). However, does this hold also for other data modalities, especially those without rich priors on how to do augmentation in the first place?

3. __Minor: only single source & single target domain considered__
- One minor drawback is that the paper considered only single source & single target domain. For self-supervised pre-training, uncurated data is more likely to come from different domains [4]. The analysis under such case however is missing. It would be helpful to discuss the differences / similarities when there are multiple source domains exist.

4. __Minor: no actual algorithms / application scenarios provided__
- I understand that as a starting point, the paper only analyzed the behavior for CL across disparate domains. It would be nice to also give practical algorithms on how should one select (sub)sets of unlabeled data for better transfer. The analysis in the paper is good; however, I find it a bit disconnected with practical application scenarios.

## Other issues / questions
- Since the overall objective is agnostic to the data modality, is the proposed scheme also be extended to other data domains beyond images, like time series / texts? What would be the most challenging parts when extending to other modalities?
- How much is the performance dependent on the quality/effectiveness of the augmentation set $\mathcal{A}$?

## References
[1] SimCSE: Simple Contrastive Learning of Sentence Embeddings.

[2] Representation Learning with Contrastive Predictive Coding.

[3] Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding.

[4] Divide and Contrast: Self-supervised Learning from Uncurated Data.
summary_of_the_review: Overall, the idea is interesting and intriguing, however there are still limitations and ambiguities in the current draft. The detailed comments/questions are listed in the weaknesses / questions part.

The paper has a great potential to the CL field, and could be beneficial and inspiring for broader audience; but issues need to be addressed / made clear. I'm happy to change my score if the feedback addresses my concerns. Please refer to the points in the weaknesses / questions part. I would like to see feedbacks on these comments/questions.
review_title: null

Meta-review: This is a borderline paper. While reviewers believe the findings from this paper may be of potential interest, they are fully convinced. For instance, if the authors want to claim the proposed mechanism is general for UDA, then they should demonstrate its effectiveness to other application domain(s), such as the NLP domain, where the pretrain-finetuning strategy is widely adopted for transfer learning. However, the authors did not provide correspondingly additional experiments as requested by a reviewer but claimed they only focused on the CV domain. If the focus is on the CV domain, then the authors need to explain in detail why in the CV domain, the proposed mechanism works well (while in other domains, it may not). There are many other concerns about the assumptions, experimental settings, etc. 

In summary, this is a borderline paper below the acceptance bar of ICLR.
Decision: Reject