Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes PACMAC, a method for unsupervised domain adaptation tailored to vision transformer architectures (ViT) pre-trained using self-supervised approaches (SSL). 
The method works by 1) continued SSL training on the union of the labeled source domain and unlabelled target domain, and 2) joint fine-tuning on the source domain and self-training on the target domain.
For the self-training on the target domain, the paper proposes a mechanism based on the consistency of the model's prediction on differently masked inputs to select examples for self-training. 
These input masks are generated based on the ViT self-attention scores and a greedy selection mechanism.
Experiments demonstrate improvements over prior work in various benchmarks and include ablation experiments to demonstrate the importance of continued SSL pre-training and the example selection strategy for self-training. 
strengths_and_weaknesses: Strengths:
- Domain adaptation is of practical importance, and investigating approaches for modern architectures and pre-training strategies (ViT + SSL) is useful
- Using consistency of predictions between differently masked inputs is an interesting novel idea
- Extensive experimental evaluation
- Good performance in various domain adaptation benchmarks
- A good set of ablation experiments is provided


Weaknesses:
- I found the presentation quite confusing. The main reason for this is that it is unclear how self-training (which typically leverages class predictions) relates to a self-supervised model (the focus of this paper). Only at L193 is it mentioned that a classifier is learned on the source data after SSL pre-training, and I assume that those predictions are then used for the self-training (+mask consistency, etc.). This makes the paper up to that point rather confusing. I also assume this fine-tuning is performed after the continued SSL on source+target data, but it is unclear based on the paper. 
- It appears that the novelty of the method is limited mainly to the mask-consistency strategy to select examples for self-training. For example, the SSL pre-training on pooled source+target data was proposed in [28]. Furthermore, [9] offers a selection mechanism based on consistency across augmentations, which is conceptually quite similar to the mask consistency proposed here. Therefore, it would be important to see how a combination of [28] and [9] would do, especially since [9] by itself appears to perform well already. 
- It is unclear whether the selection mechanism through mask consistency benefits the method or the additional augmentations through the use of masking during self-training. Indeed, only using the selection appears to lead to subpar performance (see L268). It might be good to investigate this observation further and disentangle these two factors. 

POST REBUTTAL UPDATE:
The authors addressed some of my concerns in terms of presentation and the benefits of the selection mechanism (compared to additional augmentations). I still think the novelty over [28]+[9] is somewhat limited, but improvements appear to be solid. I'm thus raising my rating. 
questions: I would appreciate it if the authors could address the weakness listed above, i.e., 
- How they aim to improve the presentation and where to introduce the classifier training.
- Clarify the novel contributions and how the method compares to a combination of [28]+[9], which appears conceptually very similar.
- Clarify whether the benefit of the masking is more due to example selection or additional data augmentation. 
limitations: Limitations were addressed well in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on adapting self-supervised ViTs (MAE[26] and DINO[25]) for unsupervised domain adaptation (UDA). The goal is the find reliable pseudo labels for unlabeled target domain images and then train the model using pooled source and target data. The paper uses a setup prevalent in the existing literature to find the reliable image-pseudo-label pairs – filter the target domain images with consistent predictions across multiple augmented versions and consider the predicted labels for such images as reliable pseudo labels. The main contribution of this paper lies in designing ViT-specific data augmentation, namely attention-guided masking. The presented method is dubbed ‘PACMAC’, short for Probing Attention-conditioned Masking Consistency for UDA.
strengths_and_weaknesses: Strengths:

* The paper attempts to find ViT-specific data augmentation for finding reliable image-pseudo-label pairs using FixMatch[R1]-style setup.
* The paper is well-presented with a good set of experiments.

Weakness:

* The learning setup is very similar to the previous method called SENTRY [45]. As described in L239-244, the only difference between the two methods is the data-augmentation scheme and the loss function. Further, Sec 1 of supplementary material establishes the superiority of the loss function used in SENTRY over the one presented in the paper.
* Augmentation being the main contribution, the paper lacks a comparison between attention-guided masking augmentation and other types of augmentations (such as random or attention-guided cropping).
* It is mentioned in L240 that PACMAC intends to design a data augmentation scheme that matches the design of the SSL pretraining. However, the presented method only matches the design of MAE, not DINO. One can observe that the improvement offered by PACMAC over the previous state-of-the-art is lower for DINO than MAE.

References:

[R1] Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." NeurIPS, 2020.

questions: * Looking at the experiment presented in Sec 1 of supplementary material, the improvements offered by PACMAC over SENTRY can be attributed to data augmentation and not the loss function. Please explain which other features of PACMAC are superior to SENTRY.
* Please compare the attention-guided masking augmentation and other types of augmentations such as attention-guided cropping (e.g. one that matches the multi-crop augmentation of DINO).

limitations: Limitations discussed. There are no ethical concerns.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a new method for adapting Visual Transformers (ViT) with modern self-supervised learning (SSL) pretraining adapted to unsupervised domain adaptation (UDA). The proposed approach, named PACMAC, starts with a weights pretrained on Imagenet and follows of two steps: (i) in-domain SSL on both source and target datasets and (ii) self-training model on target based on a "reliance measure" (based on predictive consistency across many masked inputs). The work shows good results on multiple standard UDA evaluation datasets.


=======================
### Post-rebuttal updates
I thank the authors for their feedback. After looking at the rebuttal and other reviewers' comment, I decided to keep my “Weak Accept” (6) rating.  
I continue thinking that the paper achieves good empirical results (and the authors provide a good range of experiments), but the novelty is fairly limited—as agreed by all the reviewers. 
strengths_and_weaknesses: ### Pros
* The paper is clear, easy to follow in most parts and provides a good literature review.
* The authors show good results compared to current models and provide good ablation studies.
* The idea of using "attention-conditioned" masking makes sense and "consistency" on self-training are interesting (however, they are not related to UDA and would be better studied in a general context rather than in the particular case of unsupervised domain adaptation).
* The problem of domain shift, OOD and domain adaptation in neural networks is important to the community.

### Cons
* The main weakness of this paper IMO is the lack of novelty. As pointed out in the manuscript, most of the components of the proposed method have already been explored in previous literature (SSL for UDA, self-training for UDA). The main contribution is mostly adapting those approaches to a new architecture (ViT).
* The main contribution then becomes the application of ViT-specific tricks on either the SSL or self-training components (eg, the attention-conditioning mask). This trick, however, could as well be used in any SSL/self-training method that relies on ViT, not only on UDA problems.
questions: * I found Table 6 very difficult to understand/parse. Could the authors elaborate a bit more what is it about? And more importantly, I would recommend the authors to update the manuscript and make it more clear as well.
* How does capacity of the model and inference time change between PACMAC and other SOTA methods (eg, SENTRY, Shen et al.)?

limitations: The authors were clear with the limitation of the method (slower and not same results as supervised initialization).
review_title: null

Meta-review: metareview: This work looks at adapting ViT-like models for unsupervised domain adaptation, by cleverly finding pseudo-labels with 'attention-guided masking'. There's a weak consensus among the reviewers that this work has good empirical results, but somewhat limited novelty. I think the rebuttal discussion has helped improve this work quite a bit, and given the good results, ablations, and the importance of unsupervised domain adaptation, I am recommending acceptance.
confidence: Certain
Decision: Accept