Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency
Authors: Viraj Uday Prabhu
Sriram Yenamandra
Aaditya Singh
Judy Hoffman
Venue: NeurIPS 2022 Accept (2022)
Keywords: domain adaptation
self-supervised learning
vision transformer
object recognition

Abstract: Visual domain adaptation (DA) seeks to transfer trained models to unseen, unlabeled domains across distribution shift, but approaches typically focus on adapting convolutional neural network architectures initialized with supervised ImageNet representations. In this work, we shift focus to adapting modern architectures for object recognition -- the increasingly popular Vision Transformer (ViT) -- initialized with modern pretraining based on self-supervised learning (SSL). Inspired by the design of recent SSL approaches based on learning from partial image inputs generated via masking or cropping -- either by learning to predict the missing pixels, or learning representational invariances to such augmentations -- we propose PACMAC, a two-stage adaptation algorithm for self-supervised ViTs. PACMAC first performs in-domain SSL on pooled source and target data to learn task-discriminative features, and then probes the model's predictive consistency across a set of partial target inputs generated via a novel attention-conditioned masking strategy, to identify reliable candidates for self-training. Our simple approach leads to consistent performance gains over competing methods that use ViTs and self-supervised initializations on standard object recognition benchmarks. Our code is available at https://github.com/virajprabhu/PACMAC.

Adapting Self-Supervised Vision Transformers by Probing Attention-Condttioned Masking Consistency
Viraj Prabhu"
Sriram Yenamandra' {virajp, aerrajpssiiree sriramy, asingh, b judyj @gatech. edu Georgia Institute Technology
Aaditya Singh
Judy Hoffman
Abstract
Visual domain adaptation (DA) seeks to transfer trained models to unseen, unla- beled domains across distribution shift, but approaches typically focus on adapting convolutional neural network architectures initialized with supervised ImageNet representations. In this work, we shift focus to adapting modern architectures for object recognition- - the increasingly popular Vision Transformer (ViT) - initialized with modern pretraining based on self-supervised learning (SSL). Inspired by the design of recent SSL approaches based on learning from partial image inputs gen- erated via masking or cropping either by learning predict the missing pixels, learning representational invariances to such augmentations we propose P PACMAC, two-stage adaptation algorithm for self-supervised ViTs. PACMAC first performs in-domain SSL on pooled source and target data to learn fask-discriminative fea- tures, and then probes the model's predictive consistency across set of partial target inputs generated via a novel attention-condee conditioned masking strategy. to iden- tify reliable candidates for self-training. Our simple approach leads consistent performance gains over competing methods that use ViTs and self-supervised initializations standard object recognition benchmarks. Our code available
Introduction
which they were g 9 seeks to adapt models trained on labeled source domains unseen and unlabeled target domains, but most existing approaches focus on adapting convolutional neural network (CNN) architectures initialized with supervised representations, typically from ImageNet [TO]. In this work, we seek "modernize" domain adaptation by focusing on adapting state-of- architectures for object recognition Vision Transformers (ViTs) initialized with modern pretraining strategies based on self supervisdd learning (SSL).
Deep models struggle generalize visual domains that deviate from the trained (1] Unsupervised domain adaptation 2
3 5
6
Vision Transformer (ViT) architectures have recently gained traction as an effective alternative CNNs for computer vision tasks [12], achieving impressive performance despite fewer inductive biases. With their in-built self-attention mechanisms and improved calibration under distribution shift over their CNN counterparts
13 ViT' may be particularly well-suited to domain adaptation [14] 15
Similarly, self-supervised representation learning (SSL) is rapidly replacing supervised learning as the de-facto pretraining strategy for deep networks, due to improved scalability (unlabeled data is easier to collect) and generality (domain-specific SSL is often preferable to one-fits-all ImageNet pretraining [6 (7). Many successful methods that optimize proxy objectives based on
pretext tasks [8] 20 instance discrimination [2] 22 23 24), self-distillation [25], and image
reconstruction 26 have been proposed.
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite the increasing popularity of ViTs and SSL, to our knowledge no prior work has focused on adapting self-superrised ViTs. Recent work has proposed additional self-supervised contrastive learning on the pooled source and target domain as strong initialization for DA methods [27] 28). but focuses on adapting CNNs. We follow these works to perform additional in-domain pretraining. finding that it leads to learning daskiiiscrmminative features with ViT's as well. Interestingly, Shen et al. (28) find that such contrastive pretraining learns features that disentangle domain and class information and can generalize to the target without being domain invariant, subverting traditional wisdom in UDA. We experimentally find that this causes several off-the-shelf DA methods (e.g. based on domain adversarial learning) sometimes fail at adapting ViTs initialized with SSL representations, suggesting the need for specialized solutions. Our work attempts to fill this gap.
Concurrent work has also studied the problem of adapting ViTs [4 [5], but focuses on supervised rather than self-supervised initializations. Crucially, none of these prior works propose an adaptation strategy specifically designed for adapting self-supervised initializations, whereas we explicitly incorporate components of the SSL pretraining in our adaptation algorithm for better transfer.
Concretely, we observe that several recent state-of-the-ard SSL methods for ViTs focus on learning
from partial inputs [2] 23 26
generated via masking or cropping strategies. In the context
of vision transformers, such methods learn to either reconstruct the missing information 226 29 or to be invariant to such cropping [25]. We find that the predictive consistency of a model initialized with such representations across partial inputs generated via masking acts as an effective self supervisdd reliability measure on the unlabeled target domain. Further, rather than masking inputs randomly, we leverage the ViT attention mechanism to perform attention-conditione set of disjoint masks that only keep highly attended patches from the input image. We then probe the reliability of given target instance by measuring the model's predictive consistency across such masked images and the original image, and mark target instances with high consistency as reliable. We call our selection strategy Probing IicentionConddtionnll Masking Consistency (PACMAC).
masking: we generate
We further mark instances with high confidence as reliable, and then selectively self-train the model against the current prediction on target instances identified as reliable via either scheme. This leads to an easy to implement selective self-training adaptation algorithm that outperforms competing methods on standard benchmarks. We make the following contributions:
We propose an algorithm to adapt self-supervised ViT's to unseen domains that i) performs in-domain SSL on the pooled source and target data, and ii) self-trains the model on target instances identified to be reliable by probing the model's predictive consistency across a set of cecettion-coniitione masks applied to the target image, in combination with confidence.
Our approach outperforms more complex DA methods from prior work at adapting self- supervised ViTs on the OfficeHome 30, DomainNet 31 and VisDA 32 adaptation benchmarks for object recognition.
2 Related work
Unsupervised domain adaptation (UDA). UDA seeks to transfer a model trained on a labeled source domain to an unlabeled target domain across distribution shift. A variety of successful approaches based on aligning feature spaces via optimizing domain divergence measures 4 3] and distribution matching via domain adversarial learning 6 7 (often incorporating additional pixel-level matching constraints), have been proposed. Recently, selective self-training [33] 9 on the model's predictions on target data has emerged as a simple and effective UDA technique.
With the upsurge in adoption of Vision Transformer (ViT (TI) architectures for object recognition, some recent works specifically focus adapting ViTs 15). These methods leverage the ViT attention mechanism for incorporating patch-level transfcrability [4] and category-level feature alignment [155. However, these works focus adapting models initialized with supervised ImageNet weights. In this work, we propose an adaptation algorithm designed for self-supervised ViTs.
Self-supervised learning. Self-supervised learning (SSL) leverages unlabeled data to learn rep- resentations that can be transferred efficiently to downstream tasks. Several approaches based on pretext tasks [8] 19 20 masked-image modeling 26 34 context prediction (35] and instance
discrimination 22 23 have been proposed to learn strong semantic representations the absence of
labeled data. Some of the state-of-the-an SSL methods are designed for ViTs: DINO (55 performs

---

self-distillation by training a student network that is given strongly augmented image as input to match the oulput of teacher network that sees a global view of the image. MAE [26] is a masked
image modeling (MIM) approach that learns to predict complete image given only a fraction of the
image patches as input. Recently, ADIOS [66] improves MIM methods by learning an adversarial masking function to obscure salient regions of the image. A concurrent work, MSN (29 improves upon DINO by passing in a masked version of an augmented image to the student network. We derive inspiration from the local-to-global self- self-supervisory signal used by these methods in designing our adaptation algorithm, but instead of learning reconstruction or invariance to missing information, measure the model's predictive confidence under targeted missing information self-supervised probe to determine reliability.
Self-supervised learning for domain adaptation. While most UDA methods focus on adapting models initialized with supervised initializations, typically from ImageNet, a few recent works have
studied self- seff-supervised domain adaptation [29] 27]- Kim er al. (27 propose CDS, pretraining
strategy for domain adaptation that makes use of in-domain contrastive learning in conjunction with
cross-domain matching as superior initialization alternative to ImageNet pretraining. Shen et al. [28] study the transferability of representations learned via additional in-domain contrastive learning on the source and target domain, finding that such pretrained features can be transferred effectively to the target by simply finetuning on the source, despite not being domain invariant. However, these methods restrict their experiments to ResNet [37] architectures while work focus adapting ViTs. Crucially, neither of these works propose a DA algorithm catered to the model's ' selff-uperrviedd initialization, whereas do so explicitly.
3 Approach
3.1 Notation
Let X' and y denote input and output spaces. In unsupervised domain adaptation (UDA) given access to labeled source instances (xs, Uss) ~ Ps((')),, and unlabeled target instances Xx PPX)), where S and T correspond to source and target domains. The goal is to learn a model f= )()())) (ó denotes the encoder and h denotes classifier), parameterized by θ with minimum Vision Transformer or
error the target dataset. In our experiments, θ is parameterized ViT [T],, which takes as input a linear embedding of of N image patches in addition to a class token embedding. These N+1 embeddings are then appended with positional encodings and passed through several transformer layers, each of which comprises of a sequence of multi-headed self-attention (with M attention heads), multilayer perceptron, and layernorm modules. Features from the final encoder layer are typically fed to a classifier layer and used to predict the output. We consider adapting models trained for C-way image classification: the inputs x are images, and labels y are categorical variables E {1, 2, ... Cj. Let pe (ylx) denote the final probabilistic output from the model. For a target instance xT (X))), we compute a "pseudolabel" as 9 = argmax pe (.)
3.2 Preliminaries
Self-supervised learning (SSL) for ViTs. We consider SSL methods for ViTs that learn from partial inputs, under two popular formulations: i) Masked Image Modeling, wherein the model is given a partial image and trained to predict the missing content (at pixel or token level) via learning masked
or denoising autoencoder (26) 38 39 34). ii) Joint-embedding networks, which learn a model that produces similar features for different views of a given image, across strong cropping and additional augmentation 25 29]. Let m(xT) denote the partial image generated from target image X under transformation m-() which could correspond to either masking or augmentation strategy. Broadly speaking, SSL corresponds to minimizing an objective of the form LSSL
)), o(x))), which could correspond to a reconstruction invariance based objective operating on encoded features. We now instantiate Lsst. in the context of the the two SSL strategies we use in this paper: MAE 37
and DINO 25
MAE 37
MAE learns a visual transformer autoencoder to reconstruct images Xy given only random subset of patches from the original image m(x)). Let d denote a decoder learned for image reconstruction. The MAE objective minimizes the mean square error between the original and reconstructed image in pixel space:
LSSL(xT) = ||xT d(((m((x)))
(I)
3

---

Atttettion-coodittoonedd Masking
Consistency-based Reliability
"marker"
Mask flow
Target image (flipflops
Mask flow
encoder classifier
Conf. threshold
Greedy alternating
Reliability checker
"pencil"
assignment
"flowers"
LSST
Figure 1: Overview of PACMAC. Left. First, the model's attention over given target image is used to generate set of disjoint masks that only keep highly attended patches of the input image via greedy assignment strategy. Right. Next, the model's predictive consistency across the original and masked images is used supeevupe reliability measure to select target instances for self-training.
For target image Xy. DINO passes two transformed versions of each target image mi (XT) and m²(xx) to a student network Ø, and teacher network Φi respectively, and trains the student network to predict the output of the teacher by minimizing the cross-entropy H between the two output distributions:
DINO 25
(sSL(XT) (m) (x 6s (mz(x))))
Self-training for unsupervised DA. Self-training for UDA typically involves training the model on its predictions on unlabeled target data ("pseudolabels"), optimizing either a conditional entropy minimization [40] or cross-entrope objective. However, under a domain shift many of the model's (especially under severe distribution shift), and unconstrained self-training may lead to amplifying model errors.
predictions may initially be incorrect [4] 42 g
To combat this, few recent works have proposed selective self-training predictions that have high likelihood being correct. Recent works propose to identify such reliable predictions either
based model confidence 333 43 predictive consistency under data augmentation 9 44], or
combinations of the two [45]. Let r(x7) denote a binary reliability value for target instance æy. We
optimize the following selective self-training objective on target data Lssr:
LSSt ((xTiiT)~PP (o(e)
In this work, we propose an alternative selection criterion designed for self-supervised representations based on predictive consistency under attention-conditioned masking, which now introduce.
3.3 PACMAC: Probing Attention-conditionee Masking Consistency for UDA
Following prior work 228 27 we first learn task-discriminative features by optimizing following
in-domain self-supervised pretranning objective LIDP pooled source and target data:
LIDP = -xx==xxxxPPP (3s= o(xs)) + Exy (/s
6(xx))
(4)
We then finetune the learned representations end-to-end labeled source data:
LoE (1xxs.u))~Py(CCstt us)]
Finally, we perform selective self-training. For target instance X PT, we generate committee of masked versions. However, applying random masks may lead to capturing irrelevant backgroun features: instead, propose aatention-conditionne masking (see Algo. I. We first obtain the model's

---

Algorithm Attention-conditionne Masking
1: Input: 27: per-patch attention êy, masking ratio mr, committee size k
2: L (1-mr)x len(āT)
3: Sy t argsort((T)
D Sort patch- wise attention in ascending order b Initialize masks, 0 is hide, is keep
4: for irri + to 1: do M, < veros_like((
5: i+
6: while i < L do
7:
itit1
8:
for j + 1 to do
9:
p Soppop last() My -
> Pop next-most attended patch index Greedy round-robin assignment
for < to k do m (IT) Mj ohT
12: Return (mT)) mx(IT))
Algorithm 2 PACMAC Optimization
1: Input: Socee,,er, TgtL loaer,, model parameters θ masking ratio mr, committee size k; confi- dence threshold loss weight a
2: for epoch +- 1 to MAX_ MAA_CE do
3: for æs ys in SreLoader and zT in TgtLoader do
4:
Pr. ār < at((pe(y(=T)) {m1 ..... (xT))) < AttentoocCCndttioooclllssngggggs ây, mr, C (m,:(T)) argmax (P()))
max pe (y(lr)
>> Get top-1 confidence and prediction original image
5:
> Get per-perteeee per-patch attention
6:
7:
8:
LSST =0
9:
if Ie k OR pT T then
Consistent Confident
10: LSST f LC pe y (x
11:
Minimize Leg (xs, . ys) aLssr
per-patch attention score är over the original image (in practice, we obtain the patchwise self- attention from the last encoder layer of the class token with respect all image patches for each attention head, and average across the M heads). We then initialize all k masks with zeros (hide all patches). Next, we sort patches in descending order of attention scores and perform a greedy round-robin assignment to the k masks to only unhide highly attended patches, until the desired masking ratio satisfied. We then apply these masks to the original image to generate masked images (xy), m,(xx)...,,((xx)) (x,)))
We make predictions for each of these k masked versions (following prior work [37] only feed in unmasked patches), and measure consistency between the model's prediction for the original image and for each of the k masked versions. If the model's prediction for all masked versions matches its prediction on the original image, we consider the instance as "reliable", and as "unreliable" if not. Further, also consider instances reliable if the model's predictive confidence the original image is higher than a threshold T'. Let [-] denote an indicator function. Formally, we assign target instance reliability r(xT) as:
confident
r(xy) = Jargmax (e(ylm, = argmax /V( 1..k or max (o(y T
(6)
Algorithm lists the steps involved in computing the PACMAC objective. Without loss of generality, we minimize the cross cross-entropy between the model's prediction on the last consistent masked image my (xy) and its consistent pseudolabel. We empirically find this to act as an effective data augmenta- tion strategy and provide stronger learning signal than when using the model's prediction on the original unmasked image (see Sec. 4.5). We additionally optimize a cross-entropy loss over labeled source examples. For loss weight a, the full Lpacmac objective is:
LPaCuA (Exxs.ss)> (Ceee))/CCCC(((eeeeeee us)] + Halci//smmooo////s////////////a/
(7)
5

---

Table 1: Target test set accuracy OfficeHome across MAE original methad, Sec.
26
and DINO 25
pretraining.
Variation of
INIK Init. Method
A
A
R
R
P
R
R
R
P
AVG
source
46.4
$7.6
71.0
$1.1
00.0
62.6
51.4
46.9
70.5
66.3
52.2
77.2
59.4
CDAN
45.3
58.8
69.1
51.6
60.7
61.5
53.4
45.5
72.4
67.7
49.9
78.0
59.5
51.1
40.3
Sheu at
28
57.1
63.6
71.9
$7.9
63.6
67.1
53.5
56.7
71.2
69.0
62.6
79.4
64.8
SENTRY
54,8
74.4
69.8
57.6
54.9
75.5
68
60.0
65.5
PACKAC (Ours)
58.9
68.2
74.1
67.1
67.2
57.3
59.2
74.4
68.6
63.9
82.7
66.8
source
53.1
65.0
75.2
62.0
66.2
70.4
60.8
50.5
77.0
72.8
53.9
81.2
65.7
CDAN
71.2
64.7
47.0
79.9
66.5
DINO 25
MCC
44.4
74,2
79.6
61.9
67.6
72.4
63.0
40.1
79.2
73.3
47.1
82.8
65.5
TVT
57.1
754
71.0
63.3
Shen et
28
53.1
69.4
76.7
62.6
68.9
71.4
62.2
51.8
76.9
73.5
56.3
81.8
67.0
SENTRY
59.5
72.0
76.8
66.1
71.1
73.4
63.7
56.2
77.8
72.4
63.0
81.9
69.5
(Ours)
4 Experiments
We first describe our datasets and metrics (Sec. 4.1). implementation details (Sec. 4.2) and baselines
(Sec. 4.3 Next, we present results (Sec. 4.4) method ablations (Sec. 4.5 and analysis (Sec. 4.6)
4.1 Datasets and metrics
We evaluate PACMAC on three classification benchmarks for domain adaptation: i) OfficeHome (30 is a classification-basse
benchmark comprising of 12 shifts spanning 65 categories of objects found in home and office environments. It consists of 4 domains: Real-world (Rw), Clipart (CI), Product (Pr), and Art (Ar). ii) DomainNet [3 is a large benchmark for adapting object recognition models. Matching prior work 28 g. experiments, which reports performance over 12 shifts comprising 40 common classes from domains: Real (R), Clipart (C), Painting (P), and Sketch (S). fii) VisDA2017 [32]is large image
we use the subset of DomainNet proposed Tan al. [33 for our
classification benchmark for synthetic- adaptation with >20k images from 12 classes.
Metric. Matching prior work report standard accuracy the target test set as metric.
4.2 Implementation details
We use ViT-base ITI architecture with 16x16 image patches. We official codebases for
MAE (26] and DINO 25 and initialize with checkpoints pretrained on ImageNet1 [10]. We
pretrain the combined source and target domain for 800 epochs (MAE) and 200 epochs (DINO).
For pretraining, we linearly scale the learning rate to 4x 10 (MAE) and 5 50 5 (DINO) during a 40 epoch warmup phase followed by cosine decay. We use the AdamW [46] optimizer. For PACMAC, use k = 2, mr 0.75, T = 50%, and a = 0.1. We use RandAugment [47 with N = 3 and M 4.0 during pretraining and N 1 and M 2.0 during DA. On OfficeHome and DomainNet,
finetune on the source and adapt for 100 epochs each, and perform 10 epochs of each phase on VisDA. We use learning rate of 2 10 and weight decay of 0.05. All experiments use PyTorch (88..
4.3 Baselines and SSL strategies
Lacking baselines designed for adapting self-supervised VITs, compare against diverse DA methods proposed for CNNs with supervised initializations: based on domain adversarial learning (CDAN 77). minimizing classifier confusion (MCC [49], self-training (SENTRY 45)). and contrastive learning (Shen et al. (28)). We also benchmark concurrent method for adapting ViTs (TVT 14) with supervised initializations. 1) CDAN 77: CDAN improves upon domain adversarial learning with multilinear conditioning, by capturing cross-covarianee between feature representations and classifier predictions improve discriminability. Recent work (50) finds CDAN outperform more recent DA methods when combined with new architectures like VITs. 2) MCC [4]] Minimum
DA baselines:
Classifier Confusion a non-adversarial DA method that aligns domains by minimizing pairwise class confusion the target domain estimated from model predictions. 3) SENTRY p SENTRY measures model predictive consistency across randomly augmented versions of each target image and selectively minimizes entropy to increase predictive confidence highly consistent instances,

---

Table 3: Target test set accuracy DomainNet actoss MAB original methad, see Sec.
26 and DINO 25 pretraining (* Variation of
INIK Init. Method
C
R
P
R
R
P
P
S
C
P
AVG
Souree
71.0
77.6
62.9
73.7
61.5
63.3
82.4
63.1
66.1
76.6
71.9
69.6
70.1
CDAN
72.2
74.5
59.3
80.6
57.3
59.2
78.5
57.4
61.2
81.4
73.2
68.7
Shen et ml.
83.0
80.2
77.7
84.2
74.6
74.7
85.0
77.5
76.5
83.7
80.7
76.2
79.5
SENTRY
9
84.2
82.8
76.4
86.9
77.1
74.1
86.9
76.2
73.3
88.8
81.6
77.6
80.5
PACHAC (Ours)
86.0
81.9
78.8
86.0
74.8
76.3
87.4
84.0
77.5
85.2
83.1
78.3
81.6
75.7
82.8
68.2
81.9
73.9
71.1
82.6
70.6
69.5
80.8
76.9
78.1
76.0
CDAN 2
81.1
84.2
77.2
84.8
76.2
72.5
84.6
69.0
69.1
84,9
79.8
80.1
78.6
DINO 25
TVT 14
70.4
79.7
64.2
80.2
68.1
65.2
81.6
61.9
65.6
80.3
71.4
74,1
71.9
Shen o7 ol PS
76.6
81.9
76.8
82.1
74,9
74,3
84.7
69.7
74.3
77.9
80.
78.0
SENTRY J
81.8
80.9
73.4
89.1
79.4
75.8
86.5
75.6
716
88.3
81.9
82.4
80.6
(Ouns)
78.3
87.3
75.5
79.6
while maximizing it decrease confidence on highly inconsistent ones. 4) Shen et al. 28] Proposes contrastive learning on the pooled source and target domain followed by finetuning on source labels. We adapt their upproach by using SSL strategies designed for ViTs (described below), and by starting from ImageNet SSL. weights rather than from scratch. 5) TVT [4]] Transferable Vision Transformer (TVT) injects learned transferability measure into the transformer attention blocks, in addition to performing global domain aovainaadversarial alignment and discriminative clustering.
SSL Strategies: As discussed in Sec. 3 PACMAC is designed as an adaptation strategy for self- supervised representations. We report benchmark performance on top of two popular SSL strategies for ViTs (see supp. for more details):
MAE [26): Masked Autoencoding (MAE) is self- supervised learning strategy that learns a visual transformer autoencoder to reconstruct images given only a random subset of patches from the original image. By masking out large portions of images (~75%), the MAE encoder is shown to learn strong representations that can be effectively finetuned to downstream tasks. 2) DINO
Deff-Distillation with No Labels supervised learning strategy that passes two transformed versions of each image to student and teacher network respectively, and trains the student network predict the output of the teacher. In supp. for completeness we also benchmark PACMAC on top of a supervised ImageNet initialization.
visual self-
4.4 Results
Tables and present results. We observe:
Table 2: VisDA. Target accu- racy with MAE 25 init.
Several existing DA methods underperform top of SSL repre- sentations. On both OfficeHome and DomainNet, we observe existing DA methods (CDAN (7). MCC [49]. TVT [4) to frequently underper- form even the source model on average, despite hyperparameter tuning. The fact that many existing DA methods struggle with such initializa- tions suggests the need for specialized solutions, particularly as SSL pretraining becomes more common. SENTRY 45), Shen et al. (28 and PACMAC however offer consistent improvements.
Method
Acc.
source
63.5
CDAN 7
72.4
TVT 14
62.3
Shen el al.
28
79.3
SENTRY
76.0
PACKAC (Ours)
81.0
b PACMAC outperforms prior work across benchmarks and initializations. On OfficeHome (12 shift average), PACKAC improves over the next-best method by 1.3% (MAE init.) and 0.2% (DINO). On DomainNet, we observe gains of 1.1% (MAE) and 0.4% (DINO), and 1.7% (MAE) on VisDA.
We note here that PACMAC and SENTRY both make use of selective self-training on reliable instances identified via predictive consistency, but differ in important ways: i) PACMAC roughly matches the design its SSL pretraining and measures consistency across partial masked inputs rather than random augmented images, ii) P PACMAC incorporates model knowledge in its selection strategy to focus on salient image regions, rather than random augmentations
by using intention-condiitinn sampled from manually pre-defined set. These differences lead to its improved performance, despite being considerably simpler losses, Eq. 7 and no diversity regularizers or entropy maximization).
4.5 Ablating PACMAC
In Tables 4 we ablate PACMAC with DINO initialization OfficeHome Cl-Pr. We observe:
7

---

Table 4: OfficeHome CI-Pr. Gray is ours: a) Ablating pretraining. Cross-domain KNN and finetuning (FT) accuracies for MAE and DINO strategies (S=Source, T=Target, PT=Pretrain, dom.=domain). b) Varying PACMAC init. Transfer accuracy with & without S+I pretraining c) Ablating selection strategy with DINO init.
(a) Pretraining ablations
(b) Vary initialization
(c) Ablating selection strategy
select on target
Acc.
Pretraining domains
MAE
DINO
MAE
DINO
all
59.8
KNN
FT
KNN
FT
Source
60.0
66.2
confident
71.3
INIK PT
29.6
60.0
57.9
66.2
PACMAC
67.1
74.0
consistent
73.7
+ S+T PT
41.4
66.2
56.5
68.9
S+T PT
64.7
71.3
consistent AND confident
72.2
dom. decoders
19.7
57.7
N/A
N/A
consistent OR confident
74.0
correct (oracle)
94.0
& In-domain pretraining (4a We report two metrics: cross-domain k-Nearest Neighbor (k=7) accuracy on the target domain using source domain embeddings from the trained encoder (KNN column), and target accuracy after finetuning on the source (FT column). Across MAE and DINO initializations, we observe additional in-domain pretraining on the pooled source and target domain to improve finetuning performance (Row 2 v/s I, +6.2% FT acc. with MAE, +2.7% % with DINO). With MAE, we additionally try pretraining only on source target domains and find them to underperform S+T pretraining (not in table). We also try S+T pretraining with MAE by learning separate decoders to reconstruct source and target images, and find that to underperform no pretraining (Row 3 v/s 1, -2.3%) Interestingly, we find pretraining to improve kNN accuracy in all cases except Row 3 with MAE, indicating better domain alignment in encoder feature space. With DINO, we observe higher kNN accuracies than MAE, but see a slight drop after in-domain pretraining.
> PACMAC benefits from S+T pretraining (4b) We apply PACMAC to ImageNet SSL features without S+T pretraining and observe worse accuracy (Row v/s 2, -2.4% with MAE, -2.7% % with DINO).
& Combining masking consistency and confidence is an effective selection measure. In Table 4 we first self-train on all (Row 1) or only confident (model confidence >> 50%, Row 2) target examples. We find both to underperform selection based on our proposed autention-conditionn sistency scheme (Row 3. +11.5% and +13.5% ). We further combine our consistency measure with confidence and find selecting instances marked as reliable by atleast one measure to perform best (Row 5). In Row 6, as we report upper-boune performance of an oracle method which only self-trains on correct pseudolabels, and find it to achieve high accuracy of 94%. We also try self-training on the original rather than masked image (not table) and find it to underperform by 3.2%. However, self-training on the masked image without using it for selection does significantly worse (-9.7% ): clearly, the primarly benefit of masking is due to better selection rather than regularization.
masking con-
> Matching selection strategy to DINO 255 So far, for simplicity we match MAE's 26 design for PACMAC'S selection and measure consistency across masked images. To demonstrate generality, we now match DINO's design and instead measure predictive consistency across local-and- -lobal image crops (both random and with srcention-seedinge We find that for DINO initialization this further improves accuracy (74.3% on CI-PPr), suggesting that closely matching the design of the selection criterion to the SSL pretraining is beneficial (details and visualization in supp.).
> Comparing selection strategy to SENTRY ( We swap out PACMAC'S attention-conditionn masking consistency scheme with the committee consistency across augmentation scheme used by the next best performing method, SENTRY ( We match original hyperparameters and use a committee size of 3, RandAugment [47] parameters of N=3 and M=2.0, and majority voting. Averaged over 12 OfficeHome shifts, we find this to obtain a transfer accuracy of 66.1% 1% (MAE) and 67. 67.4% (DINO), while PACMAC obtains 66.8% (MAE) and 69.6% (DINO). This clearly establishes that PACMAC'S gains over SENTRY are due to improved selection rather than a stronger initialization.
Attention-conditionnng helps, does committee consistency. In Table 5a we vary the com- mittee size k and masking strategy (random v/s aitention-conditionnel masking). As seen, attention conditioning consistently improves upon random masking across l (+0.5 to 1.8%).
> A Ablating masking ratio and confidence threshold. In Tables 5b and 5c we vary the masking ratio mr and confidence threshold T, and find m5=75% and 5=50% to work best.
8

---

Table 5: Ablating the consistency checker on OH CI-Pr: a) Varying committee size (k), masking strategy (mdd-rrnddmm, att.=atiention, U=unanimous, M=majority voting). h) Varying masking ratio e) Varying confidence threshold for selection. Gray is our method. (a) Ablating consistency checker
(b) Vary masking ratio
(c) Vary confidence threshold
Masking ratio
Acc.
conf. thresh.
Acc.
k=1
1=2
f=3 (U/M)
rnd. mask
68.1
72.6
71.3/72.0
50%
72.3
25%
73.7
all. mask
68.6
74.0
73.1/72.8
75%
74.0
50%
74.0
90%
71.0
75%
72.9
200.2
Selection atredggy precsion
100.0
Percaniage samples selecied
Classifier Error (%)
PT
same C diff. C diff. C DA
60-2
X0L0
strat.
diff. D diff D same D diff. D score
70.2
70.0
MAE
8.1
10.6
6.6
3.3
3.8
PACMAC- DINO PT
DINO
10.4
8.5
2.6
1.0
5.9
PACMAC
45
163
326
Sup.
14.2
11.9
3.2
1.6
8.5
Epoch
Figure 2: Left. Precision of our selection strategy with DINO (bluc) and MAE (ycllow) initializations. Right. Percentage of target examples selected for self- self-training across epochs.
Table 6: Understanding SSL initialization. Error of lincar classifier trained distinguish features for: C2. Domains. C3. Same class, different domains. C3. Different class, same domains. C4. Different class, different domains. C5. Domain alignment score. baged across OfficeHome shifts.
4.6 Analyzing PACMAC
Evaluating reliability estimation, In Fig. 2 we evaluate our proposed reliability estimation scheme across MAE and DINO initializations. We observe high precision (70-80% across both) across epochs, indicating that our method correctly identifies reliable instances with a low false-positive rate (per-class analysis in supp.). We also find that the percentage of target instances selected for self-training increases over time, and is particularly high with DINO pretraining.
Variance across runs. To measure the performance variance of our method PACMAC, we run it across random seeds on the OfficeHome Clipart-PProdutt shift and observe a target accuracy mean of 73. 33.1% 1% with a standard deviation of 0.59%.
Understanding SSL initialization In Tab. 6 we contrast self-supervised (MAE and DINO) and supervised ImageNet initialization, by reporting the error of linear classifiers (28) (averaged over all OfficeHome shifts) trained to distinguish different sets of class token features, such as: C2. Different domains. We observe higher error for the supervised init, indicating better domain alignment. C3. Same class but different domains. Low error with DINO indicates less per-class domain alignment. C4. Different classes but same domain. High error with MAE indicates more inter-class confusion. C5. Different domains and classes. C6. We define a domain alignment (DA) score = C3-max(C4, C5) higher means that cross-domain examples from different domain but same class are on average closer than examples from different classes and either the same or different domains. Intuitively, high DA score will translate to better performance for CDAN, which aligns logits-conditione source and target feature We find that SSL inits indeed have lower DA scores, possibly explaining why methods like CDAN struggle.
Visualizing attention-conditiones masking consistency. In Fig. we visualize our proposed attention attention-conditionn conditioned masking consistency scheme for random target images from OfficeHome. As seen, attention-conditee conditioning ensures that the masked images focus on disjoint, highly attended regions of the target image (Row 2). Columns 1-5 illustrate examples for which our selection strategy is correctly able to identify reliable (Cols 1-2) and unreliable instances (Cols 3-5). Cols 6-7 denote
"To validate this, we measure Pearson correlation between DA score and transfer accuracy with CDAN on all OH shifts, and observe high correlation: 0.9 (MAE), 0.85 (DINO), and 0.75 (supervised).

---

candle
flowers
paper clip
eraser
fridge
post-its
Original candle
flowers
SPOOD
ruler
computer
post its
batteries
Attention
Calendar
notebook
batteries
Masked
candle
flowers flowers
spoon
oven
cabinet
notebook
batteries
Masked
canele
penci
reliable
reliable V
unreliable /
unreliable v
unreliable v
unreliable X
reliable X
Figure 3: Visualizing PACMAC. Row I: Ground truth label. Row 2: Original image. Row Per-patch attention.
Rows 5-6: Masked images. We include model predictions the of each image, color coded as green (correct)
and red (incoreect. Row 6: PACMAC consistency (tick and cross denote correct and incorrect assessment).
failure cases, with the first showing a false negative (correctly classified instance being misidentified as unreliable) and the second denoting false positive.
Effect of in-domain pretraining per-class kNN accuracy. In supplementary, report per-class cross-domain kNN accuracy on the OfficeHome CI-Pr shift before and after in-domain pretraining across MAE and DINO initializations. We find that accuracy improves several classes in both cases, particularly with MAE. While prior work has observed benefits from additional in-domain corroborate this finding with sulf-supe ViTs.
pretraining for CNNs 28 27).
In the appendix, we analyze the effect of in-domain pretraining on out-of-distribution confidence calibration (trends vary across shifts). and include t-SNE [51] visualizations of the feature space learned by encoders before and after in-domain pretraining.
5 Limitations
PACMAC requires additional i) in-domain pretraining and ii) forward passes over masked target images (but no additional backpropagation), which makes it slower. Further, PACMAC'S effectiveness at identifying reliable instances varies across categories (see supp.). In our paper we only experiment with ViTs trained for image classification, and the generality of our approach across architectures and tasks is not established. Finally, the performance of DA methods with SSL initializations still lags behind supervised initializations on standard benchmarks like OfficeHome and DomainNet (comparison in supp.). We hypothesize that this may be due to strong category overlap between ImageNet and these benchmarks and may not translate to other datasets, but this requires further study.
Conclusion
We focus on adapting modern vision architectures (ViT's initialized with state-of-the-art pretraining (self-supervised learning SSL.). Our method PACMAC first performs SSL source and target data, and then uses predictive consistency across partial versions of target images generated via an attention- conditioned masking strategy to judge reliability for selective trainin. Despite its simplicity, PACMAC outperforms prior work at adapting self-supervised ViTs trained for object recognition.
Acknowledgements. This work was supported in part by funding from the DARPA LwLL project and ARL.
10

---

References
[1] A. Torralba and A. A. Efros, "Unbiased look at dataset bias," in CVPR 20 pp. 1521-1528, IEEE, 2011.
[2] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting visual category models to new domains," in European conference computer vision. pp. 213-226, Springer. 2010.
[3] E. Tzeng. J. Hoffman, N. Zhang. K. Saenko, and T. Darrell, "Deep domain confusion: Maximizing for domain invariance," arXiv preprint
2014.
[4] M. Long, Y. Cao, J. Wang, and M. Jordan, "Learning transfcrable features with deep adaptation networks," in International conference on machine learning, pp. 97-105, PMLR, 2015.
[5] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, "Adversarial discriminative domain adaptation." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
[6] Y. Ganin and V. Lempitsky. "Unsupervised domain adaptation by backpropagation." " in International conference on machine learning. pp. 1189, PMLR, 2015.
M. Long, Z. Cao, J. Wang, and M. I. Jordan, "Conditional adversarial domain adaptation," Advances in Neural Information Processing Systems, vol. 31, 2018.
[8] J. Hoffmman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrlll "Cycada: Cycle- consistent adversarial domain adaptation,"" " in International conference on machine learning, pp. 1989-1998, PMLR, 2018.
[9] V. Prabhu, Khare, D. Kartik, and Hoffman. "Sentry: Selective entropy optimization via committee con- sistency for unsupervised domain adaptation." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558 8567, 2021.
[10] O. Russakovsky, J. Deng, Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bemstein, et al., "Imagenet large scale visual recogmition challenge," International journal of computer vision. vol. 1 115, no. 3, pp. 211-252, 2015.
[11] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference
Learning Representations, 2020.
[12] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, "Transformers in vision: A survey," ACM Computing Surveys (CSUR), 2021.
[13] M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlshy, D. Tran, and M. Lucic, "Revisiting the calibration of modern neural networks," Advances in Neural Information Processing Systems, vol. 34, 2021.
[14] J. Yang, J. Liu, N. Xu, and J. Huang. "Tvt: Transferable vision transformer for unsupervised domain adaptation." arXiv preprint arXiv:2108 059988 2021.
[15] T. Xu, W. Chen, P. Wang, F. Wang, H. Li, and R. Jin, "Cdtrans: Cross-domain transformer for unsupervised domain adaptation," arXiv preprint arXiv: (0l::210.000 109,06165, 2021.
[16] M. Raghu, C. Zhang, J. Kleinberg, and S. Bengio, "Transfusion:
Understanding transfer learning for medical imaging," Advances in Neural Information Processing Systems, vol. 32, 2019.
[17] S. Azizi, B. Mustafa, F. Ryan, Z. Beaver, J. Freyberg, J. Deaton, A. Loh, A. Karthikesalingam, S. Kornblith, T. Chen, et al., "Big self-supervised models advance medical image classification," in Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 3478-3488, 2021.
[18] C. Docrsch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction"" in Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
[19] X. Wang and A. Gupta, "Unsupervised learning of visual representations using videos," in Proceedings of the IEEE international conference computer vision, pp. 2794-2802, 2015.
[20] M. Noroozi and P. Favaro, "Unsupervised learning of visual representations by solving jigsaw puzzles," in European conference computer vision, pp. 69-84, Springer, 2016.
[21] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non- con-parametric instance discrimination." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733- 3743- 2018.
[22] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations." in International conference machine learning, pp. 1597-1607, PMLR, 2020.
[23] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
11

---

[24] M. Caron, I. Misra. J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments," Advances in Neural Information Processing Systems, vol. 33, pp. 9912-9924, 2020.
[25] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, "Emerging properties
in self-supervised vision transformers," in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1-5009660,,4 2021.
[26] K. Hc, X. Chen, S. Xic, Y. Li, P. Dollár, and R. Girshick, "Masked autocncoders are scalable vision learners," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
[27] D. Kim, K. Saito, T.-H. Oh, B. A. Plummer, S. Sclaroff, and K. Saenko, "Cds: dross-domain self- supervised pre-training." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9123-9132, 2021.
[28] K. Shen, R. Jones, A. Kumar, S. M. Xie, J. Z. HaoChen, T. Ma, and P. Liang, "Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation." 2022.
[29] M. Assran, M. Caron, Misra, Bojanowski, F. Bordes, P. Vincent, A. Joulin, Rabbat, and N. Ballas, "Masked siamese networks for -ffeleefficient learning." arXiv preprint arXiv:2204.07144 2022.
[30] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, "Deep hashing network for unsu- pervised domain adaptation." " in Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 5018-5027, 2017.
[31] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, "Moment matching for multi-source domain adaptation," in Proceedings of the IEFFICVF international conference on computer vision, pp. 1415- 2019.
[32] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko, "Visda: The visual domain adaptation challenge," arXiv preprint arXiv:1710.06924 2017.
[33] S. Tan. X. Peng. and K. Saenko, "Class-imbalances domain adaptation: an empirical odyssey." in European Conference on Computer Vision, pp. 585-658 Springer, 2020.
[34] Н. Bao, L. Dong, and F. Wei, "Beit: Bert pre-training of image transformers," arXiv preprint arXiv:2106.08Z2
2021.
[35] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting." " in Proceedings of the IEEE conference on computer vision and pattern recognition.
pp. 2536-2544,2016
[36] Y. Shi, N. Siddharth, P. H. Torr, and A. R. Kosiorek, "Adversarial masking for self-supervised learnin,"' in International Conference on Machine Learning, 2022.
[37] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[38] Z. Xie, Z. Zhang. Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, "Simmim: A simple framework for masked image modeling," " arXiv preprint arXiv:21 11.0888 2021.
[39] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer, "Masked feature prediction for sule-supe visual pre-training"" arXiv preprint arXiv:21 2021.
[40] Y. Grandvalet and Y. Bengio, "Semi-supervised learning by entropy minimization," Advances Neural Information Processing Systems, vol. 17, 2004.
[41] C. Chen, W. Xie, W. Huang, Y. Rong, X. Ding. Y. Huang, T. Xu, and J. Huang, "Progressive feature alignment for unsupervised domain adaptation," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 627-636, 2019.
[42] X. Jiang, Q Lao, S. Matwin, and M. Havaci, "Implicit class candiioondiio domain alignment for unsupervised domain adaptation," in International Conference on Machine Learning, pp. 4816 4816-4827, PMLR, 2020.
[43] Y. Zou, Z. Yu, B. Kumar, and J. Wang. "Unsupervised domain adaptation for semantic segmentation via balsnealaneed self self-training." Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
[44] J. Kim, I. Hwang, and Y. M. Kim, "Ev-tta: Test-time adaptation for event-based object recognition," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[45] V. Prabhu, S. Khare, D. Kartik, and J. Hoffman, "S4t: Source-free domain adaptation for semantic segmentation via self-supervised selective self-training," arXiv preprint 107 2021.
[46] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in International Conference on Learning Representations, 2018
[47] E. D. Cubuk, B. Zoph. J. Shlens, and Q. V. Le, "Randaugment: Practical automated data augmentation with reduced search space," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 703-703, 2020.
12

---

[48] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, et al., "Pytorch: An imperative style, perfoomaeeee deep learning library," Advances in Neural Information Processing Systems, vol. 32, 2019.
[49] Y. Jin, X. Wang, M. Long, and J. Wang. "Minimum class confusion for versatile domain adaptation." in European Conference Computer Vision, pp. 464 480, Springer, 2020.
[50] D. Kim, K. Wang. S. Sclaroff, and K. Saeuko, "A broad study of pre-training for domain generalization and adaptation," arXiv preprint arXiv:2203. 1819, 2022.
[51] L. Van der Maaten and Hinton, "Visualizing data using t-sne." Journal of machine learning research, vol. 9, no. 11, 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A] See Sec.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Sec. 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or a URL)? [Yes] See supplementary.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. Sec. and supplementary.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Sec 4.1
(b) Did you mention the license of the assets? [Yes] See supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [Yes] See supplementary.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See supplementary.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sCClK7Rz4CN", "reviewer": "Reviewer_MM52", "summary": "In this paper, the authors propose a new method for adapting Visual Transformers (ViT) with modern self-supervised learning (SSL) pretraining adapted to unsupervised domain adaptation (UDA). The proposed approach, named PACMAC, starts with a weights pretrained on Imagenet and follows of two steps: (i) in-domain SSL on both source and target datasets and (ii) self-training model on target based on a \"reliance measure\" (based on predictive consistency across many masked inputs). The work shows good results on multiple standard UDA evaluation datasets.\n\n\n=======================\n### Post-rebuttal updates\nI thank the authors for their feedback. After looking at the rebuttal and other reviewers' comment, I decided to keep my “Weak Accept” (6) rating.  \nI continue thinking that the paper achieves good empirical results (and the authors provide a good range of experiments), but the novelty is fairly limited—as agreed by all the reviewers. ", "questions": "* I found Table 6 very difficult to understand/parse. Could the authors elaborate a bit more what is it about? And more importantly, I would recommend the authors to update the manuscript and make it more clear as well.\n* How does capacity of the model and inference time change between PACMAC and other SOTA methods (eg, SENTRY, Shen et al.)?\n", "limitations": "The authors were clear with the limitation of the method (slower and not same results as supervised initialization).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Pros\n* The paper is clear, easy to follow in most parts and provides a good literature review.\n* The authors show good results compared to current models and provide good ablation studies.\n* The idea of using \"attention-conditioned\" masking makes sense and \"consistency\" on self-training are interesting (however, they are not related to UDA and would be better studied in a general context rather than in the particular case of unsupervised domain adaptation).\n* The problem of domain shift, OOD and domain adaptation in neural networks is important to the community.\n\n### Cons\n* The main weakness of this paper IMO is the lack of novelty. As pointed out in the manuscript, most of the components of the proposed method have already been explored in previous literature (SSL for UDA, self-training for UDA). The main contribution is mostly adapting those approaches to a new architecture (ViT).\n* The main contribution then becomes the application of ViT-specific tricks on either the SSL or self-training components (eg, the attention-conditioning mask). This trick, however, could as well be used in any SSL/self-training method that relies on ViT, not only on UDA problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cPNDJioZWKx", "reviewer": "Reviewer_pAyV", "summary": "The paper focuses on adapting self-supervised ViTs (MAE[26] and DINO[25]) for unsupervised domain adaptation (UDA). The goal is the find reliable pseudo labels for unlabeled target domain images and then train the model using pooled source and target data. The paper uses a setup prevalent in the existing literature to find the reliable image-pseudo-label pairs – filter the target domain images with consistent predictions across multiple augmented versions and consider the predicted labels for such images as reliable pseudo labels. The main contribution of this paper lies in designing ViT-specific data augmentation, namely attention-guided masking. The presented method is dubbed ‘PACMAC’, short for Probing Attention-conditioned Masking Consistency for UDA.", "questions": "* Looking at the experiment presented in Sec 1 of supplementary material, the improvements offered by PACMAC over SENTRY can be attributed to data augmentation and not the loss function. Please explain which other features of PACMAC are superior to SENTRY.\n* Please compare the attention-guided masking augmentation and other types of augmentations such as attention-guided cropping (e.g. one that matches the multi-crop augmentation of DINO).\n", "limitations": "Limitations discussed. There are no ethical concerns.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper attempts to find ViT-specific data augmentation for finding reliable image-pseudo-label pairs using FixMatch[R1]-style setup.\n* The paper is well-presented with a good set of experiments.\n\nWeakness:\n\n* The learning setup is very similar to the previous method called SENTRY [45]. As described in L239-244, the only difference between the two methods is the data-augmentation scheme and the loss function. Further, Sec 1 of supplementary material establishes the superiority of the loss function used in SENTRY over the one presented in the paper.\n* Augmentation being the main contribution, the paper lacks a comparison between attention-guided masking augmentation and other types of augmentations (such as random or attention-guided cropping).\n* It is mentioned in L240 that PACMAC intends to design a data augmentation scheme that matches the design of the SSL pretraining. However, the presented method only matches the design of MAE, not DINO. One can observe that the improvement offered by PACMAC over the previous state-of-the-art is lower for DINO than MAE.\n\nReferences:\n\n[R1] Sohn, Kihyuk, et al. \"Fixmatch: Simplifying semi-supervised learning with consistency and confidence.\" NeurIPS, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IKyp9sc4zcL", "reviewer": "Reviewer_LLh8", "summary": "The paper proposes PACMAC, a method for unsupervised domain adaptation tailored to vision transformer architectures (ViT) pre-trained using self-supervised approaches (SSL). \nThe method works by 1) continued SSL training on the union of the labeled source domain and unlabelled target domain, and 2) joint fine-tuning on the source domain and self-training on the target domain.\nFor the self-training on the target domain, the paper proposes a mechanism based on the consistency of the model's prediction on differently masked inputs to select examples for self-training. \nThese input masks are generated based on the ViT self-attention scores and a greedy selection mechanism.\nExperiments demonstrate improvements over prior work in various benchmarks and include ablation experiments to demonstrate the importance of continued SSL pre-training and the example selection strategy for self-training. ", "questions": "I would appreciate it if the authors could address the weakness listed above, i.e., \n- How they aim to improve the presentation and where to introduce the classifier training.\n- Clarify the novel contributions and how the method compares to a combination of [28]+[9], which appears conceptually very similar.\n- Clarify whether the benefit of the masking is more due to example selection or additional data augmentation. ", "limitations": "Limitations were addressed well in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Domain adaptation is of practical importance, and investigating approaches for modern architectures and pre-training strategies (ViT + SSL) is useful\n- Using consistency of predictions between differently masked inputs is an interesting novel idea\n- Extensive experimental evaluation\n- Good performance in various domain adaptation benchmarks\n- A good set of ablation experiments is provided\n\n\nWeaknesses:\n- I found the presentation quite confusing. The main reason for this is that it is unclear how self-training (which typically leverages class predictions) relates to a self-supervised model (the focus of this paper). Only at L193 is it mentioned that a classifier is learned on the source data after SSL pre-training, and I assume that those predictions are then used for the self-training (+mask consistency, etc.). This makes the paper up to that point rather confusing. I also assume this fine-tuning is performed after the continued SSL on source+target data, but it is unclear based on the paper. \n- It appears that the novelty of the method is limited mainly to the mask-consistency strategy to select examples for self-training. For example, the SSL pre-training on pooled source+target data was proposed in [28]. Furthermore, [9] offers a selection mechanism based on consistency across augmentations, which is conceptually quite similar to the mask consistency proposed here. Therefore, it would be important to see how a combination of [28] and [9] would do, especially since [9] by itself appears to perform well already. \n- It is unclear whether the selection mechanism through mask consistency benefits the method or the additional augmentations through the use of masking during self-training. Indeed, only using the selection appears to lead to subpar performance (see L268). It might be good to investigate this observation further and disentangle these two factors. \n\nPOST REBUTTAL UPDATE:\nThe authors addressed some of my concerns in terms of presentation and the benefits of the selection mechanism (compared to additional augmentations). I still think the novelty over [28]+[9] is somewhat limited, but improvements appear to be solid. I'm thus raising my rating. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept