Subsidiary Prototype Alignment for Universal Domain Adaptation
Authors: Jogendra Nath Kundu
Suvaansh Bhambri
Akshay Ravindra Kulkarni
Hiran Sarkar
Varun Jampani
Venkatesh Babu Radhakrishnan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain Adaptation
Universal Domain Adaptation
Object Recognition
Bag of visual words

Abstract: Universal Domain Adaptation (UniDA) deals with the problem of knowledge transfer between two datasets with domain-shift as well as category-shift. The goal is to categorize unlabeled target samples, either into one of the "known" categories or into a single "unknown" category. A major problem in UniDA is negative transfer, i.e. misalignment of "known" and "unknown" classes. To this end, we first uncover an intriguing tradeoff between negative-transfer-risk and domain-invariance exhibited at different layers of a deep network. It turns out we can strike a balance between these two metrics at a mid-level layer. Towards designing an effective framework based on this insight, we draw motivation from Bag-of-visual-Words (BoW). Word-prototypes in a BoW-like representation of a mid-level layer would represent lower-level visual primitives that are likely to be unaffected by the category-shift in the high-level features. We develop modifications that encourage learning of word-prototypes followed by word-histogram based classification. Following this, subsidiary prototype-space alignment (SPA) can be seen as a closed-set alignment problem, thereby avoiding negative transfer. We realize this with a novel word-histogram-related pretext task to enable closed-set SPA, operating in conjunction with goal task UniDA. We demonstrate the efficacy of our approach on top of existing UniDA techniques, yielding state-of-the-art performance across three standard UniDA and Open-Set DA object recognition benchmarks.

Subsidiary Prototype Alignment for Universal Domain Adaptation
Jogendra Nath Kundu Suvaansh Bhambri
Akshay Kulkarni
Hiran Sarkar'
Varun Jampani
R. Venkatesh Babu²
Indian Institute of Science
Google Research
Abstract
Universal Domain Adaptation (UniDA) deals with the problem of knowledge transfer between two datasets with domain-shift as well as category-shift. The goal is to categorize unlabeled target samples, either into one of the "known" categories or into a single "unknown" category. A major problem UniDA negative transfer, i.e. misalignment of "known" and "unknown" classes. To this end, we first uncover an intriguing tradeoff between megative-transfer-ee invariance exhibited different layers of deep network. It turns out can strike a balance between these two metries at a mid-level layer. Towards designing an effective framework based on this insight, draw motivation from Bag-of-visual Words (BoW). Word-prototypes in a BoW-like representation of a mid-level layer would represent lower-level visual primitives that are likely to be unaffected by the category-shift in the high-level features. We develop modifications that encourage learning of word-prototypes followed by word-histogram based classification. Following this, subsidiary prototype-space alignment (SPA) can be seen as a closed- set alignment problem, thereby avoiding negative transfer. We realize this with a novel word- word-histop histogram ceaelgrra--rlltted related pretext task to enable closed-set SPA, operating in conjunction with goal task UniDA. We demonstrate the efficacy of our approach on top of existing UniDA techniques' yielding statc-of-the the-art performance across three standard UniDA and Open-Set DA object recognition benchmarks.
and domain-
Introduction
Despite the success of deep networks trained on large-scale datasets, they are found to be brittle under input distribution shift i.e. domain-shift [6]. Thus, adapting trained model for new target environment becomes challenging as data annotation is too expensive time-consuming [7] for every new target dataset. Unsupervised Domain Adaptation (DA) [11] is one of the solutions to this problem where knowledge is transferred from a labeled source domain to unlabeled target domain.
While most works [12, 25, 19] focused on Closed-Sas DA, where source and target label sets are shared (C, C.), recent works introduced disjoint label set scenarios like Partial DA [51, 4] (C, C C.) and Open-Set DA [37, 17] (C, C.). However, the most practical setting Universal DA (UniDA) [48] where the relation between the source and target label sets is unknown i.e. with any number of shared, source-private and target-private classes. UniDA, a model is trained to categorize unlabeled target samples into one of the shared classes ("known" classes) or into single "unknown" class.
The major problem in UniDA private classes degrades the adaptation performance. On the other hand, we have the domain- problem, which is usually handled by learning domain-invariant features [48, 18, 21, 34]. So, we first
rngative-transier [38] where misalignment between the shared and
"equal contribution
Project Page: https://sitee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

fg UniDA objectives
Source
0000
4
4
SPA
hon
030066089
DDC
fn Realized via Bow-inspired framework Pretext task to aid SPA
Target
Closed-set prototype-space alignment 02 2
67 0000
Figure UniDA involves source and target data with shared and private classes. The colored blobs represent target features w.r.t. the word-prototype (numbered) that represent lower-level visual primitives. With word-related pretext task guiding the Subsidiary Prototype-space Alignment (SPA) to avoid word-level misalignment, the word-histogram output-space can better represent the intrinsic class-structure (including unknown classes), leading to better UniDA performance.
perform a control experiment to analyze the (DIS) different layers of the deep network (Fig. 2). NTR measured through the class-specificity of the features via an entropy-based shared-vs-unkmng as the inverse of the standard A-distance [1] that quantifies domain discrepancy. We observe that NTR and DIS are at odds with each other i.e. NTR increases while DIS decreases we move from lower to higher-level (deeper) layers. Thus, we arrive contradicting solutions where avoiding gggative-transfer requires adaptation at a lower-level layer while effective domain-invariann invariance requires adaptation at deeper layer. While a balance can be naively struck at a mid-level layer, we ask, can we further develop and constrain the mid-level representation space to avoid negative transfer?
(NTR) and cammain-nnvaiancc-scom
binary classifier while DIS is measured
Motivated by Bag-of-visual- Words (BoW) [49], we hypothesize that BoW-like representation at mid-level layer would represent lower-level visual primitives that are unaffected by the category-shift in the higher-level features. We illustrate this idea in Fig. with UniDA scenario where source and target have private and shared classes. Note that, in most practical scenarios, the private categories are usually related to the shared categories (e.g. a tractor and an excavator may be private classes in rural and urban scenes respectively. but they share some common visual attributes like their chassis). Thus, in a word word-prototype-sppe (including unknown classes) and a closed-set alignment of target features with the primitives can be performed. Next, we explain how to realize this subsidiary prototype alignment (SPA).
different visual primitives can be shared across domains and classes
First, we propose architecture modifications to introduce explicit histogram output. We analyze the alignment of feature vectors with different word-prototypes. Here, better prototype-space alignment would imply sparser word word-bistograms as specific word-prototype would have significant contribution to the word-histogram w.r.t. other prototypes. Intuitively, with higher sparsity, the word-histogran space can better represent the intrinsic class structure (including unknown classes). To enforce this sparsity i.e. the subsidiary prototype-space alignment (SPA), we minimize a self-entropy loss at the word-histogram level. However, this remains susceptible to word-level misalignment due to a lack of word-related priors.
and extract a word-
Thus, we seek a word-related prior that can be cast into a self-supervised pretext task so that both labeled source and unlabeled target can be used. Given that each instance be represented word-histogram, we find a simple property based on grid-shuffling of image crops (Fig, 4). Here, the word-histogram entropy of a grid-shuffled image increases with the number of distinct instances that contribute crops, Based on this, we create a novel pretext task to classify the number of instances used in an input grid-shufiled image. Intuitively, it encourages prototype alignment (SPA) distinguishing different classes in this pretext task becomes easier with higher sparsity of word mirthhistg.amss.
To summarize, contributions are as follows,
We are the first to uncover and analyze the tradeoff between negative- transfer and domain invaaiinnne in UniDA. While naive balance can be struck, we introduce BoW-inspired word- word-prototypee prototypes and subsidiary prototype-space alignment (SPA) objective to further alleviate negative transfer.
regative-transfer.
We devise a word-related prior and cast it as self-supervised pretext task to further improve SPA by avoiding word-level misalignment.
Our approach, coupled with existing UniDA approaches, yields state-of-the-an performance across three standard Open-Set DA and UniDA benchmarks for object recognition.

---

2 Related work
Open-Set DA (OSDA) has been studied in several scenarios [37, 29, 3, 8]. We focus the case given by [37], where target domain contains private classes, unknown to source. [37] presented an adversarial learning method where feature generator learns known-unknown separation. Other works focus on anomaly measurement [30] or learn to discriminate known and unknown samples [47, 33, 28]. ROS [3] uses rotation prediction to separate known and unknown samples whereas our word-based pretext task regularizes UniDA and implicitly improves known-unknown separation.
Universal DA (UniDA) [48] is complex DA scenario that assumes no prior knowledge of the relationship between the source and target label spaces. Similar to Open-Set DA, UniDA also requires identification of target-private classes. Prior works [10, 48, 38, 3] computed confidence score for known classes, and data with a score below a threshold were considered unknown. [38] proposed neighbourhood clustering understand the target domain structure and entropy separation loss for feature alignment. OVANet [36] used binary classifiers in a one-vs-all manner to identify unknown samples, and DCC [23] used domain consensus knowledge find discriminative clusters in both shared and private data. In contrast to prior arts, we draw motivation from wo--of-vsual-Worde and construct pretext task to complement these works by enhancing their intrinsic domain structure.
Bor-teet works. Early works utilized Bag-of visual Words (BoW) representations for down- stream applications like object recognition [44], object detection [2]. image retrieval [39], etc. More recent work [13, 14] leveraged BoW prediction for self-supervised learning of representations for downstream tasks. To the best of our knowledge, we the first to utilize BoW concepts in UniDA.
3 Approach
3.1
Preliminaries
Consider labeled source dataset D = [xx, (() :r, X. C.} where C, is the source label set, X is the input space, and Fg drawn from the marginal distribution p.. Also consider an unlabeled dataset D.- = {Iz, It X3 where is drawn from the marginal distribution Pt. Let C, denote the target label set. Universal DA [48], the relationship between C. and C, unknown. Without loss of generality, the shared label set is C C₂ C, and the private label sets for source and target are C, - C. C, and C, = C. \ C, respectively. Next, we define two measures for our insights.
Negative-Tranee Risk (NTR). Negative transfer [43] is major problem in DA where class-level misalignment occurs (e.g. source class "dog" may get wrongly aligned with target class "cat" due to some similarities between the two classes). The problem is aggravated in UniDA as both source and target may have private classes which may be wrongly aligned with the shared classes [48]. Thus, we introduce a
(NTR) "wrrehh) for given feature extractor h X - 2, where 2 is an intermediate feature-space. NTR is computed as the target shared-vs-privat classification accuracy via a self-entropy threshold on source-trained linear task-classifier (see Suppl. for complete details),
(h)
E
1
(sp)
(ep)
(sp) Here, g/-p) is the prediction (shared or private) using fixed self-entropy threshold and y represents shared- shared-private label (0 for shared, 1 for private). We access the shared- -hived- labels for a subset of target data only for analysis (not for training). Intuitively, if the features h(z) are highly class-specific, then target-private samples would yield highly uncertain predictions (as they are unseen by the source-trained linear classifier) compared to shared-label samples. Thus, target-private samples would be easily separable with the self-entropy threshold, leading to a high NTR.
(DIS). The standard A-distance [1] measures the discrepancy between two domains. is computed using the accuracy of a linear domain classifier (source vs target). Since we aim to measure domain invariance, i.e. the inverse of domain discrepancy, we use the inverse of A A-iistance between the source and target datasets for the given feature extractor h,
Yos (D., D.) - d D.)
1
(2)
2
Here, .)....) denotes A-distance computed using feature outputs of h. Note that 0 < d.(.,.) <
3

---

3.2 Balancing negative-transfer-iskk and fomain-invarianee
Given that UniDA is highly susceptible to negative- due to the category-shift problem, we analyze the regative-transfeerriikk at different layers of the deep network. Here, we consider that shallow layers encode low-level visual features like edges, corners, etc. while deeper layers encode more abstract, class-specific features [50]. In the context of UniDA, the feature space of the deeper layers for source and target would be more difficult to align due to the disjointness of the source and target label sets. We empirically observe the same, i.e. NTR increases (solid blue curve in Fig. 2). This suggests that adaptation should be performed at a shallower layer.
go deeper in the model
While we have considered the category-shift problem to understand where to perform the adaptation, we cannot overlook the domain-shift problem. Most DA works [12] perform adapta- tion at deeper layers, which tend to learn increas- ingly more domain-specific features [40, 20]. The higher capacity of the deeper layers leads to unregularized Noomain-specife learning. We also empirically observe that DIS (inverse of domain-specificctty) decreases for the deeper lay- (solid pink curve in Fig. 2). This suggests that adaptation should be performed at a deeper layer to encourage -anainninvariance.
Domain-invann invariance vs negative-transfer tradeoff
Source
CNN
domain
domain
h
Rs
h4
NTR (negative- transfer-risk)
50
70
NTR
Post- adapt
DIS
30
50
NTR
DIS (domain-
invariance-
Layer
30
score)
Red
Rec
Rest
Thus, the problems of domain-shift and category-shift are contradictory in suggesting adaptation at a deeper and shallower layer re- spectively. In other words, there is tradeoff between DIS and NTR (Fig. 2) where reducing the Seeative-transfr--ris negatively affects the domain-invariance and vice versa. Thus, define criterion for minimal negative-transfe and maximal
Figure 2: creases in deeper layers while domain-invariane (DIS) (f) decreases. Thus, we adapt at mid-level layer, where NTR reduces and DIS increases.
Negative-transee transfer-rissk (NTR) (4) in-
Definition 1. (Optimal tradcoff between negative-transfer and domain-invarianee Consider that adaptation is performed at the d layer of the backbone h (let iú, denote the backbone upto th layer). Then adaptation at hr, will encounter minimal negative-transfe and exhibit maximum domain-inw invariance if with at least (1 5) probability, (Wix(h)) does not exceed Sn by more than En. and yDus (hz) exceeds Ça by no less than Ed. i.e.,
P((TNT((hn) S Snn + &.) (Tous(hi)) 2 Sa -a)) 1
Thus, an optimal tradeoff requires NTR to be less than the threshold G, and DIS to be greater than the threshold Ça simultaneously. Empirically, we find a good tradeoff at mid-level layer (green vertical line in Fig, 2. e.g. Res3 block in ResNet), i.e. a compromise between the contradicting suggestions. Note that we re-use h to represent the backbone upto an optimal layer instead of h2 for simplicity.
Why is low NTR desirable? High NTR implies known and unknown samples are well-separated. However, unsupervised adaptation a higher NTR layer is more susceptible to misalignment between shared and private (unknown) classes because target-private classes get grouped into a single unknown cluster. In contrast, lower NTR feature space can better represent all the different classes without grouping the target-private classes into single unknown cluster (i.e. better intrinsic structure). Hence, alignment in this space would better respect the separations of private classes than higher NTR feature space, which is necessary to avoid misalignment in UniDA. For example, consider "hatchback" (compact car) and "SUV" (large-sized car) as a shared and target-private class, respectively, in an object recognition task. Before adaptation, at a high NTR layer, hatchback and SUV features would be separated as SUV is yet unseen to the source- source-trained model. However, during unsupervised adaptation, the similarities between batchbacks and SUVs may align the single sarget-private cluster (containing SUV features) with the hatchback cluster. Due to this, other target-private classes also become closer to this hatchback class which increases the misalignment. In contrast, at lower NTR layer, the target-privatt classes (including SUV) would not grouped together. Hence, misalignment of hatchback and SUV clusters would not disturb other clusters unlike the higher NTR scenario.

---

3.3
Conceptualizing Bag-of-visual Words (BoW) for UniDA
Now we ask whether a meaningful representation space can be developed at this mid-level layer to effectively mitigate negative-transfer To this end, we draw motivation from traditional Bag-of-visual- Words (BoW) concepts. Consider a vocabulary V = [V1 V2...., D.....,Vg, Vkl where each word protttyoe
V& &R V & {1, ..... K}. i.e. K word-prototy Here, the word- perotpptsssspss are representative of lower-level visual primitives [49] (e.g. SIFT-like features). Under the BoW idea [45], histogram of word-prototype a representation of an image for downstream applications. Through the following insight, we argue that BoW concepts can be leveraged in UniDA, for the problems arising from disjoint label sets.
where each Vy is Na-dimensional vector.
wword-histogram) can be used as
Insight 1. (Suitability of BoW concepts for UniDA) Word-prototypes represent lower-level visual primitives which are largely unaffected by calegory-shifi in the high-level features. Thus, subsidiary closed-set pond-prototype-pppe space alignment assists UniDA with minimal regaaive-transfer-risk (NTR).
Remarks. We conceptually illustrate this in Fig. 1. Usually, the private categories are somewhat related the shared categories. Thus, we hypothesize that set of lower-level visual primitives (or word-prototypes) are capable of representing all categories (even unknown) in the word-protolype space shared across domains and classes. In Fig. 1, before adaptation, different target features are scattered around the word word-prototypes. Then, performing closed-set alignment between the features and word-pree cord-prototypes can better capture the intrinsic strscttr to support UniDA.
3.4 Subsidiary Prototype-space Alignment (SPA)
An obvious question remains: How to realize the closed-set word-prototype-space alignment de- scribed in Insight 1? This is crucial because word-prototypes are abstract concepts which need be explicitly realized. To this end, we propose minor architectural modifications in the backbone.
We insert block v after the backbone h (see Fig. 3). First, 1 x 1-conv. layer converts the Na- dimensional spatial features to K --dimensional spatial features (same spatial size as a 1 x filter is used with stride 1). We interpret its weight matrix (V E R RNNokK) vocabulary containing K number of Na-dimensional word-prototypes. The softmax activation performs soft-quantization of the input features h(r) w.r.t. the word pond-prototypes in V.
Consider (x) E RR, the feature vector at spatial location E (1.2....., .U} (where U is the number of spatial locations in the features) and the vocabulary V [V1, .2...., VK] where Va represents the 1t word-prototype. Then, the soft-quantization (soft equivalent of number of occurrences a word-bistogram) at a spatial location for the th word- prot-pytttyppeeee Vı is,
fn
Ni
M()) HxW X.
V
d(z) HxWx K
vxx) Wondshistogram
(VT)(())) sof-quannization
GAP
(x))
(0)())) En hxp(vx.""(x)
(4)
BoW-inspired architecture: 1 1-conv. layer V (vocabulary) and softmax a soft-quantizes the features in terms of word-prototypes in V.
Figure 3:
Following this, we employ the remaining Res4-like conv-layers block (as Res3 was chosen for adaptation in Fig. 2) and global average pooling (GAP) to obtain feature vector (b(r) E R from the spatially dense features o(x). Thus, repurpose simple x 1 conv. layer to implement soft word-quantization with the layer weights as the word- word-prototypes. Now, we introduce a prototype- alignment-score (PAS) to further motivate the effectiveness of BoW concepts.
Prototpe-Alignnen--Sorr (PAS). Consider the backbone feature vector h²(x)) at spatial location u. We compute PAS, YPAs, as k-means loss i.e. distance of h. (I)) to the closest word- word-prototype in V.
(Pas(h" V) = min Ecos (x), vk)
(5)
V.EV
Here, foor denotes cosine-distance Intuitively, "PAS indicates how close the feature vector at u is to one of the word-prototy in V. Since ()"(2)) is a word-histogram (softmax probabilities), a higher TPAs indicates that the closest prototype would have much higher contribution in the hord-histogram than the others i.e. sparser word word-histogram. Based on PAS, we arrive at the following insight.
5

---

A. Invariance of Word-histogram under cropping
C. Pretext dataset extraction
Class-7
Class-2
Class-3
Dataset
Each crop yields a similar word-hist,
Image
samples
Crop-
level
Word-
hist.
use Mins no. of instances for -suffllffiing (at least one crop from each of the Mins instances)
B. Forming word-related self-supervised pretext task
#inst.
#inst.
2
#inst.
Vins
U11
Grid-
Entropy of word-hist increases with #inst.
Vins
M/ms =2
Vmns N
shuffled
images
Image-
level
Class-1
Class-2
Class-3
Class-4
Word-
N
Grid -Siee (4 here)
hist.
gs
Figure 4: A. Different crops of an image yield similar word-histogramy B. The word-histogram
self-entropy of a grid-shuffled image increases with the number of instances contributing the crops.
C. Pretext samples are created by grid-shuffling of crops from yins (pretext label) no. of images.
(Encouraging Prototype-Alignnen for UniDA) Since better prototype-de cotpoppe-algnmmen implies sparser historiissss,mss, the prrr-histogrum-speee spacccu 6(x) would better represent the intrinsic class structure (including private classes). Thus, by encouraging higher PAS, DA at word- word-histogram level would exhibit lower negative-Irannferr
Insight 2.
as word-prototyper are common across domains.
Remarks. Consider Fig. as an example. The shallower layers would extract generic shapes like rectangles, circles, lines, etc. while deeper layers would extract semantic shapes like windows, arms, chassis, etc. Intuitivelyy the hord-histogram space at generic- shaeeie-s cannot be sparse for the object recogition task while sparsity is desirable at deeper layers. Hence, we seek objectives that ensure word word-histogram sparsity at sufficiently high semantic level, catering to the UniDA problem. With this, the pre-classifier features better capture the class-level intrinsic structure that improves UniDA performance. Note that a good intrinsic structure refers to scenario where individual classes, including private classes, are well clustered in the feature space.
Insight 2 encourages sparser word-histograms and a naive way to enforce this would be a self-entropy minimization objective on the word hirr-histogram vectors d d' (x) V u. Intuitively, this objective would increase the contributions of the closest word-prototype in o² (r) i.e. increase the PAS. Formally,
min E N DED
E [C where C = - (x) log(o"
(6)
However, the self-entropy objective susceptible to word-level misalignment due to a lack of constraints. For example, different classes may be mapped to the same word-prototype, or different same-class samples may be mapped to distinct word-prototypes.. As the target domain is unsupervised with unseen private classes and the word-prototype concepts are abstract, it very difficult develop explicit constraints. Thus, we look for implicit constraints via self-supervision to enforce Insight 2.
3.5 Bo W-bssed pretext task
We seek word-related prior property that be used to encourage prototype-aligmment and better word-prototypes V through a self-supervised pretext task. We hypothesize that different crops from the same image would yield similar word-histograms (Fig. 4A) assuming that all crops are extracted with some part of the object inside the crop. This is reasonable as long as partially of-mmfmm crops are not used. Based on this, we provide the following insight formulate our pretext task.
Insight 3. (Word-histggral self-entropy prior in grid-shuffled crops) Consider a grid-shuffled image where crops from Yuus distinct instances assembled (Fig. 4B). The word- hind-hisstoramm self- entropy of the grid-shuffled image, say H He(ym)), would increase with the number of distinct instances Yins involved in the construction of the shufffeffledd image, assuming constant grid size Ngs-
He (ymns +1 He (yns) V Vius
(7)

---

Remarks. Intuitively, combining crops from more number of distinct instances would increase the contributions of distinct word-prow prod-ppot (Fig. 4B), thereby increasing word-histogram entropy. This behavior would be consistent when distinct instances also come from distinct task categories, as it ensures minimal overlap of word-proo pord-prototypes among instances. The assumption of distinct categories for Vins instances is reasonable when goal task has large number of categories, which usually holds in common DA benchmarks for object recognition. On the other hand, crops from the same image would only affect small set of the same word-proootyy marginally affected. While the actual word-histograms may vary with the instances used, the entropy (ymss for given number of instances Yins would lie in small range.
and word-histogram entropy can only be
The pretext task also encourages better prototype-aligmmee bility of different pretext classes would improve with sparser word-histograms. Concretely, crops from multiple instances (in grid-shufling) would each have few distinct word-prototypes with significant contributions and Insight 3 would be better supported. Then, the pretext classes (Fig. 4C) can be easily separated by word hirt-histogram entropy. Note that we use a learnable classifier since these intuitions may not hold at the start of training but pretext objectives can still guide the training.
i.e. higher PAS. This is because separa-
The pretext task helps avoid word-level misalignment, i.e. cases where different classes are aligned with the same word-prototype or different samples of a class are aligned to distinct word-prototypes. Consider Fig. 4A with the worst-case of misalignment where all classes (class-1, class-2, class-3) are represented by the same word-histogne be identical for any no. of instances used for patch- shufh-sh and the pretext task of identifying no. of instances would fail. The above example shows, similar to proof by contradiction, that the pretext task cannot allow word-level misalignment as misalignment would hurt the pretext task performance.
Then, in Fig. 4B, the image-level word- word-histograms would
Based on Insight 3, we construct an entropy-bin for each number of instances for a novel pretext task of entropy-bin classification. Concretely, a pretext classifier fa fa:I Cn operates the output of (Fig. 3) and is trained to predict yisss of input grid-shuffled images.
E {1, 21,2,... 3
Procurement of pretext-task samples. We illustrate this process in Fig. 4C. The same process is followed separately for both source dataset D, and target dataset D, where goal-task category labels are not required. First, the number distinct instances yins is sampled from a uniform distribution U(1, Ng.)) and serves as the pretext-label. Next, a batch of Yim. instances is sampled from (say) the source dataset D,. Now, crops are sampled randomly from the yim instances such that each instance contributes at least one crop. Finally, these crops randomly assembled into the grid-shuffled image denoted by Va.n- The same process can be performed for the target dataset to obtain Lt,n- In summary, a source-pretext dataset Da,n {< Mins) Fa,n X, yins E Cn and target-pretext ,ins
dataset Dr,,. = {(Ka,n Vins) Ji,n E X, Vins E } are extracted.
Training algorithm. We aim to demonstrate that our proposed approach is complementary to exist- ing UniDA methods. Thus, simply add our ar- chitecture modification (i.e. Y in Fig. 3) and the pretext classifier head fn while keeping the other
Goal task iteration
horb
fg
J
(UniDA-Algo
Øt
shared
Vt
Lem
components from an existing method like [36, 23].
Pretext task iteration
Consider existing UniDA training algorithm de-
Lyno
noted by UniDA-Algof (D., D.) that trains the back-
hoab
Sn
Namm
bone i and goal-classifier ʃg R.K + C. Ad- ditionally, we introduce the pretext-task objectives Ca,== Loe (fn, o.(..n) 3ius) and Ci,n (defined simi- larly) for the source and target data respectively (Fig. 5). Here, Lce denotes the standard cross-entropy loss. Formally, the overall objective is,
Ztt
Mt.n
Lr,re
Figure 5: Data samples and objectives for UniDA and pretext task training from Eq. 8.
min J (UniDA-Algo(D, De) + min R.fa
E D...
Len +
E Li. Di.
min h
E D,UD,
E
C
(8)
Here, J.) represents the objective or loss function and the third term is borrowed from Eq. 6. We update only the backbone h and goal-classifier fg using UniDA-Alge while the backbone along with the word-prototype layer s and pretext-classifier fn are updated via the pretext-task objectives. Intuitively, the word-prototypes are learnt only through the pretext-task as they are the implicit constraints, discussed under Insight 2.
7

---

Table 1: Universal DA (UniDA) on Office-31 and DomainNet benchmarks with HOS metric.
Office-31
DomainNet
Method
A-D
A-W
D
W
W-D
D->
W-A
Avg
P-R
R-P
P-S
S-P
R-S
S-R
Avg
UAN [48]
58.6
59.7
70.6
60.1
71.4
60.3
63.5
41.9
43.6
39.1
39.0
38.7
43.7
41.0
CMU [10]
67.3
68.1
79.3
71.4
80.4
72.2
73.1
50.8
52.2
45.1
44.8
45.6
51.0
48.3
ROS [3]
71.3
71.4
94.6
81.0
95.3
79.2
82.1
20.5
36.9
30.0
19,9
28.7
23.2
26.5
DANCE [38]
71.5
78.6
91.4
79.9
87.9
72.2
80.3
38.8
48.1
43.8
39.4
43.8
20.9
39.1
HUAN [46]
71.5
79.4
81.5
80.7
81.0
83.0
79.5
USFDA [16]
79.8
85.5
90.6
83.2
88.7
81.2
84.8
Zhu et al. [53]
86.1
83.2
89.8
88.0
86.7
86.6
86.7
OVANet [36]
79.4
85.8
95.4
80.1
94.3
84.0
86.5
56.0
51.7
47.1
47.4
44.9
57.2
50.7
SVA
80.9
85.4
92.3
82.5
97.5
82.5
86.9
61.1
51.7
47.6
48.7
45.1
58.9
52.2
DCC [23]
78.5
88.5
79.3
70.2
88.6
75.9
80.2
56.9
50.3
43.7
44.9
43.3
56.2
49.2
DCC+SPA
83.8
90.4
90.5
83.1
88.6
86.5
87.2
59.1
52.7
47.6
45.4
46.9
56.7
51.4
Table 2: Universal DA (UniDA) on Office-Homme benchmark with HOS metric.
Office-Home
Method
Ar-Cl
Ar-Pr Ar-Rw Cl-Ar Cl-Pr Cl-Rw Pr-Ar Pr-C Pr-Rw Rw-Ar Rw-CI Rw-Pr
Avg
[48]
51.6
51.7
54.3
61.7
57.6
61.9
50.4
47.6
61.5
52.6
65.2
56.6
CMU [10]
56.0
56.9
59.2
67.0
64.3
67.8
54.7
51.1
66.4
68.2
57.9
69.7
61.6
I-UAN [46]
54.1
63.1
65.2
70.5
68.3
73,2
61.9
51.8
63.8
69.8
55.6
70.7
64.0
ROS [3]
54.0
77.7
85.3
62.1
71.0
76.4
68.8
$2.4
83.2
71.6
57.8
79.2
70.0
OVANet [36]
62.8
75.6
78.6
70.7
68.8
75.0
71.3
58.6
80.5
76.1
64.1
78.9
71.8
SPA
62.0
77.7
86.3
70.0
70.1
70.0
58,8
82.5
76.8
64.0
80.5
73.2
DCC [23]
58.0
54.1
58.0
74.6
70.6
77.5
64.3
73.6
74.9
81.0
75.1
80.4
70.2
DCC+SPA
59.3
79.5
81.5
74.7
71.7
82.0
68.0
74.7
75.8
74.5
75.8
81.3
74.9
Inference. We discard the pretext-classifier f. and use only the goal-classifier fg at inference time. For fair comparison, we use the same known-unknown demarcation algorithm as MhiDA-Algas
Experiments
Dataset. We report results on three different benchmarks. Office-31 [35] contains three domains: DSLR (D), Amazon (A), and Webcam (W). Office-Home [42] is more difficult benchmark, with 65 classes and domains, Artistic (Ar), Clipart (CI), Product (Pr), and Real-World (Rw). DomainNet [32] is the largest DA benchmark and the most challenging due to highly diverse domains and huge class-imbalance. Following [10], we use three subsets, namely Painting (P), Real (R), and Sketch (S).
Evaluation. We report the mean results of three runs for each experiment. Following [36, 23], target-private classes are grouped into a single unknown class for both Open-Set and UniDA. We report H-score metric (HOS), i.e. the harmonic mean of accuracy of shared and target-private samples.
Implementation Details. We
recent prior arts, OVANet [36] and DCC [23], separately as UniDA-Algo (Fig. 5). We initialize ResNet50 with ImageNet-pretrained weights and retain other hyperparameters from the original baseline (UniDA-Algo). Unless otherwise mentioned, UniDA-Algo will be OVANet (for most analysis experiments). We also keep the optimizers, learning rates, and schedulers for both goal task iterations and pretext task iterations same as the baseline. We follow DCC [23] for the dataset and shared-private class splits. See Suppl. for complete details.
4.1 Comparison with prior arts
Open-Set DA. The benchmark comparisons for OSDA are presented in Table 3 and for Office- Home and Office-31 benchmarks respectively. Our method surpasses all current methods, even those tailored for OSDA [3, 37, 24]. With a 2.2% H-score gain over OVANet [36] and a 4.6% gain over DCC [23], consistently outperform all OSDA baselines for Office-Home. Similarly, our SPA on top of DCC and OVANet improves on Office-31 by 6.4% and 1.3% over the baselines. Overall, we achieve superior balance between shared class categorization and private sample identification.
UniDA. On the Office-31 benchmark (Table l), the proposed approach outperforms all other methods in terms of H-score. We improve upon the earlier state-of-the-ar methods, DCC [23] and OVANet [36] by 7.0% and 0.4% respectively, again demonstrating a superior balance between shared and private sample identification. Office-Home (Table 2) is more difficult benchmark, with more

---

Table Open-Set DA (OSDA) on Office-Home benchmark with HOS metric.
Office-Homee
Method
Pr Ar
CI Ar
Rw
Ar
-w Rw-Ar
Rw
Avg
STA [24]
55.8
54.0
68.3
57.4
60.4
66.8
61.9
53.2
69.5
67.1
54.5
64.5
61.1
OSBP [37]
55.1
65.2
64.7
70.6
63.2
53.2
73.9
66.7
54.5
72.3
64.7
GDA [27]
59.9
67.4
74.5
59.5
66.8
70.8
60.7
58.4
70.9
65.6
61.3
73.8
65.8
ROS [3]
60.1
69.3
76.5
58.9
65.2
68.6
60.6
56.3
74,4
68.8
60,4
75.7
66.2
OVANet [36]
58.4
66.3
69.3
60.3
65.1
67.2
58.8
52.4
68.7
67.6
58.6
66.6
63.3
SPA
59.4
67.9
75.3
62.7
65.6
70.2
61.4
54.2
71.3
68.3
58.3
71.9
65.5
DCC [23]
52.9
49.8
67.0
59.5
52.8
64.0
56.0
76,9
62.7
63.0
DCC+SPA
55.2
76.0
79.5
56.2
66.2
74.0
64.2
52.5
72.2
63.8
74.4
77.0
67.6
Table 4:
Open-Set DA (OSDA) Office-31 benchmark
Table 5: components Office-Home. Method
Ablation study of our
with HOS metric.
Office-31
OSDA UniDA
Method
A-D A-
D-W
W-D
D-A
W-A
Avg
OVANet [36]
63.3
71.8
architecture mod
63.6
71.8
ROS [3]
82.1
82.4
96.0
77.9
99.7
77.2
85.9
arch-mod + Lem
64.1
72.2
CMU [10]
70.5
71.6
81.2
80.2
70.8
70.8
74.2
our pretext task
64.9
72.8
DANCE [38]
74.7
82.0
82.1
68.0
82.5
52.2
73.6
all (SPA)
65.5
73.2
Inheritune [17]
81.4
78.0
92.2
83.1
99.7
91.3
87.6
OSHT-SC [9]
92.4
91.3
95.2
90.8
96.0
89.6
92.5
DCC [23]
63.0
70.2
OVANet [36]
84.9
89.5
93.7
89.7
85.8
88.5
88.7
architecture mod
63.5
70.9
OVANet+SPA
89.7
90.2
96.9
82.6
99.8
86.8
91.0
arch-mod fem.
64.2
71.5
DCC [23]
87.1
85.5
91.2
85.5
87.1
84.4
86.8
our pretext task
66.1
73.6
DCC+SPA
91.7
92.3
96.0
90.0
97.4
91.5
93.2
(SPA)
67.6
74.9
private classes than the shared classes (55 vs. 10). Our approach exhibits a stronger capability for the separation of shared and private classes in this extreme circumstance, benefiting from proposed pretext task. On the Office-Home benchmark, our method combined with DCC yields 4.7% improvement in H-score and an additional 1.4% gain with OVANet. On the large-scale scale DomainNet dataset (Table 1), our strategy improves OVANet [36] by 1.5%. This shows that our approach outperforms prior arts in variety of settings, Lee. varying degrees of openness.
4.2 Discussion
a) Ablation study. We perform a thorough ablation study on Office-Home for the components of our approach (Table 5). First, since our architecture modification presents a small increase in computation, we demonstrate that simply using the arch. mod. only marginally improves the performance (0.2% over OVANet and 0.6% over DCC). Further, simply adding the word-histogram self-entropy loss Lem (Eq. 6) with the arch. mod. also yields fairly low improvements over the baseline (0.4% over OVANet and 1.2% over DCC). Next, we assess the improvement from the proposed pretext task (but without Dem) and observe gains of 1.1% over OVANet and 3.2% over DCC. Finally, including fem with the pretext task gives further gains of 0.5% over OVANet and 1.4% over DCC. Thus, the arch. mod. and fem independently give marginal gains. However, they give significant improvements combined with our pretext task, underlining the importance of every component.
b) Evaluating clustering of target-private classes. To further support Insight 2, i.e. private class samples are better clustered with our approach, we apply linear evaluation protocol for the privet-privae classes on the word-prototype features Φ(z). Here, use the labels of target- analysis) to train linear classifier on the frozen features from ((x) and compute the accuracy for target-private classes (Table 6). We observe a significant gain (+5.6%) over the baseline
Table 6: Target-private accuracy with linear evaluation on frozen o(x) for UniDA on Office-Home,
Method
Ar-CII CI-+Pr
classes (only for this
OVANet arch-mod
70.7
80.8
+ Ours
74.9
87.8
OVANet, which indicates that the target-private classes are better clustered with SPA.
c) Correlation between UniDA performance, pretext task performance, PAS and NTR. For UniDA Office-Home, we study the correlation between goal task performance (HOS), pretext task performance, negative-transfer-rikk (NTR) and protorype-aligmment
(PAS) (averaged over D,) in Fig. 6C. We observe that goal task performance is positively correlated with both pretext task

---

Effect of varying degree of openness
Effect of varying grid-size gz
Correlation performance NTR & PAS
60
86
80
Cow
40
80
Alc
30
73
20
HOS
75
72
0.14-
88
190)
70
71
0.10
58
10
2x2
4s4
Ex6
BxB
0.06
100
200
300
400
45
No. target-private classes
Grld Size N
Iterations
Homie, A. We evaluate the effect of varying openness ie. no. of target-private classes (Sec. 4.2d), B. We report the effect of varying grid-size Ngs (Sec. 4.2c), C. We study the correlation of goal and pretext task performance with PAS and NTR (Sec. 4.2b). Dashed curves represent experiment performed without our pretext task objectives (as a baseline).
Figure 6: For UniDA
performance and PAS, as in Insight Further, goal task performance is negatively correlated with NTR which supports Insight 1. Further, compared to the baseline OVANet [36] (dashed curves), our OVANet+SPA (solid curves) achieves better and faster convergence (~~000 vs. ~300 iterations). Further, we also observe that PAS does not change much when our pretext task is not employed.
<) Comparison with other pretext tasks. We compare the effectiveness of our word-related pretext task, for Open-Set DA and Universal DA on Office-Home, with existing pretext tasks in Table 7. Note that we include our EM loss and architecture modification with each of the pretext tasks for a fair comparison. First, we compare with dense-output based pretext tasks like colorization [52, 22] and inpainting [31]. We observe marginal or no improvements with these tasks because the output spaces of these tasks are usually Nomain--eeendedd of rainy scenes must consider rain, a domain-specifee factor), which hinders the adaptation. Next, we consider non-dense classification tasks like rotation [26, 15], jigsaw [5] and patch-
Table 7: Comparisons with other pretext tasks on Office-Home.
Method
OSDA UniDA
OVANet arch-mod
63.6
71.8
+ colorization
63.5
71.8
+ inpaintinn
63.7
72.0
+ jigsaw
63.8
72.0
(e.g. inpainting
+ patch-loc
64.0
72.1
+ rotation
64.3
72.4
Ours
65.5
73.2
location [41]. While these tasks achieve some marginal gains, our proposed pretext task outperforms
them as it effectively aids UniDA and Open Set DA by encouraging woototye--aligmenn (Insight 2).
d) Effect of varying degree of openness. Fig. 6A studies the effect of altering the degree of openness for UniDA in Office-Home, Following [36], we report the average results over five scenarios to cover variety of domains while varying the amount of unknown classes. Our approach exhibits a low sensitivity to the degree of openness and consistently outperforms other baselines.
Effect of grid-size Ngs used in grid-shufiling. For UniDA Office-Home, we report sensitivity analysis for grid-size Ng. (Fig. 6B) which controls the number of entropy-bins iee. number of pretext task classes. We observe marginal decrease in the performance the grid-size increases, but it outperforms the baseline (OVANet) across a wide range of grid-sizes (2x2 to 6x6). Mitsuzumi et al. [27] show that, as grid-size increases beyond 4x4, shuffling the grid patches significantly alters domain information. Thus, beyond 4x4, performance drops as more domains are introduced which make adaptation difficult. Another reason is that the pretext task itself becomes more difficult as the grid-size is increased. Based on this, we choose grid-size 2x2 (Ng==4) =4) for our experiments.
Conclusion
In this work. we address the problem of Universal DA via closed-set Subsidiary Prototype-space Alignment (SPA). First, we uncover a tradeoff between negative-transe transeerrrsff and at different layers of deep network. While balance can be struck mid-level layer, we draw motivation from Bag-of-visual-Wont Words (BoW) to introduce explicit word- word-prototypes followed by word-histogran based classification. We realize the closed-set SPA through novel word-hister histogram based pretext task operating in parallel with UniDA objectives. Building on top of existing UniDA works, we achieve state-of-the-art results three UniDA and Open-Set DA benchmarks.
Acknowledgements. This work was supported by MeitY (Ministry of Electronics and Information Technology) project (No. 4(T)), Govt. of India
10

---

References
[1] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurIPS, 2006. 2,3
[2] Marco San Biagio, Loris Bazzani, Marco Cristani, and Vittorio Murino. Weighted bag of visual words for object recognition. In ICIP, 2014. 3
[3] Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In ECCV, 2020. 3, 8,9
[4] Zhangjie Cao. Kaichao You. Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In CVPR, 2019. 1
[5] Fabio Maria Carlucci, Antonio D' Innocentee, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019. 10
[6] Yi-Hsin Chen, Yu Chen, Yu- Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun. No more discrimination: Cross city adaptation of road scene segmenters. In ICCV, 2017.
[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiole. The cityscapes datasct for semantic urban scene understanding. In CVPR, 2016. 1
[8] Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang. Open set domain adaptation: Theoretical bound and algorithm. IEFE transactions neural networks and learning systems, 42010)2430--432, 2020. 3
[9] Zeyu Feng, Chang Xu, and Dacheng Tao. Open-set hypothesis transfer with semantic consistency. IEEE Transactions Image Processing, 30:6473-6444 2021.
[10] Bo Fu, Zhangjic Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In ECCV, 2020. 3, 8,
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICMI.,
2015. 1
[12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversenn adversarial training of neural networks. The Journal of Machine Learning Research,
(00:20996220000 2016. 1, 4
[13] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Perez, and Matthicu Cord. Learning representa- tions by predicting bags of visual words. In CVPR, 2020. 3
[14] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthicu Cord, and Patrick Perez. OBoW: Online vig-of--isuullworde generation for self supervied learning. In CVPR, 2021. 3
[15] Xu Jiaolong, Xiao Liang, and Antonio M. López. Self supervised domain adaptation for computer vision tasks. IEEE Access, 7:156694- 1567066, 2019. 10
[16] Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu, Universal source-free domain adaptation. In CVPR, 2020. 8
[17] Jogendra Nath Kundu, Naveen Venkat, Amhareesh Revanur, Rahul and R. Venkatesh Babu. Towards inheritable models for open-set domain adaptation. In CVPR, 2020. 1,9
[18] Jogendra Nath Kundu, Akshay Kulkarni, Amit Singh, Varun Jampani, and R. Venkatesh Babu, Generalize then adapt: Source-free domain adaptive semantic segmentation. In ICCV, 2021.
[19] Jogendra Nath Kundu, Suvaansh Bhambri, Akshay Kulkarni, Hiran Sarkar, Varun Jampani, and R. Venkatesh Babu. Concurrent subsidiary supervision for unsupervised source-free domain adapta- tion. In ECCV, 2022.
[20] Jogendra Nath Kundu, Akshay Kulkarni, Suvaansh Bhambri, Varun Jampani, and Venkatesh Babu Radhakr- ishnan. Amplitude spectrum transformation for open compound domain adaptive semantic segmentation. In AAAI, 2022. 4
[21] Jogendra Nath Kundu, Akshay Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Kulkarni, Varun Jampani, and R. Venkatesh Babu. Balancing discriminability and transferability for source- free domain adaptation. In ICML, 2022.

---

[22] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017. 10
[23] Guangrui Li. Guoliang Kang. Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for universal domain adaptation. CVPR, 2021. 3, 8,9
[24] Hong Liu, Zhangjie Cao, Mingsheng Long. Jianmin Wang. and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In CVPR, 2019, 8,
[25] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Newr/PS, 2018.
[26] Samarth Mishra, Kate Sacnko, and Venkatesh Saligrama. Surprisingly simple semi-supervised domain adaptation with pretraining and consistency. In BMVC, 2021.
10
[27] Yu Mitsuzumi, Go Irie, Daiki Ikami, and Takashi Shibata. Generalized domain adaptation. In CVPR, 2021.
9, 10
[28] Poojan Oza and Vishal M Patel. C2AE: Class conditioned auto-encoder for open-set recognition. In CVPR, 2019. 3
[29] Yingwei Pan, Ting Yao, Yehao Li, Chong-Woi Wah Ngo, and Tao Mei. Exploring category-agnostic clusters for open-set domain adaptation. In CVPR, 2020. 3
[30] Pau Panareda Busto and Juergen Gall, Open set domain adaptation. In ICCV, 2017.
[31] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders:
Feature learning by inpainting. In CVPR, 2016. 10
[32] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019. 8
[33] Pramuditha Perera, Vlad Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel. Generatieediisrimmiaaive feature representations for open-set recognition. In CVPR, 2020. 3
[34] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and R. Venkatesh Babu. A closer look at smoothness in domain adversarial training. In ICML, 2022.
[35] Kate Sacnko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010. 8
[36] Kuniaki Saito and Kate Saenko. OVANet: One-vs-all network for universal domain adaptation. In ICCV, 2021. 7, 9, 10
[37] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV, 2018. 3, 8, 9
[38] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurlPS, 2020. 3,
[39] Ravi Shekhar and C.V. Jawahar. Word image retrieval using bag of visual words. In IAPR Workshop, 2012.
[40] Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Sue Yeon Chung. On the geometry of generalization and memorization in deep neural networks. In ICLR, 2021.
[41] Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros, Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv:1909. 1885, 2019. 10
[42] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[43] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In CVPR, 2019. 3
[44] Sheng Xu, Tao Fang. Deren Li, and Shiwei Wang. Object classification of aerial images with bag visual words. IEEE Geoscience and Remote Sensing Letters, 7(2):366331 370, 2010. 3
12

---

[45] Jun Yang. Yu-Gang Jiang, Alexander G. Hauptmann, and Chong- Wah Ngo. Evaluating bag-of -iwoooo-vvalwwwwweoe words representations in classification. In MIR, 2007. 5
[46] Yueming Yin, Zhen Yang, Xiaofu Wu, and Haifeng Hu. Pseudo mardin- margin-based based universal domain adaptation. Knowledge-Basee Based Systems, 229:107315, 2021. 8
[47] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Clasiffcctioon-ccoosstttttttt learning for open-set recognition. In CVPR, 2019. 3
[48] Kaichao You, Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Michael Jordan, Universal domain adaptation. In CVPR, 2019. 3,
[49] Junsong Yuan, Ying Wu, and Ming Yang. Discovery of collocation patterns: from visual words to visual phrases. In CVPR, 2007. 5,5
[50] Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[51] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In CVPR, 2018.
[52] Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. ECCV, 2016.
[53] Ronghang Zhu and Sheng Li. Self-supervised universal domain adaptation with adaptive memory separa- tion. ICDM, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix B
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix C
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] It will be available at https: / /ttp/ssttss.eee//ee////e/eeeeeee google. com/view/ cool/viewsppaaun.e... ssia-uida...
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4 and Appendix D.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix E
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.2
4. If you using existing assets (c.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix D.1
(b) Did you mention the license of the assets? [Yes] See Appendix D.I
(c) Did you include any new assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l rou're using/curating? [N/A] We used publicly available standard object recognition datasets that do not contain any personally identifiable information (PII).
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] See answer to 4d.
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xiJvVJFws9o", "reviewer": "Reviewer_5kcu", "summary": "This paper studies the universal domain adaptation problem. Unlike partial DA or open-set DA, UniDA does not require knowing relation between source and target domain label sets. The authors start by analyzing the behaviors of source-trained classifier on target domain data. Specifically, the authors propose a new metric called negative-transfer-risk (NTR). It measures the class-specificity of the features in a specified layer via a shared-vs-unknown binary classifier. The other metric is the inverse of A-distance. Motivated by the analysis, the authors propose the do the adaptation in the mid-level layer of the network. Inspired by the BOW, the authors design a word-prototypes in the selected mid-layer representation, and regularize such a BOW-like representation via a subsidiary prototype-space alignment. To realize SPA, the authors design a novel word-histogram based pretext task with UniDA objectives. The experiments are conducted on three DA benchmarks, and competitive results are reported.", "questions": "\n* As defined in Eq 1, NTR measures the class-specificity of the features in a source-trained specified layer via a shared-vs-unknown binary classifier. So Fig 2 blue solid curve only suggests that in deeper layers (e.g. Res4/FC), it is easier to tell apart the known/unknown given the feature of the source-domain trained network, compared with the feature extracted from the shallow layers. I am not clear why this observation suggests “(line 131: adaptation should be performed at a shallower layer.)”, since the final DA classification task will be performed in the final deep layer.\n\n    Also, I’m not clear if a higher NTR score is better or the other way around. Intuitively, I think higher NTR the better. While when I see the dashed blue curve (post adapt) in Fig 2, it looks that the lower the better. Please clarify.\n\n* Similarly, DIS score only shows the expressive capability of each layer in terms of domain-invariance. Then the conclusion is drawn that the adaptation should be performed in the mid-level layer, which seems contradictory to the existing multi-level adaptation works such as [ref-1].   \n[ref-1] Xie et al., Multi-Level Domain Adaptive Learning for Cross-Domain Detection, 2019\n\n* Please conduct an ablation study of applying SPA on different layers and all layers.\n", "limitations": "No potential negative societal impact identified.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* The studied problem Universal DA is a much practical setting, where much weaker assumption is posed compared with standard, partial or open-set DA.\n\n* The technical detail part are generally well written, such as the design of word-prototype.\n\n* The reported results seem competitive. It shows the benefit by applying SPA to both OVAnet and DCC. \n\nWeakness\n* The introduction and motivation of NTR are unclear to me and not easy to follow. For detailed questions, please see the below question section.\n\n* Experiments. To support the key statement of mid-level adaptation, the authors miss an ablation study of applying SPA on different layers and all layers.\n\n* From Office-31, [49] seems the best existing method. I’d enough the authors to apply SPA to [49] as [32,20] to show the generalizability of SPA.\n\n* The proposed SPA is based on convolution operator. As vision transformer is becoming more popular and powerful, the applicability of the proposed method may be limited to ConvNet only.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TH2NZ5AtnNq", "reviewer": "Reviewer_S37P", "summary": "This paper addresses universal domain adaptation (UniDA) for object recognition: target sample is classified as either one of the \"known\" classes or \"unknown\". This work proposes an add-on strategy that helps improve results of existing approaches, reaching SOTA performance in UniDA.\n\n** Analysis **\nFor UniDA, one wants to increase the domain-invariant score (DIS) between source/target while trying to alleviate the potential negative-transfer risk (NTR) between \"known\" and \"unknown\" classes. However, the two measures are often at odds with each other. Quantifying NTR as known-unknown classification accuracy using entropy threshold (like [17]) and DIS as the $\\mathcal{A}-$distance (like [1]), the paper provides an analysis on the NTR/DIS trade-off of resnet features at different depths. Empirically, feature of the res3-block balances NTR and DIS. Based on this finding, res3-block feature is used throughout experiments.\n\n** Proposal 1 **\nTo make domain alignment possible while avoiding the risk of negative-transfer, the paper proposes to align in the Bag-of-visual-words space. The intuition is that BoW contains visual primitives that are universally shared among known and unknown classes. To this end, res3-block features $h\\in\\mathbb{R}^{H\\times W\\times N_d}$ are passed through a soft-assignment module including a vocabulary matrix $V\\in\\mathbb{R}^{N_d \\times K}$ ($K$ is the vocabulary size) and a soft-quantization operation (Eqn. 4). For both source and target samples, one minimizes the BoW-wise entropy of the corresponding soft-assignment tensor $\\in\\mathbb{R}^{H\\times W \\times K}$. That is to implement the sparsity assumption of the word-histogram space.\n\n** Proposal 2**\nA novel pretext task that help encourage prototype-alignment: a pretext dataset is composed of images stitched by a varying number of random image crops from different images. Pretext task is to predict the number of crops, based on the global word-histogram of the stitched image.\n\n** Experiment **\nOn several benchmarks (Office-31, DomainNet, OfficeHome), the two proposed strategies complement with existing approaches and improve performance, achieving SOTAs in UniDA.", "questions": "- What's unclear to me is the sparsity assumption on the word-histogram space. Could the authors elaborate more the intuitions?\n\n- As discussed in L216-220, there are risks of word-level misalignment when minimizing the BoW-entropy. Could the authors provide some real examples when the misalignment happen? How does the proposed pretext task help mitigate the misalignment risks?", "limitations": "Limitations and potential negative societal impact are given in the supplementary material. I find those adequate.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-written, easy to follow. Most arguments are sound with adequate empirical supports; there are only a few concerns (see below).\n\n- Originality: the two proposed strategies are novel in the context of UniDA. \n\n- Extensive ablation studies validate the proposed strategies and experimental choices.\n\n- Complementary to existing approaches. The combinations with either OVANet [32] or OCC [20] obtain SOTA results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8jxNnJDPad", "reviewer": "Reviewer_6Jjx", "summary": "In this paper, the authors tackle the problem of universal domain adaptation where the source domain consists of fully-labeled images and unknown category images while the target domain consists of unlabeled images of known and potentially unknown categories. To tackle this problem, they first study the trade-off relation between the negative transfer risk (NTR), i.e., class-level misalignment between the shared and private classes, and domain invariance. Motivated by the trade-off between negative transfer risk and domain invariance, they propose to align mid-level features of deep neural networks. To this end, they propose to align bag-of-word style features with self-entropy regularization to encourage the sparser BoW features. They further regularize the model training by adding a pretext task of entropy-bin classification on the grid-shuffled images from multiple instances. Plugged into the existing UniDA methods, the proposed method shows favorable performance on universal and open-set domain adaptation settings on three public benchmarks.", "questions": "Please address all of my concerns in the weaknesses part above. Also, I do have a few more questions for the authors here.\n\nQ1: Why the NTR in Eq (1) does measure negative transfer risk? Basically, it is the accuracy of the pseudo labeling of the shared-private label using entropy thresholding. If we have a high NTR value, we know what are known samples and what are unknown samples. Intuitively it would reduce class-level misalignment.\n\nQ2: In L142-144, the authors claim the domain adaptation should be performed at a deeper layer to encourage domain invariance. However, Fig 2. Shows deeper layers show lower DIS. What is a reasonable conclusion here? Should we more align the deeper layers with lower DIS as they are less domain invariant? Or should we more align the shallower layers with higher DIS as they are domain more promising?\n\nQ3: How do we train a linear domain classifier to measure DIS? \n\nQ4: In Table 6, are the other pretext baselines equipped with the proposed BoW-style architecture modification and L_{em} as well? If not, we need to compare with “our pretext task” baseline in Table 5 (64.9% and 72.8%) instead of full SPA (65.5% and 73.2%).\n\nQ5: In L271-272, why the word prototypes are learnt only through the pretext task? The main task classifier is also attached on top of the word-histogram feature as shown in Fig. 3.\n\n", "limitations": "The authors did not really address the limitations of their work. Please describe what the failure modes of the proposed method are and when the failures happen. They have not discussed the potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper are as follows.\nS1: The problem tackled in this work, universal domain adaptation is challenging and interesting. \nS2: Aligning mid-level features for the task of UniDA seems reasonable. Intuitively, high-level features are often specific to classes, therefore prone to category-shift. In contrast, low-level features are robust to category shift. However, they are prone to domain-shift. Therefore, intuitively mid-level features such as BoW could be a good balance between category-shift and domain-shift.\nS3: The proposed method shows favorable performance on three public benchmarks. The proposed components, the BoW-style architecture, entropy regularization, and pretext task, contribute to the performance.\n\nHowever, this paper has several weaknesses as well.\nW1: My biggest concern is the study on the relation between NTR and DIS (domain invariance score). Fig. 2 shows Res1 block has the lowest NTR and highest DIS. Compared to Res1 block, Res3 block shows higher NTR and lower DIS. If NTR and DIS are the only criteria, why do we need to use Res3 block feature for alignment? Why we cannot align Res2 and Res1 as well? \n\nW2: The proposed architecture modification does not seem to align the mid-level feature only. By minimizing L_{em} we align the mid-level feature. However, minimizing L_{s,n} and L_{t,n} also align the high-level feature as f_n is attached on top of Res4 and GAP. This is a self-contraction with the claim made from Fig. 2.\n\nW3: The improvement from each component is not very surprising. BoW-style mid-level feature + L_{em} only improves 0.4 ~ 0.5 points on top of OVANet. The results make me doubt that the proposed BoW-style feature alignment is really optimal. In addition, the proposed pretext task improves only 0.4 ~ 0.6 points compared to the rotation task.\n\nW4: Presentation quality should be improved. In general, the paper is not straightforward to read. For example, it is hard to understand why NTR in eq. (1) measures negative transfer risk. Another confusion is that do we need to more align the layers with higher DIS or layers with lower DIS.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept