Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation
Authors: Yuxiang Lai
Yi Zhou
Xinghong Liu
Tao Zhou
Venue: ICLR 2024 poster (2024)
Keywords: Transfer Learning
Universal Domain Adaption
Memory-Assisted Network
Sub-Prototype Mining
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose Memory-Assisted Sub-Prototype Mining(MemSPM), which explores the memory mechanism to learn sub-prototypes for improving the model’s adaption performance and interpretability.

Abstract: Universal domain adaptation aims to align the classes and reduce the feature gap between the same category of the source and target domains. The target private category is set as the unknown class during the adaptation process, as it is not included in the source domain. However, most existing methods overlook the intra-class structure within a category, especially in cases where there exists significant concept shift between the samples belonging to the same category. When samples with large concept shift are forced to be pushed together, it may negatively affect the adaptation performance. Moreover, from the interpretability aspect, it is unreasonable to align visual features with significant differences, such as fighter jets and civil aircraft, into the same category. Unfortunately, due to such semantic ambiguity and annotation cost, categories are not always classified in detail, making it difficult for the model to perform precise adaptation. To address these issues, we propose a novel Memory-Assisted Sub-Prototype Mining (MemSPM) method that can learn the differences between samples belonging to the same category and mine sub-classes when there exists significant concept shift between them. By doing so, our model learns a more reasonable feature space that enhances the transferability and reflects the inherent differences among samples annotated as the same category. We evaluate the effectiveness of our MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our method achieves state-of-the-art performance on four benchmarks in most cases.

Published as a conference paper at ICLR 2024
MEMORY-ASSISTED SUB-PROTOTYPE MINING FOR UNIVERSAL DOMAIN ADAPTATION
Yuxiang Lai Yi Zhou 1.2+ Xinghong Liu Tao Zhou
School of Computer Science and Engineering, Southeast University. China
2 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
3 School of Computer Science and Engineering, Nanjing University of Science and Technology, China
ABSTRACT
Universal domain adaptation aims to align the classes and reduce the feature gap between the same category of the source and target domains. The target private category is set as the unknown class during the adaptation process, it is not included the source domain. However, most existing methods overlook the intra-class structure within category, especially in cases where there exists significant concept shift between the samples belonging to the same category. When samples with large concept shifts are forced to be pushed together, may negatively affect the adaptation performance. Moreover, from the interpretability aspect, is unreasonable to align visual features with significant differences, such as fighter jets and civil aircraft, into the same category. Unfortunately. due to such semantic ambiguity and annotation cost, categories are not always classified in detail, making it difficult for the model to perform precise adaptation. To Sub-Prototype Mining
address these issues, we propose a novel Memory-Assisee (MemSPM) method that can learn the differences between samples belonging the same category and mine sub-classes when there exists significant concept shift between them. By doing so, model learns more reasonable feature space that enhances the transferability and reflects the inherent differences among samples annotated as the same category. We evaluate the effectiveness of our MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our method achieves state-of-the-art performance on four benchmarks in most cases.
INTRODUCTION
Unsupervised Domain Adaptation (UDA) (Ganin and Lempitsky, 2015; Kang et al., 2019; Saito et al., 2018; Shu et al., 2018; Chen et al., 2016; Hsu et al., 2015; Kalluri et al., 2022) enables models trained on one dataset to be applied to related but different domains. Traditional UDA assumes shared label space, limiting its applicability in diverse target distributions. Universal Domain Adaptation (UniDA) addresses these limitations by allowing the target domain to have distinct label set. UniDA flexibly classifies target samples belonging to shared classes in the source label set, treating others as "unknown." This approach, not relying on prior knowledge about target label sets, broadens the adaptability of domain-invariant feature learning across diverse domains.
Despite being widely explored, most existing universal domain adaptation methods (Li al., 2021; You et al., 2019; Saito and Saenko, 2021; Saito et al., 2020; Chang et al., 2022; Qu et al., 2023; Chen al., 2022: Liang et al., 2021) overlook the internal structure intrinsically presented within each image category. These methods aim to align the common classes between the source and target domains for adaptation but usually train a model to learn the class "prototype" representing each annotated category. This is particularly controversial when significant concept shifts exist between samples belonging the same category. These differences can lead to sub-optimal feature learning and adaptation if the intra-class structure is neglected during training. Since this type of semantic ambiguity without fine-grained category labels occurs in almost all of the DA benchmarks, all the methods will encounter this issue.
"Correspondence Yi wwoo (OOOOUYYHHOU.SS SZCN @ CMM)).COM)

---

Published as a conference paper at ICLR 2024
(8)
Sub-Cless1
Suas-C
Suh Classl
Sub-Class2
Sub-Class3
Sub Class3
Class Alarm Clock
Class: Airplane
(b)
Class: Alarm Clock
Cancept Shift
Source Domain
Source Domatn
Previous
Ours
Sub Classl
Sub-Class2
1139%
Target Domain
Class1 Sample:
Sub-Class 1:
Sub-Class Class 2:-
Class2 Sample:
Figure I: Illustration of our motivation. (a) Examples of concept shift and intra-class diversity in DA benchmarks. For the class of alarm clocks, we find that digital clocks, pointer clocks, and alarm bells should be set in different sub-classes. For the class of airplane, we find that images containing more than one plane, single jetliner, and turboprop aircraft should be differently treated for adaptation. (b) Previous methods utilize one-hot labels to guide classifying without considering the intra-class distinction. Consequently, the model forces all samples from the same class to converge towards single center, disregarding the diversity in the class. Our method clusters samples with large intra-class differences into separate sub-classes, providing a more accurate representation. (c) During domain adaptation by design, the samples in the target domain also be aligned near the sub-class centers with similar features rather than just the class centers determined by labels.
In this paper, our objective is to propose a method that learns detailed intra-class distinctions and extracts sub-prototypes' enhance alignment and adaptation. These sub-prototypes represent further subdivisions within each category-level prototype, corresponding to the 'sub-classes' of the annotated categories. Our approach revolves around employing a learnable memory structure to derive sub- prototypes for their respective sub-classes. This can optimize the construction and refinement of the feature space, bolstering the classifier's 's ability to distinguish class-wise relationships and improve the model's transferability across domains. As illustrated in Figure 1, the samples that annotated as the same category often exhibit significant intra-class differences. However, previous work mainly forced them to align together for adaptation. Therefore, these methods more likely classify unknown classes into known classes incorrectly. Moreover, features of different sub-classes still have gaps in the feature space, making it unreasonable align samples from distinct sub-classes, both from human perspectives and in the feature space. Aligning target domain samples at the sub-class level with source domain samples mitigates the drawback of aligning significantly different samples, making adaptation more reasonable.
Our proposed approach, named memory-assisted sub-prototype mining (MemSPM), is inspired by the memory mechanism works (Gong et al., 2019; Chen et al., 2018; 8; Sukhbaatar et al., 2015; Rae al., 2016). In our approach, the memory generates sub-prototypes that embody sub-classes learned from the source domain. During testing of the target samples, the encoder produces embedding that is compared to source domain sub-prototypes learned in the memory. Subsequently, an embedding for the query sample is generated through weighted gub-prototype sampling in the memory. This results in reduced domain shift before the embedding is passed to the classifier. Our proposal of mining sub- sub-prototypes, which are learned from the source domain memory, improves the universal domain adaptation performance by promoting more refined visual concept alignment.
MemSPM approach has been evaluated on four benchmark datasets (Office-31 (Saenko et al., 2010), Office-Home (Venkateswara et al., 2017), VisDA (Peng et al., 2017), and Domain-Net (Peng et al., 2019)), under various category shift scenarios, including PDA, OSDA, and UniDA. Our MemSPM
2

---

Published as a conference paper at ICLR 2024
method achieves state-of-the performance in most cases. Moreover, designed visualization module for the sub-prototype learned by our memory to demonstrate the interpretability of MemSPM. Our contributions can be highlighted as follows:
We study UniDA problem from a new aspect, focusing on the negative impacts of ignoring intra-class structures within the same category when using one-hot labels.
We propose Memory-Assised Sub-Prototype Meminn(MMmSPMM), which explores the mem- ory mechanism to learn aub-prototypes for improving the model's 's adaption performance and interpretability. Meanwhile, visualizations reveal the sub-prototypes stored memory, which demonstrate the interpretability of the MemSPM approach.
Extensive experiments on four benchmarks verify the superior performance of our proposed MemSPM compared with previous works.
2
RELATED WORK
Universal Domain Adaptation (UniDA). You et al. (2019) proposed Universal Adaptation Network (UAN) deal with the UniDA setting that the label set of the target domain is unknown. Li al. (2021) proposed Domain Consensus Clustering to differentiate the private classes rather than treat the unknown classes as one class. Saito and Saenko (2021) suggested that using the minimum inter-class distance in the source domain as a threshold can be an effective approach for distinguishing between "known" and "unknown" samples the target domain. However, most existing methods (Li et al., 2021; You et al., 2019; Saito and Saenko, 2021; ; Saito et al., 2020; Chang et al., 2022; Qu et al., 2023; Chen al., 2022; Liang et al., 2021; Liu et al., 2023; Zhou et al., 2022) overlook the intra-class distinction within the same category, especially there exists significant concept shift in same category.
Concept of Prototypes. prior research (Kundu et al., 2022; Liu et al., 2022), prototypes have been discussed, but they differ from our MemSPM. First, in Kundu et al. (2022), subsidiary prototypes lack complete semantic knowledge and cannot address concept shifts within categories. In contrast, our bob-prototype can represent sub-class within category. Second, the purpose of Liu et al. (2022) is distinct from MemSPM. They aim to differentiate unknown classes. In contrast, MemSPM identifies sub-classes within category. More details in Appendix C.
PROPOSED METHODS
3.1 PRELIMINARIES
In unsupervised domain adaptation, we are provided with labeled source samples D ====0 (xf v)) )
and unlabeled target samples D' { As the label set for each domain in UniDA setting may not be identical, we use C, and C, to represent label sets for the two domains, respectively. Then, we denote C' = C2 n C2 as the common label set. Ć., Ce are denoted as the private label sets of the source domain and target domain, respectively. We aim to train model on De and D' to classify target samples into |C| classes, where private samples are treated as unknown classes.
Our method aims to address the issue of intra-class concept shift that often exists within the labeled categories in most datasets, which is overlooked by previous methods. Our method enables the model to learn an adaptive feature space that better aligns fine-grained sub-class concepts, taking into account the diversity present within each category. Let X denote the input query, Z denote the embedding extracted by the encoder, L denote the data labels, 2 denotes the embedding obtained from the memory, X denote the visualization of the memory, É denotes the prediction of the input query, and the K denotes the top-K K relevant sub-prototypes, respectively. The overall pipeline is presented in Figure 2. More details will be described in the following sub-sections.
3.2 INPUT-ORIENTED EMBEDDING Vs. TASK-ORIENTED EMBEDDING
Typically, the image feature extracted by visual encoder is directly used for learning downstream tasks. We call this kind of feature input-oriented embedding. However, it heavily relies on the original image content. Since different samples of the same category always vary significantly in their visual
3

---

Published as a conference paper at ICLR 2024
Retzieved Sub-Prototype
Itrelevant Sub-Prototype:
Source Sample
Rncoder
Memory Addressing
Matched Sub-Class:
Input- Oriented Embedding
X
Tasle-
Oriented
Embedding
Encoder
Memory
Cycle Consistent Matching
Target Sample
Sub-Protolype Feature Space
Unit
X
MemSPM
Figure 2: Our model first utilizes fixed pre-trained model as the encoder extract input-oriented embedding given an input sample. The extracted imput-oriented embedding is then compared with sub-prototypes learned memory to find the closest K. These K are then weighted-averagee into task-oriented embedding to represent the input, and used for learning downstream tasks. During the
UniDA process, we adopt the cycle-consistent matching method on the task-oriented embedding ž generated from the memory. Moreover, decoder is designed to reconstruct the image, allowing for visualizing the sub- uub-prototypes in memory and verifying the effectiveness of sub-class learning.
features, categorization based on the input-oriented embedding is sometimes unattainable. In our pipeline, we simply adopt CLIIP-based (Radford et al., 2021) pre-trained visual encoder to extract the input-oriented embeddings, which is not directly used for learning downstream task.
In MemSPM, we propose to generate task-oriented embedding, which is obtained by serving input- oriented embedding as query to retrieve the sub-prototypes from our memory unit. We define Sencode (.)= X 7 represent the fixed pre-trained encoder and Jetass AUnsDA(.) 2 - f. to represent the UniDA classifier. The input-oriented embedding Z is used to retrieve the relevant sub-prototypes from the memory. The task-oriented embedding 2 is obtained using the retrieved sub-prototypes for classification tasks. In conventional ways, Z = Z, which means Ž is obtained directly from Z. Our method obtains Ž by retrieving the sub-prototyper from the memory, which differentiates Ž from Z and reduces the domain-specific information from the target domain during testing phase. Therefore, the task-oriented information retrieved from memory will mainly have features from the source domain. Then, classifier can effectively classify, similar to how works in source domain.
3.3 MEMORY-ASSISTE
SIB-PROTOTYPE MINING
The memory module proposed MemSPM consists of two key components: a memory unit responsible for learning sub- cut-prototypes, and an attention-basee addressing (Graves al., 2014) operator to obtain better task representation 2 for the query, which is more domain-ine ionain-invariant.
3.3.1 MEMORY STRUCTURE WITH PARTITIONED SUB-PROTOTYPE
The memory structure in MemSPM is represented matrix denoted by M NxSxD where N indicates the number of memory items stored, S refers to the number of sub-prototypes partitioned within each memory item, and D represents the dimension structure has learnable parameters and For convenience, we assume D is the same to the dimension of RC (=)R Let the vector mij. Vi [N] denote the i-th of M, where [N] denotes the set of integers from I to N, Vj [S] denote the j-th sub-prototype of M items, where [S] denotes the set integers from to S. Each
each sub-prototype. The memory the uniform distribution to initialize memory items.
m, denotes memory item. Given embedding Z E D, the memory module obtains 2 through soft addressing vector W E 1 as follows:
2 = W. M = Eawa Misd (Einstein summation),
(1)
Wi.=s;; argmax (ingma
4

---

Published as a conference paper at ICLR 2024
where W is vector with non-negative entries that indicate the maximum attention weight of each item's 's sub-prototype. 8; denotes the index the sub-prototype in the i-th item, and Wi,f=e, denotes the i,j sy-th entry of W. The hyperparameter N determines the maximum capacity for memory items and the hyperparameter defines the number of sub-prototypes each memory item. The effect of different settings of hyper-parameters is evaluated in $4.
3.3.2 SUB-PROTOTYPE ADDRESSING AND RETRIEVING
In MemSPM, the memory M is designed to learn the sub-prototypes to represent the input-oriented embedding Z. We define the memory as content addressable memory (Gong et al., 2019; 9; Chen et al., 2018; Sukhbaatar al., 2015; Rae et al., 2016) that allows for direct referencing of the content of the memory being matched. The sub-prototype is retrieved by attention weights W which are computed based the similarity between the sub-prototypes in the memory items and the input-oriented embedding Z. To calculate the weight Wi,j, we use softmax operation:
(sp(d(z,ma.))
Wa S د exp(d(z, m),,))
where -( denotes cosine similarity measurement. As indicated by Eq. and 3, the memory module retrieves the sub-prototype that is most similar to Z from each memory item in order obtain the new representation embedding 2. As consequence of utilizing the adaptive threshold addressing technique(Section 3.3.3), only the top-K K be utilized to obtain task-oriented embedding 2, that serves to represent the encoded embedding Z.
3.3.3 ADAPTIVE THRESHOLD TECHNIQUE FOR MORE EFFICIENT MEMORY
Limiting the number of sub-prototypes retrieved can enhance memory utilization and avoid negative impacts unrelated sub- cub-prototypes during model parameter updates. Despite the natural reduction in the number of selected memory items, the attention- addressing mechanism may still lead to combining small attention-weij
items into the output embedding 2. which has negative impact on the classifier and uub-proootypes in memory bank. Therefore, it is necessary impose mandatory quantity limit on the amount of the relevant sub-prototypes retrieved. To address this issue, we apply adaptive threshold operation restrict the number of sub-prototypes retrieved in the forward process.
Wii=
Wij=a, A
Wjj==s
(4)
0,
other
where Wr,j=,, denotes the i.j = si-th entry of the denotes the adaptive threshold:
argmin(topk(un))
Directly implementing backward for the discontinuous function in Eq. 4 is challenging. We utilize method (Gong et al., 2019) which rewrites operation using continuous ReLU activation function as:
max(
X, 0) Wi.j=s
Wij==
(6)
Wij==:: >1 +E
where max(:,0) 0) is commonly referred to the ReLU activation function, and is small positive
scalar. The prototype å will be obtained by 2 W M. The adaptive threshold addressing encourages model to represent embedding Z using fewer but more relevant
leading
learning more effective features memory and reducing the impact on irrelevant uub-prototypes
3.4 VISUALIZATION AND INTERPRETABILITY
We denote f (.) 2 + X to represent the decoder. The decoder is trained to visualize what has been learned in the memory by taking the retrieved sub-prototype as input. From an interpretability perspective, each encoded embedding Z calculates the cosine similarity find the
5

---

Published as a conference paper at ICLR 2024
top-K K fitting sub-prototype representation for the given input-oriented embedding. Then, these sub-prototypes are combined represent the Z in Z. The sub-prototype in this process can be regarded as the visual description for the input embedding Z. In other words, the input image is much like the sub-classes represented by these peb-protttpps. In this way, samples with significant intra-class differences will be matched to different sub-prototypes, thereby distinguishing different sub-classes. The use of reconstruction auxiliary task can visualize the sub-prototypes memory to confirm whether our approach has learned intra-class differences for the annotated category. The results of this visualization are demonstrated in Figure
3.5
CONSIISON ALIGNMENT AND ADAPTION
Once the aub-prototypes mined through memory learning, the method of cycle-consistent match- ing, inspired by DCC (Li et al., 2021), is employed to align the embedding 2. The cycle-consistent matching is preferred due to it can provide a better fit to the memory structure compared to other UniDA methods. The other method, One-vs-All Network (OVANet), proposed by Saito al. (Saito and Saenko, 2021), needs to train the memory multiple times, which can lead to significant computa- tional overhead. In brief, the Cycle-Consistent Alignment provides solution by iteratively learning a consensus set of clusters between the two domains. The consensus clusters are identified based on the similarity of the prototypes, which is measured using a similarity metric. The similarity metric is calculated on the feature representations of the prototypes. For unknown classes, we set the size N of our memory during the initial phase to be larger than the number of possible sub-classes that may be learned in the source domain. This size is hyperparameter that is adjusted based on the dataset size. Redundant uub-prototypes are invoked to represent the 2, when encountering unknown classes, allowing for an improved distance separation between unknown and known classes in feature space.
Training Objective. The adaptation loss in training is similar to that of DCC, LDA:
LDA = Loe + Ae + X2Cregs
where the Loe denotes the cross-entropy loss source samples, Load denotes the domain alignment loss, and Creg denotes the regularize (more details in Appendix E). For the auxiliary reconstruction task, we add a mean seraan-suuree--ere (MSE) loss function, denoted as Lree:
L= + Arhree Lee + A2 + >xZFeeg Xyhree
(8)
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
We first conduct the experiments in the UniDA setting (You al., 2019) where private classes exist in both domains. Moreover, we also evaluate our approach on two other sub-cases, namely Open-Set Domain Adaptation (OSDA) and Partial Domain Adaptation (PDA).
Datasets. Our experiments are conducted on four datasets: Office-31 (Saenko et al., 2010), which con- tains 4652 images from three domains (DSLR, Amazon, and Webcam); OfficeHome (Venkateswara et al., 2017), a more difficult dataset consisting of 15500 images across 65 categories and 4 domains (Artistic images, Clip-Art images, Product images, and Real-World images); VisDA (Peng al., 2017), large-scale dataset with synthetic source domain of 15K images and a real-world target domain of 5K images; and DomainNet (Peng et al., 2019), the largest domain adaptation dataset with approximately 600,000 images. Similar to previous studies (Fu et al., 2020), we evaluate our model on three subsets of DomainNet (Painting, Real, and Sketch).
Table 3: The division label set, Common Class (C) Private
As in previous work (Li et al., 2021; Saito et al., 2018; Busto et al., 2018; Cao et al., 2018; You et al., 2019), we divide the label set into three groups: common classes C, source-private classes Ö., and target-priag privettprivate classes C, The separation of classes for each of the four datasets is shown in Table 3 and is determined according alphabetical order.
Class (C.) / Target Private Class (C+).
Class SelntCCC (6)
Daiset
PDN
USINN
UnDA
Offia:31
10/2170
1010711
10/10111
Offcctione
25/40/0
25/0/40
10/5/50
ViDDA
6r620
610/5
6/373
DomuisNai

---

Published as a conference paper at ICLR 2024
Table 1: H-score (%) comparison in UniDA scenario on DomainNet, VisDA and Office-31 ,ome
results are cited from (Li et al., 2021; Qu et al., 2023)
Mathoal
Pretnin
P26
P25
R2P
DomainNe
R25
S2F
S26
VisDA
S2K
A2D
A2W
D2A
Office 31
D2W
W2A
W20
Arg
UAN (You at. 2019)
35.7
34.8
71.4
03.5
CMU (T 2020)
50.8
45.1
52.2
45.6
44.8
51.0
16.3
52.9
68.1
67.3
71.4
79.3
72.2
90.4
73,1
DOC 2021)
43.3
88.5
58.6
80.2
OVANs (Salto Sacako, 2021)
ImsgeNNet
56.0
47.1
51.7
44.9
47.4
57.2
50.7
53.1
85.8
79.4
80.1
95.4
54.0
94.3
86.5
UMAD (Linng atall, 1021)
98.0
44.3
50,1
42.1
52.0
55.3
47.1
58.3
79.1
77.4
87.4
90.7
90.4
97.2
67.0
(Chen al. 2022)
57.4
47.6
49.5
52.1
56.4
87.7
94.1
87.6
UniT (Chung et 2022)
99.3
47.8
51.8
46.3
48.3
58.3
52.0
57.3
83.7
85.8
71.4
91.2
70,9
90.84
82.2
87.8
GLC Qu (2023)
74.4
63.4
00.0
6239
62.0
74.3
64.6
80.3
80.5
80.4
77.5
95.6
77.7
96.9
84.8
DCC (LL
CLIP
49.3
61.2
$8.7
82,1
MemSPM-DCC
72.4
62.8
58.5
633
50.4
72.6
63.3
80.5
88.0
84.6
88.7
87.6
87.9
94.3
88.5
Table H-score (%) comparison in UniDA scenario on Office some results are cited from (Li et al., 2021; Qu et al., 2023)
Method
Pretmis
Office loome
Ar3C3
Ar2P/
Ar2RW
C12AA
C12P/
C12Rw
P:2.M
Pr201
PVZRW
Rw/AA
Rw2(1
Ruu2P/
Avg
UAN (You at. 2019)
51.7
51.3
51.7
37.6
52.0
55.0
CMU (fu 2020)
$6.0
56.9
59.2
57.0
64.3
67.8
54.7
51.1
66.4
68.2
57.9
69:7
61.6
DOC 2001)
54.1
58.0
70.6
77.5
75.1
70.2
OVANer (Satoo and Saenko, 2021)
IraggeNett
62.8
75.6
78.6
70.7
68.8
75.0
71.3
58.6
80.5
76.1
64.1
18.9
71.8
UMAD (Liing et atll. 2021)
61.1
76.3
82.7
70.7
67.7
75.7
64.4
55.7
76.3
73.2
60.4
77.2
70.1
(Chen et al, 2012)
63.8
75.9
81.4
74.0
72.1
79.8
74.7
711.
K2.7
79.1
71.5
75.6
UnIT (Chang et 2022)
67.2
90.5
85.0
73.5
77.3
84.3
75.5
63.3
$6.0
77.8
65.4
$1.9
76.6
(Ou
78.2
75.6
GLC es 2003)
79.4
$3.9
90.8
76.3
84.7
19.0
71.5
72.9
85.7
78.2
79.4
90.3
82.6
DCC (Li 2021)
CLI
62.6
88.7
87.4
63.3
68.5
79.3
67.9
63.8
69.8
74.4
MemSPM-DCC
78.1
90.3
90.7
81.9
90.5
88.3
79.2
77.4
87.8
78.8
76.2
01.6
84.2
Evaluation Metrics. We report the average results of three runs. For the PDA scenario, we calculate the classification accuracy over all target samples. The usual metrics adopted to evaluate OSDA are the average class accuracy over the known classes OS*, and the accuracy of the unknown class UNK. In the OSDA and UniDA scenarios, we consider the balance between "known" and "unknown" categories and report the H-score (Bucci et al., 2020):
OS x UNK
H-score = 2 x OS + UUK'
(9)
which is the harmonic mean of the accuracy of "known" and "unknown" samples.
Implementation Details. Our implementation is based on PyTorch (Paszke al., 2019). We use CLIP (Dosovitskiy et al., 2020) as the backbone pretrained by CLIP (Radford et al., 2021) for the MemSPM is hard to train with a randomly initialized encoder. The classifier consists of two fully-connected layers, which follow the previous design (Cao et al., 2018; You et al., 2019; Saito al., 2018; Fu al., 2020; Li al., 2021). The weights in the are empirically set as À1 0.1, d2 = 3 and A3- = 0.5 following DCC (Li et al., 2021). For fair comparison, we also adopt CLIP as backbone for DCC (Li et al., 2021) and state-of-ar method GLC (Qu al., 2023). We use the official code of DCC (Li et al., 2021) and GLC (Qu et al., 2023) (Links in Appendix D). Regarding computational resources, MemSPM demonstrates efficient training on the Office-Home dataset using single RTX 3090. The entire training process is completed within one day.
4.2 COMPARISON WITH OFATE-OF-THE-AEE
We compare our method with previous state-of-the-art algorithms in three sub-cases of unsupervised domain adaptation, namely, object-specific domain adaptation (OSDA), partial domain adaptation (PDA), and universal domain adaptation (UniDA).
Results on UniDA. In the most challenging setting, i.e. UniDA, our MemSPM approach achieves state-of-the -frt performance. Table 7 shows the results on DomainNet, VisDA, and Office-31, and the result of Office-Home is summarized in Table 2. We mainly compare with GLC and DCC using ViT-B/16 16 as the backbone, On Office-31, the McmSPM+DCC outperforms the previous state-of-art method GL. by 3.7% and surpasses the DCC by 6.4%. On visda, our method surpasses the DCC
7

---

Published as a conference paper at ICLR 2024
Table H-score (%) comparison in OSDA scenario on Office-Home, VisDA and Office-31 some results are cited from (Li et al., 2021; Qu et al., 2023)
Mediod
Home
VilDA.
Ar2iex:
(32Ar
1123v
(13l1x
ltx2Ar
Arg
A0g
Ang
OS8P 15ako ot 2013)
65.2
12.9
64.3
64.7
70.6
6322
53.2
13.5
66.7
51.5
11.3
54.7
$5.7
52.3
CMB (Po 2000
55.0
51.0
19.0
59.3
16.7
00.06
90.2
51.3
61.3
61.9
53.3
55.3
57.6
65.3
54.3
DCC (Li etd.. 2021)
56.1
67.5
56.7
-9.0
06.5
64.00
55.8
53.0
70.5
61.6
57.2
71.9
51.7
12.7
59.6
ONANa datio and Suerka, 2021)
ImageNer
58.5
66.3
59.9
62.00
65.2
68.6
59.8
51.4
69.3
68.7
59.6
65.7
54.0
91.3
2021)
71.7
so8 (Chang stal. 20035
15.5
58.9
65.3
65.6
90.6
56.3
N8.8
7533
(LC etel-0000
66.3
74.2
79.0
60.4
71.6
74.7
62.7
63.2
75.8
67.1
64.3
77.8
69.8
89.0
12.4
CLA et stal. 2023))
16.4
$1.7
64.5
76.0
62.4
53..5
00.0
39.5
14.5
73.3
06.8
833
16.2
90.1
61.8
DCC ct 2021)
CLIP
62.9
73.3
78.4
49.8
69.2
75.0
99.3
61.5
80.9
68.1
62.5
830
58.4
91.9
66.2
MenSPMteDDC
85.2
72.0
79.3
81.2
08.7
85.2
72.7
84.5
76.5
05.06
79.1
Table 5: H-score (%) comparison in PDA scenario on results are cited from (Li et al., 2021; Qu et al., 2023)
VisDA and Office-31, some
Method
Pretriis
(329)
C12Rx
Office Home
Rw2Ar
Rw:PPt
Offic:1
VilDA
RASUS (Liang 2023)
71.8
72.8
$3.4
61.6
79.3
56.1
DOC (1a ata.. 2001)
52.5
833.8
31.5
46.2
61.2
65.0
15.2
85.5
83.6
10.9
41-3
12.4
OWNNet (Saho and Seecho, 20011
IinageNe:
34.1
54.6
12.1
42.4
47.3
55.9
38.2
26.2
61.7
56.7
35.5
65.5
49.5
74.6
34.3
UMAD (1Jang i 2001)
11.3
06.5
79.2
63.1
62.9
68.2
63.3
56.4
75.9
74.5
55.9
76.3
56.3
89.5
68.5
GATE (Clss et 2022)
55.
75.9
55.3
73.6
70.2
93.0
72.1
59.5
84.7
79L6
63.9
83.8
74,0
92.7
75.6
(2.C stll. 2007)
55.9
76.0
87.5
22.5
71.8
80.7
41.7
92.4
77.5
84.3
72.5
94.1
76.1
2023)
MonSPMeDDCC
687
64.5
24.8
76.7
77.5
60.3
64.3
70.3
8538
14.5
87.8
by a huge margin of 16.1%. Our method also surpasses the GLC by 9.9% and the DCC by 4.5% on DomainNet. On the Office-Home, we surpass the DCC by 9.8% and the GLC by 7%..
Results on OSDA and PDA. In table 4 and table 5, we present the results on Office-Homne Office- 31, and VisDA under OSDA and PDA scenarios. In the OSDA scenario, MemSPM+DCC still achieves state-of-the-arr performance. Specifically, MemSPM+DCC obtains 95.6% H-score on Office-31, with an improvement of 5.5% compared GLC and 13.7% compared to DCC. In the PDA scenario. McmSPM still achieves comparable performance compared to methods tailored for PDA. The MeCSPM+DCC surpasses the DCC by 8. 8.1% on the VisDA.
4.3 ABLATION STUDIES
Visualization with Reconstruction and tSNE We first visualize what the memory learns from Office- Home by sampling a single sub prototype and adapting an auxiliary reconstruction task: X + X. We also provide the tSNE of the Ż which retrieves the most related sub- uub-prototypes. The visualization is shown in Figure 3. The tSNE visualization depicts the distribution of sub-classes within each category, indicative of MemSPM's s successful mining of sub-prototypes. The reconstruction visualization shows what has been learned by MemSPM, demonstrating its ability to capture intra-class diversity.
Memory-Assisted Sub-Prototype Mining (MemSPM) Impact. As shown in Tables 7, 2, 4, and 5, McmSPM+DCC outperforms DCC across UniDA, OSDA, and PDA scenarios. MemSPM significantly enhances DCC performance with CLIP as the backbone. CLIP is applied because MemSPM's memory module, with large latent space initialized by the random normal distribution, faces challenges in retrieving diverse sub-prototypes early in training.
Sensitivity syper-parameteryy We conducted experiments on the VisDA dataset under the UniDA setting to demonstrate the impact of hyperparameters S and N on the performance of our method. The impact of S illustrated in Figure When 2 20, the performance reaches a comparable result; for the best performance on Office-Home, S 2 40 is achieved. At the same time, the performance of the model is not sensitive to the value of N, when S = 30. For parameter K, we conducted experiments with K- = 1, signifying the selection of only the most relevant item. In the visualization results, the visualizations of memory items displayed meaningless images. The value of K is determined based the attention values. During the design process, observed that the fifth attention value is nearly zero. Consequently, we employ a top-K K approach (K = 5) to filter out other noisy memory items. We will add these results analysis and visualizations to the revised manuscript.
8

---

Published as a conference paper at ICLR 2024
(
K+1
K=3
K
Calculator
Bike
Figure 3: (a) The iSNE visualization shows the feature space of the sub-classes belonging each category, which demonstrates the MemSPM mining the sub-prototypes successfully. (b) The results of different values of S and N. (c) The reconstruction visualization shows what has been learned in the memory, which demonstrates the intra-class diversity has been learned by MemSPM. (d) The visualization of varying K shows that insufficient values hinder the learning of appearance features.
Table 6: Ablation Studies
Matbod
Peimin
Ar3el
Ar30y
Ar3llw
(12Ar
e22lww
F:X1
Ros2Ar
Ang
CLP
645
$5.3
75.1
75.7
85.2
36.5
65.1
68.7
89.6
68.5
60.4
36.6
77.2
DCC-MeuSPM
ImugeNes
57.1
55.0
58.4
68.8
61.1
55.2
83.5
76.1
87.5
$2.7
77.3
76.4
76.7
DCU*MMMSPM
CLIP
78.1
90.3
90.7
$1.9
90.5
88.3
792
774
87.8
78.8
76.2
91.6
84.1
DCCMMMeSPMM
Nove
56.7
784
85.6
502
60.7
671
582
46.1
77.9
671
60.3
81.7
64.
Fibal Tir MEmAPA
CFP
545
607
874
633
B8.5
791
630
653
814
707
858
853
7130
CTMSSSSs Wilbor Lotd
CLP
153
754
86.
301
71.6
873
701
87.1
85.7
742
755
8538
79.3
Effect of CLIP-based Feature. As shown in Table 6, we have conducted experiments to compare ViT-B/16 (pre-trained by CLIP), ViT-B/16 (pre-trained on ImageNet), and ViT-B/16 (without pre- training). The performance of MemSPM on Officehome using ViT-B/16 (ImageNet) is 76.7% (H-score), which is 7.5% lower than MemSPM using ViT-B/16 (pre-trained on CLIP). Additionally, the ViT-B/16 (without pre-training) only achieves 64.3%. 43,, which is 19.9% lower than that using ViT-B/16 (pre-trained CLIP).
Effect of Adaptive Threshold As shown in Table 6, to demonstrate the effectiveness of the adaptive threshold, find best-performed fixed threshold of 0,005 through experiments. It limits the memory to learn sub-prototypes, which only achieved 73.9% (H-score) on Officehome.
Effect of Loss As shown in Table 6, we experimented with loss contributions. Lee for classification is essential; removing Ledd led to 4.4% drop (79.8%). Optimal coefficients for Lee (A1 = 0.1) and Lead (22 3) achieves the best performance. The reconstruction loss (Crec) slightly improved performance, mainly for visualizing sub-prototypes.
CONCLUSION
In this paper, we propose the Memory-Assisted Sub-Prototype Mining (MemSPM) method, which can learn the intra-class diversity by mining the sub-prototypes to represent the sub-classes. Compared with previous methods, which overlook the intra-class structure by using the one-hot label, our Mem- SPM can learn the class feature from a more subdivided sub-class perspective to improve adaptation performance. At the same time, the visualization of the tSNE and reconstruction demonstrates the sub-prototypes have been well learned as we expected. Our MemSPM method exhibits superior performance in most cases compared with previous state-of-the-arr methods four benchmarks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMEEN
This work was partially supported by the National Natural Science Foundation China (Grants No 62106043, 62172228), and the Natural Science Foundation of Jiangsu Province (Grants No BK20210225).
REFERENCES
Bucci. S. Loghmani, M. R., and Tommasi, T. (2020). On the effectiveness of image rotation for open set domain adaptation. In Proceedings of the European Conference Computer Vixion. pages 422-438.
Busto, P. P., lqbal, A., and Gall, J. (2018). Open set domain adaptation for image and action recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 42(2):413 429..
Cao, Z., Ma, L., Long, M., and Wang, J. (2018). Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision, pages 135-150.
Cao, Z. You, K., Long, M., Wang. J., and Yang. Q. (2019). Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2985-2994.
Chang. W. Shi, Y.. Tuan, H., and Wang, J. (2022). Unified optimal transport framework for universal domain adaptation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems, volume 35, pages 29512-29524. Curran Associates, Inc.
Chen, L., Lou, Y., He, J., Bai, T., and Deng, M. (2022). Geometric anchor correspondence mining with uncertainty modeling for universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16134 16143....
Chen, Y., Zhu, X., and Gong, S. (2016). Deep cetannttucttoonclassfiaaaa networks for unsupervised domain adaptation. Proceedings of the European Conference on Computer Vision, pages 597-613.
Chen, Y., 7hu, X., and Gong, S. (2018). Semi-supervised deep learning with memory. In Proceedings of the European Conference on Computer Vision, pages 268-283.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. (2020). An image worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.118:
Fu, B., Cao, Z., Long, M., and Wang, J. (2020). Learning to detect open classes for universal domain adaptation. In Proceedings of European Conference Computer Vision, pages 567-583.
Ganin, Y. and Lempitsky: V. (2015). Unsupervised domain adaptation by hackpropagation. In Bach, F. and Blei, D., editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1180-1189, PMLR.
Gong, D., Liu, L., Le. V., Saha, B., Mansour, M. R., Venkatesh, S., and Hengel, A. d. (2019). Memorizing normality to detect anomaly: Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1705-1714.
deep autoencoder unsupervised anomaly detection. In
Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5400
Hsu, T. M. H., Chen, W. Y., Hou, C.-A., Tsai, Y.HH. H. Yeh, Y.-.., and Wang, Y.-C. F. (2015). Unsupervised domain adaptation with imbalanced cross-domain data. In Proceedings of the IEEE International Conference on Computer Vision.
Kalluri, T., Sharma, A., and Chandraker, M. (2022). Memsac: Memory augmented sample consistency for large scale domain adaptation. In Proceedings of the European Conference on Computer Vision, pages 550-568.
Kang, G., Jiang, L., Yang. Y., and Hauptmann, A. G. (2019). Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Kundu, J. N., Bhambri, S., Kulkarni, A. R.. Sarkar, H., Jampani, V., al. (2022). Subsidiary prototype alignment for universal domain adaptation. Advances in Neural Information Processing Systems, (-29649-296661
Li. G., Kang, G., Zhu, Y., Wei. Y., and Yang. Y. (2021). Domain consensus clustering for universal domain adaptation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 9757-9766.
10

---

Published as a conference paper at ICLR 2024
Liang, J., Hu, D., Feng, J., and He, R. (2021). Umad: Universal model adaptation under domain and category shift, arViv preprint arXiv:2112.08553.
Liang, J., Wang. Y., Hu, D.. He, R.,. and Feng. J. (2020). A balanced and uncertainty- approach for partial domain adaptation. In Proceedings of the European Conference on Computer Vision, pages 123-140.
Liu, X., Zhou, Y., Zhou, T., and Qin, (2023), Self-paced learning for open-set domain adaptation. Journal of Computer Research and Development, 60(8):1711 1-26..
Liu, Z.. Chen, G., Li, Z.. Kang, Y., Qu, S., and Jiang. C. (2022). Psde: A prototype-based shared-dummy classifier model for open-set domain adaptation. IEEE Trunsactions on Cybernetics.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopt, A., Yang. E., Vito, Z., Raison, M., Tejani, A.. Chilamkurthy, S., Steiner, B., Fang. L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates. Inc.
Pong. X., Bai, Q. Xia, X., Huang. Z., Saenko, K., and Wang. B. (2019), Moment matching for multi souti- domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Peng. X., Usman, B., Kaushik, N.. Hoffman. J., Wang, D., and Saenko, K. (2017). Visda: The visual domain adaptation challenge. arXiv preprint urXiv:1710.06924.
Qu, S., Zou, T., Röhrbein, F., Lu, C., Chen, G., Tao, D., and Jiang. C. (2023). Upcycling models under domain and category shift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR.
Rae. J.. Hunt, J. J., Danihelka, I., Harley, T., Senior, A. W., Wayne, G., Graves. A., and Lillicrap. T. (2016). Scaling memory-angmented neural networks with sparse reads and writes. Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Saenko, K., Kulis, B., Fritz, and and Darrell, (2010). Adapting visual category models to new domains. In Proceedings of the European Conference on Computer Vision, pages 213-226.
Saito, K., Kim, D., Sclaroff, S., and Sacnko, K. (2020). Universal domain adaptation through self supervision. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 16282-16292. Curran Associates, Inc.
Saito, K. and Sacnko, K. (2021). Ovanet: One-vs-all network for universal domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9000-9009.
Saito, K., Watanabe, K., Ushiku, Y., and Harada, T. (2018). Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.
Shu, R., Bui, H. H., Narui, H., and Ermon, S. (2018). A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv: /0t.:0882...55.
Sukhbaatar, S., szlam, a., Weston, J., and Fergus, R. (2015). End-to-end memory networks. In Cortes, C..
Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S. (2017). Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
You, K., Long, M., Cao, Z., Wang, J., and Jordan, M. (2019). Universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2720-2729.
Zhou, Y., Bai, S., Zhou, T., Zhang, Y., and Fu, H. (2022). Delving into local features for open-set domain adaptation in fundus image analysis. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 682-692. Springer.
11

---

Published as a conference paper at ICLR 2024
Appendix
In the supplementary material, we provide additional visualization results, limitations, potential negative societal impacts and compute requirements of the MemSPM. In the pursuit of reproducible research, we will make the demo and network weights our code available to the public.
This supplementary is organized follows:
Section A: Notations
Section B: Limitation
Section C: Comparison Between Related Prototype Concepts
Section D: Implementation details
Baseline details
Compute requirements
Section E: Discussion of Motivation
Section F: Visualization Results
Section G: Details of Domain Consensus Clustering
Section H: Potential societal impact
A
NOTATIONS
Table 7:
Symbol
Description
Jencode pun decode
(+)
Fixed image encoder
Unfixed reconstruction decoder
Model
DUNDA
UniDA classifier
class
M
Memory unit
W
Weight vector
D
Labeled source dataset
D
Unlabeled target dataset
C
Common label set
Space
C.
Source label set
C
Target label
Source private label set
Target private label set
X
Input image
Reconstruction of image X
Input-oriented embedding
Samples
Task-oriented embedding
Label of the image
Prediction of image
Wi.j
Attention weight measurement between and sub sub-prototype
Measures
d(---)
Cosine similarity measurement
wh.j
Adaptive threshold operation W.
N
Number of memory items
Hyperparameters
Number sub-prototypes partitioned in each memory item
Dimension of each sub-prototype
K
Top-K relevant sub-prototypes of Z
12

---

Published as a conference paper at ICLR 2024
B LIMITATION
Training the memory unit of MemSPM is challenging when adopting the commonly used ResNet-50 as the backbone. This is due to the memory unit's composition of massive randomly initialized tensors. During the early stage of training. there is a lack of discriminability in the input-oriented embedding, which leads to addressing only a few anb-prototypes. This decoupling of the memory unit from the input data necessitates using a better pre-trained model (ViT-B/16 pre-trained on CLIP) and fixing the encoder to reduce computation requirements. Additionally, the number of sub-prototypes in one memory item might need to be adjusted for the diversity of the category.
C COMPARSION BETWEEN RELATED PROTOTYPE CONCEPTS
The related concept of the prototype is mentioned in some previous works Kundu al. (2022); Liu et al. (2022), there are clear differences between theirs and our MemSPM.
First, the meaning of prototype is different between Kundu et al. (2022) and ours. In the Kundu et al. (2022), the subsidiary prototype is extracted from randomly cropped images, which means the subsidiary prototypes only represent the low-level, morphological, and partial features of the image. These subsidiary prototypes don't 't have complete semantic knowledge, and the method can't learn the concept shift the same category. Moreover, they still used the labeled category directly for alignment and adaptation. These prototypes can't represent some part of the samples in one category.
In contrast, the MemSPM allows memory items to extract complete semantic knowledge and maintain domain-invariant knowledge. To accomplish this, we use input-oriented embedding, which involves comparing the entire image feature with memory items. The memory can then sample task-oriented embedding that represents the semantic knowledge of the input-oriented embedding. Our approach is designed to obtain a shifts. As result, each sub- propotooottppe can represent sub-class in one category.
and semantic feature for categories with significant domain
Second, the purpose of Liu et al. (2022) is very different from our McmSPM. They aim to learn differences among unknown classes, which is like the DCC. It still extracts features and aligns the class across different domains directly based on one-hot labels, and is not concerned with the concept shift and difference in one category. However, our method can mine the sub-classes in one category when there exist significant concept shifts, rellecting the inherent differences among samples annotated as the same category. This helps universal adaptation with a more fine-grained alignment or to make significant decisions without human supervision.
D IMPLEMENTATION DETAILS
DCC. We use ViT-B/16 (Dosovitskiy et al., 2020) as the backbone. The classifier made up of two FC layers. We use Nesterov momentum SGD to optimize the model, which has a momentum of 0.9 and a weight decay of 5e-4. The learning rate decreases by a factor of (1 + a )-8, where and N represent current and global iteration, respectively, and we set 10 and 3 = 0.75. We use batch size of 36 and the initial learning rate is set as 1c-4 for Office-31, and 1c-3 for Office-Home and DomainNet. We use the settings detailed in Li et al. (2021). PyTorch Paszke et al. (2019) is used for implementation.
GLC. We use VIT-B/16 (Dosovitskiy al., 2020) as the backbone. The SGD optimizer with momentum of 0.9 is used during the target model adaptation phase of GLC (Qu et al., 2023). The initial learning rate is set to 1e-3 for Office- Home and 1e-4 for both VisDA and DomainNet. The hyperparameter p is fixed at 0.75 and [L] at 4 across all datasets, while is set to 0.3 for VisDA and 1.5 for Hofice-Hoome and DomainNet, which corresponds the settings detailed in (Qu et al., 2023). PyTorch (Paszke et al., 2019) is used for implementation.
Existing code used.
DCC (Li et al., 2021):
https:
com httpa////ooaeeee//ooo/aoooeeaeee CCusterrin
GLC (Qu et al., 2023):
https:://// / //tpt/ttttt/ttt// com cttc hamm//scc-abb/G.c
13

---

Published as a conference paper at ICLR 2024
PyTorch (Paszke al., 2019): httpss://ptt.tt org/
Existing datasets used.
Office-31 (Saenko al., 2010):
https://wwer wt..///wwccccooo....ooooooo/ gatech. edu /ai.i jjudy/ hauujddddday//// doma nadapt
Office-Home (Venkateswara et al., 2017):
https://ww/ hww.nthdw..tt..mmmm.mmw. wwwpettdvoorrr/g/gge wwt.or/g/ooo.oo...ooooo cetomeaaatteeeeeeeeeeeeeeeeeeeee html
DomainNet (Peng al., 2019);
actp://ai..b..edu.
VisDA al, 2017). 2017):
http://ai.bu.edde wwweeeddvvviidddddeeeeeeaa 2017/
Compute Requirements. For our experiments, we used a local desktop machine with an Intel Core i5- 12490f, single Nvidia RTX-3090 GPU, and 32GB RAM. When we adapt the batch-size used in DCC (Li et al., 2021), our MemSPM only occupies 4GB of GPU memory during training as the result of fixing the encoder.
E DETAILS OF DOMAIN CONSENSUS CLUSTERING
Domain Consensus Clustering (DCC). They leverage Contrastive Domain Discrepancy (CDD) to facilitate the alignment over identified common samples in class-aware style. They impose LCDD to minimize the intra-class discrepancies and enlarge the inter-class gap. Consequently, the enhanced discriminability, in turn, enables DCC perform httpee/ee..ccccooottt//////OOOOO
accurate clustering. Details of CDD are pro- vided in: https: http//ppaaacccccmooo ccess thecvf. cont /CVRR2021 att.:///aaappppppapppmamam emental/
Li oup.Daaii.cccaaaeeccccttttieiiin
pdf.
DISCUSSION OF MOTIVATION
Illustrated Figure 1, our motivation arises from the recognition that samples annotated within the same category often exhibit significant intra-clase differences and concept shifts. In Figure (a). we visually depict this phenomenon, showeasing samples labeled as the class "alarm clock" further divided into three distinct sub-classes: desktop alarm clock, digital alarm clock, and wall alarm clock. Similarly, the class "airplane" encompasses multiple sub-classes, such as propeller planes and jet aircraft. This demonstrates that the semantic ambiguity and annotation cost lead to the inefficiency of class alignment. Previous methods often force samples with significant concept shifts to align together during adaptation, increasing the likelihood of misclassifying unknown classes into known classes. In contrast, our proposed method addresses this challenge by introducing sub-prototypes. refining the features of known classes by separating them into sub-classes, and reducing the risk of negative transfer.
G VISUALIZATION
We provide more results of visualization in Figure 4 and Figure 5 to reveal sub-prototypes stored in the memory unit, which demonstrates that our MemSPM approach can learn the intra-class concept shift.
H POTENTIAL SOCIETAL IMPACT
Our finding of the intra-class concept shift may influence future work on domain adaption other tasks. They can optimize the construction and refinement of the feature space by considering the intra-class distinction. The MemSPM also provides method that be used to demonstrate the interpretability of the model for further deployment. However, the utilization of the MemSPM method for illegal purposes may be facilitated by its increased availability organizations individuals.
14

---

Published as a conference paper at ICLR 2024
Hyg tertee
AME
B
Figure 4: The reconstruction visualization shows what has been learned in the memory, which demonstrates the intra-class diversity has been learned by MemSPM.
3.0
The iSNE visualization shows the distribution of the retrieved sub-prototypes and demonstrates that the sub-classes have been learned by MemSPM.
Figure 5:
The MemSPM method may be susceptible to adversarial attacks as all contemporary deep learning systems. Although we demonstrate increased performance and interpretability compared to the state-of-the-ann methods, negative transfer is still possible in extreme cases of domain shift or category shift. Therefore, our technique should be employed in critical applications or make significant decisions without human supervision.
15


Reviews:
[{"review_id": "MlQAIn0txc", "reviewer": "Reviewer_59Jy", "summary": "This paper focuses on Universal Domain Adaptation (UniDA), a practical DA setting that does not make any assumptions on the relation between source and target label sets. The goal is to adapt a classifier from source to target domain such that both source and target domains may have their own private classes apart from shared classes. The paper claims that existing UniDA methods overlook the intrinsic structure in the categories, which leads to suboptimal feature learning and adaptation. Hence, they propose memory-assisted sub-prototype mining (MemSPM) that learns sub-prototypes in a memory mechanism to embody the subclasses from the source data. Then, for target samples, weighted sub-prototype sampling is used before passing the embedding to a classifier, which results in reduced domain shift for the embedding. They also propose an adaptive thresholding technique to select relevant sub-prototypes. Finally, they adopt the cycle consistent matching loss objective from DCC [24] along with an auxiliary reconstruction loss for training. They show results on UniDA, Partial DA, and Open-Set DA using standard benchmarks like Office-31, Office-Home, VisDA, and DomainNet.", "questions": "* Please see the weaknesses section.\n\n* Minor comments\n    * Paragraph above Eq. 6 has typos, add space after citation of Gong et al. 2019.\n    * Fig. 3: increase the font size of text inside the figure to match the caption font size. This will improve the readability of the plots.\n    * Paragraph on “Effect of Loss” has a typo: “We” → “we”.\n    * Table 6 (last row) has a typo: use math mode in LaTeX for $\\mathcal{L}\\_{cdd}$.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivating ideas for the approach are interesting and intuitive. Further, the technical contributions are novel as well as effective.\n\n* It is intriguing that the auxiliary reconstruction task provides interpretability, which is usually not possible in existing DA solutions.\n\n* The paper is fairly well-written and easy to understand.\n\n* With their method and the advantages of a CLIP-pretrained ViT model, they achieve large improvements over existing ResNet-based methods. While they also show improvements over some existing methods using the CLIP-pretrained model, this will serve as a new strong baseline for future UniDA work.", "weaknesses": "* Missing sensitivity analysis to hyperparameter $K$ in adaptive thresholding\n    * There is no information on how $K$ is chosen to be used in the top-$K$ operation used to compute the adaptive threshold $\\lambda$ in Eq. 5. It may be difficult to tune for new datasets.\n    * As per Table 6, using a fixed threshold drops the performance by almost 10%, which is alarming. Hence, the sensitivity to $K$ could be important and needs to be studied.\n    * One simple baseline to get rid of $K$ could be to use a higher temperature in the attention computation, i.e. to make the attention distribution sharper.\n\n* Incomplete analysis of sensitivity to hyperparameter $S$\n    * As per Fig. 3b, the performance improves drastically when $S$ is increased. But the analysis is only up to $S=40$ and it seems that the performance should improve further if $S$ is increased more.\n    * The paper says that “When $S \\geq 20$, the performance achieves a comparable level” but there is almost a 3-4% increase when $S$ increases from 20 to 30 and also from 30 to 40. Then, it seems that increasing $S$ further should yield more improvements.\n    * In any case, it would be good to identify when the performance starts saturating, to help users select the hyperparameter properly for other datasets. \n\n* Missing discussion on training time and memory requirements\n    * Given the extra prototypes involved, there should be some discussion on training time and GPU memory usage compared to the baseline DCC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLeEc6PT95", "reviewer": "Reviewer_S3ot", "summary": "In this study, the authors introduce the Memory-Assisted Sub-Prototype Mining method, which emphasizes the significance of the internal structure within the category. Unlike conventional methods that treat the classes as a whole, MemSPM refines class features and mines sub-prototypes to represent sub-classes, thereby enhancing adaptation performance. The study provides extensive experiments on four benchmarks to validate its performance.", "questions": "1.  In part 3.3.1, there are N memory items stored. However, the explanation of how these N items are produced and the relationship between them is not clear. Additionally, this paper does not provide a clear explanation of how the sub-prototypes are generated and updated.\n\n2.  In Table 6, why only show the performance without the domain alignment loss?  Need to report the performance without the regularized loss and the reconstruction loss.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1.\tThis paper proposes a novel method called Memory-Assisted Sub-Prototype Mining, designed to significantly improve the model’s adaptability. Additionally, the approach provides insightful visualizations, providing a clear understanding of the methodology.\n2.\tThe effectiveness of the proposed method has been assessed through extensive experiments.", "weaknesses": "1.\tWhile the proposed method demonstrates effectiveness, it appears that the learned sub-prototype does not effectively handle the most challenging aspect of UniDA, such as identifying common and private samples. The Sub-Prototype Mining method seems better suited for general domain adaptation problems rather than specifically addressing the complexities of UniDA.\n2.\tThe paper lacks explicit clarification on how the sub-prototype is initialized and updated, leaving a gap in understanding the methodology.\n3.\tSection 3.5 is unclear and requires further elaboration. The concept of cycle-consistent alignment needs to be more explicitly reflected in the training phase. Additionally, a comparison of performance differences with other alignment strategies would enhance the paper's comprehensiveness.\n4.\tA significant portion of the paper relies heavily on DCC, and the proposed method appears to primarily involve weighting input-oriented embeddings to obtain task-oriented embeddings, which seems to be the only deviation from DCC. This raises concerns about the novelty and originality of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DzW8HBFRod", "reviewer": "Reviewer_mKia", "summary": "This paper proposes a Memory-Assisted Sub-Prototype Mining (MemSPM) for the UniDA problem, which involves the idea of sub-prototypes.", "questions": "See the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of sub-prototype is reasonable and worth trying for UniDA.", "weaknesses": "1. The comparison of DCC and MemSPM+DCC is not fair, due to the usage of CLIP in MemSPM. The author SHOULD replace the CLIP with a learnable encoder to achieve a fair comparison.\n2. Only involving the idea of sub-prototype is not novel enough for the acceptance of top conference. How to use it to solve the concept shift in Fig.1 for UniDA is the key. However, in my opinion, the loss in MemSPM (i.e. Cycle Consistent Matching) is exactly the same with DCC. So, I can not find anything new here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZmy549oHO", "reviewer": "Reviewer_dY2C", "summary": "This paper introduces the Memory-Assisted Sub-Prototype Mining (MemSPM) method for Universal Domain Adaptation. The primary goal is to better align classes and reduce feature gaps between source and target domains, addressing the limitations of most existing models that do not consider intra-class structures, especially when significant concept shifts exist within the same category. The proposed MemSPM enhances model performance by effectively identifying and learning from sub-classes that have considerable concept shifts, leading to a more reasonable feature space, improved transferability, and interpretability. Experimental results across several scenarios (UniDA, OSDA, PDA) show that the MemSPM method outperforms existing benchmarks in many instances.", "questions": "- How does the Memory-Assisted mechanism integrate with existing architectures, and what is its computational overhead?\n\n- Are there any specific scenarios or datasets where the MemSPM method might not be as effective?\n\n- Can the authors provide more insights into the annotation/training costs mentioned and how the proposed method addresses this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The Memory-Assisted Sub-Prototype Mining (MemSPM) brings a novel perspective to the Universal Domain Adaptation domain by addressing the challenge of significant concept shifts within the same category. The paper highlights the limitation of existing models and provides a robust solution to enhance transferability and reflect inherent sample differences. Experimental results support the claims and showcase the model's superior performance in many scenarios.", "weaknesses": "While the paper introduces a novel approach, it could benefit from clearer explanations and visual aids, making the methodology more accessible to readers. Additionally, the paper could delve deeper into potential drawbacks or limitations of the proposed method. Comparison with a broader set of benchmarks might also provide a more comprehensive understanding of the model's applicability and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)