Title: Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation
Venue: NeurIPS 2023
Accepted: no
Keywords: Transfer Learning

Universal Domain Adaption

Memory-Assisted Network

Sub-Prototype Mining
Abstract: Universal domain adaptation aims to align the classes and reduce the feature gap between the same category of the source and target domains. The target private category is set as the unknown class during the adaptation process, as it is not included in the source domain. However, most existing methods overlook the intra-class structure within a category, especially in cases where there exists significant concept shift between the samples belonging to the same category. When samples with large concept shift are forced to be pushed together, it may negatively affect the adaptation performance. Moreover, from the interpretability aspect, it is unreasonable to align visual features with significant differences, such as fighter jets and civil aircraft, into the same category. Unfortunately, due to such semantic ambiguity and annotation cost, categories are not always classified in detail, making it difficult for the model to perform precise adaptation. To address these issues, we propose a novel Memory-Assisted Sub-Prototype Mining (MemSPM) method that can learn the differences between samples belonging to the same category and mine sub-classes when there exists significant concept shift between them. By doing so, our model learns a more reasonable feature space that enhances the transferability and reflects the inherent differences among samples annotated as the same category. We evaluate the effectiveness of our MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our method achieves state-of-the-art performance on four benchmarks in most cases.

Abstract

Universal domain adaptation aims to align the classes and reduce the feature gap between the same category of the source and target domains. The target private category is set as the unknown class during the adaptation process, as it is not included in the source domain. However, most existing methods overlook the intra-class structure within a category, especially in cases where there exists significant concept shift between the samples belonging to the same category. When samples with large concept shift are forced to be pushed together, it may negatively affect the adaptation performance. Moreover, from the interpretability aspect, it is unreasonable to align visual features with significant differences, such as fighter jets and civil aircraft, into the same category. Unfortunately, due to such semantic ambiguity and annotation cost, categories are not always classified in detail, making it difficult for the model to perform precise adaptation. To address these issues, we propose a novel Memory-Assisted Sub-Prototype Mining (MemSPM) method that can learn the differences between samples belonging to the same category and mine sub-classes when there exists significant concept shift between them. By doing so, our model learns a more reasonable feature space that enhances the transferability and reflects the inherent differences among samples annotated as the same category. We evaluate the effectiveness of our MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our method achieves state-of-the-art performance on four benchmarks in most cases.

1 Introduction

Unsupervised Domain Adaptation (UDA) [15, 22, 41, 44, 9, 19, 21] has become a crucial research area of transfer learning, as it allows models trained on a specific dataset to be applied to related but distinct domains. However, traditional UDA methods are limited by the assumption that the source and target domains have to share the same label space. This assumption is problematic in real-world scenarios where the target distribution is complex, open, and diverse. Universal Domain Adaptation (UniDA) represents a strategy to address the limitations of traditional unsupervised domain adaptation methods. In the UniDA, the target domain have a different label set than the source domain. The goal is to correctly classify target domain samples belonging to the shared classes in the source label set, while any samples not conforming to the source label set are treated as "unknown". The term "universal" characterizes UniDA as not relying on prior knowledge about the label sets of the target domain. UniDA relaxes the assumption of a shared class space while aims to learn domain-invariant features across a more broad range of domains.

Despite being widely explored, most existing universal domain adaptation methods [24, 47, 40, 39, 6, 34 34, 8, 26] overlook the internal structure intrinsically presented within each image category. These 35 methods aim to align the common classes between the source and target domains for adaptation, but 36

(a)

Sub-Class1

Sub-Class2

Sub-Class1

Sub-Class2

Sub-Class3

Sub-Class3

Class:Alarm Clock

Class:Airplane

(b)

(c)

Class:AlarmClock

Source Domain

SourceDomain

Previous

Ours

Sub-Class1!

!Sub-Class2

1112

11398

Target Domain

765

Class1 Sample:

Sub-Class 1:

Sub-Class 2:

Class2Sample:

usually train a model to learn the class "prototype" representing each annotated category. This is 37 particularly controversial when significant concept shift exists between samples belonging to the same 38 category. These differences can lead to sub-optimal feature learning and adaptation if the intra-class 39 structure is neglected during training. Since such kind of semantic ambiguity without fine-grained 40 category labels almost happens in all the DA benchmarks, all the methods will encounter this issue. 41

In this paper, we aim to propose a method to learn the detailed intra-class distinction and mine "sub42 prototypes" for better alignment and adaptation. This kind of sub-prototype is the further subdivision 43 of each category-level prototype, which represents the "sub-class" of the annotated categories. The 44 main idea of our proposed approach lies in its utilization of a learnable memory structure to learn sub45 prototypes for their corresponding sub-classes. This can optimize the construction and refinement of 46 the feature space, bolstering the classifier's ability to distinguish class-wise relationships and improve 47 the model's transferability across domains. A comparison between our proposed sub-prototypes 48 mining approach and previous methods is illustrated in Figure 1. In previous methods, samples within 49 a category were forced to be aligned together in the feature space regardless of whether there exist 50 significant differences among them because the labels were one-hot encoded. Contrastively, our 51 sub-prototypes' feature space distinguishes sub-classes with apparent differences within the category, 52 thus improving the model's accuracy of domain adaption and interpretability. 53

Our proposed approach, named memory-assisted sub-prototype mining (MemSPM), is inspired by the 54 memory mechanism works [17, 10, 45, 36]. In our approach, the memory generates sub-prototypes 55 that embody sub-classes learned from the source domain. During testing of the target samples, 56 the encoder produces embedding that are compared to source domain sub-prototypes learned in 57 the memory. Subsequently, a embedding for the query sample is generated through weighted sub58 prototype sampling in the memory. This results in reduced domain shifts before the embedding give 59 into the classifier. Our proposal of sub-prototypes mining, which are learned from the source domain 60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

memory, improves the universal domain adaptation performance by promoting more refined visual concept alignment.

MemSPMapproach has been evaluated on four benchmark datasets (Office-31 [37], Office-Home [46], VisDA [33],and Domain-Net [32]), under various category shift scenarios, including PDA, OSDA, and UniDA. Our MemSPM method achieves state-of-the-art performance in most cases. Moreover, we design a visualization module for the sub-prototype learned by our memory to demonstrate the interpretability of MemSPM. Our contributions can be highlighted as follows:

We study the UniDA problem from a new aspect, which focuses on the negative impacts caused by overlooking the intra-class structure within a category when simply adopting one-hot labels.

We propose Memory-Assisted Sub-Prototype Mining(MemSPM), which explores the memory mechanism to learn sub-prototypes for improving the model's adaption performance and interpretability. Meanwhile, visualizations reveal the sub-prototypes stored in memory, which demonstrate the interpretability of MemSPM approach.

Extensive experiments on four benchmarks verify the superior performance of our proposed MemSPM compared with previous works.

2 Related Work

Closed-Set Domain Adaptation (CSDA). To mitigate the performance degradation caused by the closed-set domain shift, [16, 29, 48] introduce adversarial learning methods with the domain discriminator, aiming to minimize the domain gap between source and target domains. Beyond the use of the additional domain discriminator, some studies [41, 23, 50, 30, 13] have explored the use of two task-specific classifiers, otherwise referred to as bi-classifier, to implicitly achieve the adversarial learning. However, the previously mentioned methods for CSDA cannot be directly applied in scenarios involving the category shift.

Partial Domain Adaptation (PDA). PDA posits that private classes are exclusive to the source domain. Representative PDA methods, such as those discussed in [3, 49], employ domain discriminators with weight adjustments or utilize source samples based on their resemblance to the target domain [5]. Methods incorporating residual correction blocks in PDA have been introduced by Li et al. and Liang et al. [25, 27]. Other research [7, 11, 38] explores the use of Reinforcement Learning for source data selection within the context of PDA.

Open-Set Domain Adaptation (OSDA). Saito et al. [42] developed a classifier inclusive of an additional 'unknown' class intended to differentiate categories unique to the target domain. Liu et al. [28] and Shermin et al. [43] propose assigning individual weights to each sample depending on their importance during domain adaptation. Jang et al. [20] strive to align the source and target-known distributions, while concurrently distinguishing the target-unknown distribution within the feature alignment process. The above PDA and OSDA methods are limited to specific category shift.

Universal Domain Adaptation (UniDA) You et al. [47] proposed Universal Adaptation Network (UAN) to deal with the UniDA setting that the label set of target domain is unknown. Li et al. [24] proposed Domain Consensus Clustering to differentiate the private classes rather than treat the unknow classes as one class. Saito et al. [40] suggested that using the minimum inter-class distance in the source domain as a threshold can be an effective approach for distinguishing between 'known' and 'unknown' samples in the target domain. However, most existing methods [24, 47, 40, 39, 6, 34, 8, 26] overlook the intra-class distinction within one category, especially in cases where there exists significant concept shift between the samples belonging to the same category.

3 Proposed Methods

3.1 Preliminaries

In unsupervised domain adaptation, we are provided with labeled source samples D s = { x s i , y s i ) } n s i =1 and unlabeled target samples D t = { ( x t i ) } n t i =1 . As the label set for each domain in UniDA setting may not be identical, we use C s and C t to represent label sets for the two domains, respectively.

Retrieved Sub-Prototype:---

IrrelevantSub-Prototype:

-

Source

Encoder

Z

Memory

Sub-Class:--

Sample

Input-

Addressing

MatchedSub-Class:-

Oriented

Embedding

Z

X

Task-

Oriented

Embedding

1139

Encoder

W

W-Top K

Memory

Cycle-ConsistentMatching

Target

Sub-Prototype

Unit

Sample

Decoder

FeatureSpace

MemSPM

Then, we denote C = C s ∩ C t as the common label set. ˆ C s , ˆ C t are denoted as the private label sets 110 of the source domain and target domain, respectively. We aim to train a model on D s and D t to 111 classify target samples into | C | +1 classes, where private samples are treated as unknown class. 112

113

114

115

116

117

118

119

120

Our method aims to address the issue of intra-class concept shift that often exists within the labeled categories in most datasets, which is overlooked by previous methods. Our method enables the model to learn an adaptive feature space that better aligns fine-grained sub-class concepts, taking into account the diversity present within each category. Let X denotes the input query, Z denotes the embedding extracted by the encoder, L denotes the data labels, ˆ Z denotes the embedding obtained from the memory, ˆ X denotes the visualization of the memory, ˆ L denotes the prediction of the input query, and the K denotes the top-K relevant sub-prototypes, respectively. The overall pipeline is presented in Figure 2. More details will be described in the following sub-sections.

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

3.2 Input-Oriented Embedding vs. Task-Oriented Embedding

Usually, the image feature extracted by a visual encoder is directly used for learning downstream tasks. We call this kind of feature as input-oriented embedding. However, it heavily relys on the original image content. Since different samples of the same category always varies significantly in their visual features, categorization based on the input-oriented embedding sometimes is unattainable. In our pipeline, we simply adopt a CLIP-based[35] pre-trained visual encoder to extract the input-oriented embeddings, which is not directly used for learning our downstream task.

In our MemSPM, we propose to generate task-oriented embedding, which is obtained by serving input-oriented embedding as a query to retrieve the sub-prototypes from our memory unit. We define f fixed encode ( · ) : X → Z to represent the fixed pre-trained encoder and f UniDA class ( · ) : ˆ Z → ˆ L to represent the UniDA classifier. The input-oriented embedding Z is used to retrieve the relevant sub-prototypes from the memory. The task-oriented embedding ˆ Z is obtained using the retrieved sub-prototypes for classification tasks. In conventional ways, ˆ Z = Z , which means the ˆ Z is obtained directly from Z . Our method obtains the ˆ Z by retrieving the sub-prototypes from the memory, which differenciates ˆ Z with Z , and eliminates the domain-specific information from the target domain during the testing phase. As a result, it improves the performance of f UniDA class ( · ) when performing UniDA.

3.3 Memory-Assisted Sub-Prototype Mining

The memory module proposed in MemSPM consists of two key components: a memory unit 138 responsible for learning sub-prototypes, and an attention-based addressing [18] operator to obtain 139

better task-oriented representation ˆ Z for the query, which is more domain-invariant. 140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

3.3.1 Memory Structure with Partitioned Sub-Prototype

The memory in MemSPM is represented as a matrix, denoted by M ∈ R N × S × D , where N indicates the number of memory items stored, S refers to the number of sub-prototypes partitioned in each memory item, and D represents the dimension of each sub-prototype. For convenience, we assume D is the same to the dimension of Z ∈ R C ( R D = R C ). Let the vector m i,j , ∀ i ∈ [ N ] denote the i -th row of M , where [ N ] denotes the set of integers from 1 to N , ∀ j ∈ [ S ] denote the j -th sub-prototype of M items, where [ S ] denotes the set of integers from 1 to S . Each m i denotes a memory item. Given a embedding Z ∈ R D , the memory module obtains ˆ Z through a soft addressing vector W ∈ R 1 × 1 × N as follows:

where W is a vector with non-negative entries that indicate the max attention weight of each item's sub-prototype, s i denotes the index of the sub-prototype in the i -th item and w i,j = s i denotes the i, j = s i -th entry of W . The hyperparameter N determines the maximum capacity for memory items and the hyper-parameter S defines the number of sub-prototypes in each memory item. The effect of different setting of hyper-parameters is evaluated in Section 4.

3.3.2 Sub-Prototype Addressing and Retrieving

In MemSPM, the memory M is designed to learn the sub-prototypes to represent the input-oriented embedding Z . We define the memory as a content addressable memory [17, 10, 45, 36] that allows for direct referencing of the content of the memory being matched. The sub-prototype is retrieved by attention weights W which are computed based on the similarity between the sub-prototypes in the memory items and the input-oriented embedding Z . To calculate the weight w i,j , we use a softmax operation:

where d ( · , · ) denotes cosine similarity measurement. As indicated by Eq. 1 and 3, the memory module retrieves the sub-prototype that is most similar to Z from each memory item in order to obtain the new representation embedding ˆ Z . As a consequence of utilizing the adaptive threshold addressing technique(Section 3.3.3), only the K can be utilized to obtain a task-oriented embedding ˆ Z , that serves to represent the encoded embedding Z .

3.3.3 Adaptive Threshold Technique for More Efficient Memory

Limiting the amount of sub-prototypes retrieved can enhance memory utilization and avoid negative impacts on unrelated sub-prototypes during model parameter updates. Despite the natural reduction in the number of selected memory items, the attention-based addressing mechanism may still lead to the combination of small attention weight items into the output embedding ˆ Z , which have negative impact on the classifier and sub-prototypes in the memory. Therefore, it is necessary to impose a mandatory quantity limit on the amount of the relevant sub-prototypes retrieved. To address this issue, we apply a adaptive threshold operation to restrict the amount of sub-prototypes retrieved in a forward process.

where ˆ w i,j = s i denotes the i, j = s i -th entry of ˆ w , the λ denotes the adaptive threshold:

Directly implementing the backward for the discontinuous function in Eq. 4 is not a easy task. For 178 simplicity, we use the method [17]that rewrites the operation using the continuous ReLU activation 179 function as: 180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

where max ( · , 0) is commonly referred to as the ReLU activation function, and ϵ is a small positive scalar. The prototype ˆ Z will be obtained by ˆ Z = ˆ W · M . The adaptive threshold addressing encourages the model to represent embedding Z using fewer but more relevant sub-prototypes, leading to learning more effective feature in memory and reducing the impact on irrelevant subprototypes.

3.4 Visualization and Interpretability

We denote f unfixed decode ( · ) : ˆ Z → ˆ X to represent the decoder. The decoder is trained to visualize what has been learned in the memory by taking the retrieved sub-prototype as input. From an interpretability perspective, each encoded embedding Z calculates the cosine similarity to find the topK fitting sub-prototype representation for the given input-oriented embedding. Then, these sub-prototypes are combined to represent the Z in ˆ Z . The sub-prototype in this process can be regarded as the visual description for the input embedding Z . In other word, the input image is much like the sub-classes represented by these sub-prototypes. In this way, samples with significant intra-class differences will be matched to different sub-prototypes, thereby distinguishing different sub-classes. The use of a reconstruction auxiliary task can visualize the sub-prototypes in memory to confirm whether our approach has learned intra-class differences for the annotated category. The results of this visualization are demonstrated in Figure 3.

3.5 Cycle-Consistent Alignment and Adaption

Once the sub-prototypes are mined through memory learning, the method of cycle-consistent matching, inspired by DCC [24], is employed to align the embedding ˆ Z . The cycle-consistent matching is preferred due to it can provides a better fit to the memory structure compared to other UniDA methods. The other method, One-vs-All Network (OVANet), proposed by Saito et al. [40], needs to train the memory multiple times, which can lead to a significant computational overhead. In brief, the Cycle-Consistent Alignment provides a solution by iteratively learning a consensus set of clusters between the two domains. The consensus clusters are identified based on the similarity of the prototypes, which is measured using a similarity metric. The similarity metric is calculated on the feature representations of the prototypes. For unknown classes, we set the size N of our memory during the initial phase to be larger than the number of possible sub-classes that may be learned in the source domain. This size is a hyperparameter that is adjusted based on the dataset size. Redundant sub-prototypes are invoked to represent the ˆ Z , when encountering unknown classes, allowing for an improved distance separation between unknown and known classes in the feature space.

Training Objective . The adaptation loss in our training is similar to that of DCC, as L DA :

where the L ce denotes the cross-entropy loss on source samples, L cdd denotes the domain alignment loss and L reg denotes the regularizer. For the auxiliary reconstruction task, we add a mean-squarederror (MSE) loss function, denoted as L rec . Thus, the model is optimized with:

4 Experiments

4.1 Datasets and Evaluation Metrics

We first conduct the experiments in the UniDA setting [47] where private classes exist in both domains. Moreover, we also evaluate our approach on two other sub-cases, namely Open-Set Domain Adaptation (OSDA) and Partial Domain Adaptation (PDA).

Datasets . Our experiments are conducted on four datasets: Office-31 [37], which contains 4652 images from three domains (DSLR, Amazon, and Webcam); OfficeHome

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

[46], a more difficult dataset consisting of 15500 images across 65 categories and 4 domains (Artistic images, Clip-Art images, Product images, and Real-World images); VisDA [33], a large-scale dataset with a synthetic source do- main of 15K images and a real-world target domain of 5K images; and DomainNet [32], the largest domain adaptation dataset with approximately 600,000 images. Similar to previous studies [14], we evaluate our model on three subsets of DomainNet (Painting, Real, and Sketch).

As in previous work [24, 41, 2, 4, 47], we divide the label set into three groups: common classes C , source-private classes ˆ C s , and target-private classes ˆ C t . The separation of classes

for each of the four datasets is shown in Table 1 and is determined according to alphabetical order.

Evaluation Metrics . We report the averaged results of three runs. For the PDA scenario, we calculate the classification accuracy over all target samples. The usual metrics adopted to evaluate OSDA are the average class accuracy over the known classes OS ∗ , and the accuracy of the unknown class UNK . In the OSDA and UniDA scenarios, we consider the balance between 'known' and 'unknown' categories and report the H-score [1]:

which is the harmonic mean of the accuracy of 'known' and 'unknown' samples.

Implementation Details . Our implementation is based on PyTorch [31]. We use ViT-B/16 [12] as the backbone pretrained by CLIP [35] for the MemSPM is hard to train with a randomly initialized encoder. The classifier consists of two fully-connected layers, which follows the previous design [4, 47, 41, 14, 24]. The weights in the L are empirically set as λ 1 = 0 . 1 , λ 2 = 3 and λ 3 = 0 . 5 fellow DCC [24]. For a fair comparison, we also adopt ViT-B/16 as backbone for DCC [24] and state-ofart method GLC [34]. We use the official code of DCC [24] ( https://github.com/Solacex/ Domain-Consensus-Clustering ) and GLC [34] ( https://github.com/ispc-lab/GLC ).

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

4.2 Comparison with State-of-The-Arts

We compare our method with previous state-of-the-art algorithms in three sub-cases of unsupervised domain adaptation, namely, object-specific domain adaptation (OSDA), partial domain adaptation (PDA), and universal domain adaptation (UniDA). In UniDA, we compare our method to previous universal domain adaptation approaches, which do not take into account the prior that private classes exist only in either the source domain (PDA) or the target domain (OSDA). Additionally, we compare our method to the OSDA and PDA baselines that consider the prior information unique to each sub-case.

Results on UniDA . In the most challenging setting, i.e. UniDA, our MemSPM approach achieves the state-of-the-art performance. Table 2 shows the results on DomainNet, VisDA and Office-31, and result of Office-Home is summarized in Table 3. We mainly compare with GLC and DCC using ViT-B/16 as backbone. On Office-31, the MemSPM+DCC outperform previous state-of-art method GLC by 3 . 7% and surpasses the DCC by 6 . 4% . On visda, our method surpasses the DCC by a huge margin of 16 . 1% . Our method also surpasses the GLC by 9 . 9% and the DCC by 4 . 5% on DomainNet. On the Office-Home, we surpasses the DCC by 9 . 8% and the GLC by 3 . 7% .

Results on OSDA and PDA . In table 4 and table 5, we present the results on Office-Home, Office-31 and VisDA under OSDA and PDA scenarios. In the OSDA scenario, MemSPM+DCC still achieves state-of-the-art performance. Specifically, MemSPM+DCC obtains 95 . 6% H-score on Office-31, with an improvement of 5 . 5% compared to GLC and 13 . 7% compared to DCC. In the PDA scenario, MemSPM still achieves comparable performance compared to methods tailored for PDA. The MemSPM+DCC surpasses the DCC by 8 . 1% on the VisDA.

4.3 Ablation Studies

Visualization with Reconstruction and tSNE We first visualize what the memory learns from OfficeHome by sampling a single sub-prototype and adapting an auxiliary reconstruction task: X → ˆ X . We also provide the tSNE of the ˆ Z which retrieving the most related sub-prototypes. The visualization is shown in Figure 3. The tSNE visualization depicts the distribution of sub-classes within each category, indicative of MemSPM's successful mining of sub-prototypes. The reconstruction visualization shows what have been learned by MemSPM, demonstrating its ability to capture intra-class diversity.

(a)

(b)

85

08

H-SCORE

75

70

20

30

85

targets

H-SCORE

0.0

1.0

2.0

3.0

5.0

4.0

65

6.0

7.0

8.0

60

128

256

5 12

1024

2048

9.0

(c)

1234567890

9888688

Alarm-Clock

Bed

Calculator

Bike

Effect of Memory-Assisted Sub-Prototype Mining . As the results shown in table 2, table 3, table 4 276 and table 5, the MemSPM+DCC evaluted on four benchmarks has surpassed the DCC on UniDA, 277 OSDA and PDA scenarios. The MemSPM can significantly improve the performance of the DCC 278 when using ViT-B/16 as backbone. The reason for utilizing the ViT-B/16 is that the memory module 279 of the MemSPM with huge latent space is initialized by randomly normal distribution, which make it 280 hard to retrieve the different sub-prototypes at early stages of training. So, we need ViT as backbone, 281 which have learned a more global feature space. 282

283

284

285

286

Sensitivity to Hyper-parameters . We conducted experiments on the VisDA dataset under the UniDA setting to demonstrate the impact of hyperparameters S and N on the performance of our method. The impact of S are shown in Figure 3. When S ≥ 20 , the performance achieve a comparable level. At the same time, the performance of the model is not sensitive to the value of N , when S = 30 .

287

5 Conclusion

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

References

Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In Proceedings of the European Conference on Computer Vision , pages 422-438, 2020.

Pau Panareda Busto, Ahsan Iqbal, and Juergen Gall. Open set domain adaptation for image and action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(2):413-429, 2018.

Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Partial transfer learning with selective adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2724-2732, 2018.

Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision , pages 135-150, 2018.

Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2985-2994, 2019.

Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 29512-29524. Curran Associates, Inc., 2022.

Jin Chen, Xinxiao Wu, Lixin Duan, and Shenghua Gao. Domain adversarial reinforcement learning for partial domain adaptation. IEEE Transactions on Neural Networks and Learning Systems , 33(2):539-553, 2020.

Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. Geometric anchor correspondence mining with uncertainty modeling for universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16134-16143, 2022.

Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Deep reconstruction-classification networks for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision , pages 597-613, 2016.

Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In Proceedings of the European Conference on Computer Vision , pages 268-283, 2018.

Zhihong Chen, Chao Chen, Zhaowei Cheng, Boyuan Jiang, Ke Fang, and Xinyu Jin. Selective transfer with reinforced transfer network for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12706-12714, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu. Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3937-3946, 2021.

Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In Proceedings of the European Conference on Computer Vision , pages 567-583, 2020.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1180-1189. PMLR, 2015.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030, 2016.

Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1705-1714, 2019.

Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401 , 2014.

Tzu Ming Harry Hsu, Wei Yu Chen, Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and YuChiang Frank Wang. Unsupervised domain adaptation with imbalanced cross-domain data. In Proceedings of the IEEE International Conference on Computer Vision , 2015.

JoonHo Jang, Byeonghu Na, Dong Hyeok Shin, Mingi Ji, Kyungwoo Song, and Il-Chul Moon. Unknownaware domain adversarial learning for open-set domain adaptation. Advances in Neural Information Processing Systems , 35:16755-16767, 2022.

Tarun Kalluri, Astuti Sharma, and Manmohan Chandraker. Memsac: Memory augmented sample consis353 tency for large scale domain adaptation. In Proceedings of the European Conference on Computer Vision , 354 pages 550-568, 2022. 355

356

357

358

Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G. Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , June 2019.

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10285-10295, 2019.

Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9757-9766, 2021.

Shuang Li, Chi Harold Liu, Qiuxia Lin, Qi Wen, Limin Su, Gao Huang, and Zhengming Ding. Deep residual correction network for partial domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(7):2329-2344, 2020.

Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Umad: Universal model adaptation under domain and category shift. arXiv preprint arXiv:2112.08553 , 2021.

Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng. A balanced and uncertainty-aware approach for partial domain adaptation. In Proceedings of the European Conference on Computer Vision , pages 123-140, 2020.

Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2927-2936, 2019.

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems , 31, 2018.

Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9111-9120, 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

Sanqing Qu, Tianpei Zou, Florian Röhrbein, Cewu Lu, Guang Chen, Dacheng Tao, and Changjun Jiang. Upcycling models under domain and category shift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021.

Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the European Conference on Computer Vision , pages 213-226, 2010.

Aadarsh Sahoo, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Select, label, and mix: Learning discriminative invariant feature representations for partial domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4210-4219, 2023.

Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 16282-16292. Curran Associates, Inc., 2020.

Kuniaki Saito and Kate Saenko. Ovanet: One-vs-all network for universal domain adaptation. In 411 Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9000-9009, 2021. 412

Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy 413 for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and 414 Pattern Recognition , 2018. 415

416

417

Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision , pages 153-168, 2018.

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

Tasfia Shermin, Guojun Lu, Shyh Wei Teng, Manzur Murshed, and Ferdous Sohel. Adversarial network with multiple classifiers for open set domain adaptation. IEEE Transactions on Multimedia , 23:2732-2744, 2020.

Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.08735 , 2018.

Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2720-2729, 2019.

Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang. Transfer learning with dynamic adversarial adaptation network. In 2019 IEEE International Conference on Data Mining , pages 778-786. IEEE, 2019.

Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8156-8164, 2018.

Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5031-5040, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve previous Universal Domain Adptation (UniDA) methods by further exploting the intra-class discrimination. For that, they propose a Memory-Assisted Sub-Prototype Mining (MemSPM) method. MemSPM learns to retrieve new task-oriented features given the input embedding features, and apply existing UniDA methods to the retrieving features. The paper also proposes an additional reconstruction task for the demonstration to the explainability of its proposed method as the authors claimed. Experiments on four datasets are conducted on three DA settings.\n\nSTRENGTHS:\nConsidering the effect of learning intra-class discrimination for UniDA is indeed an interesting idea to focus on, and such motivation is new in the UniDA community. By exploiting the intra-class structure, the proposed MenSPM is somehow novel to see.\n\nWEAKNESSES:\nAlthough the motivation from exploiting intra-class structure is interesting to UniDA, the analysis and the evidences to support the effectiveness of such idea is not enough. This is mainly due to the following concerns.\n\n1. Subclasses learning brings additional learning challenge and increases the learning cost to the problem, and not always the case that some classes have obvious subclasses, thus it is hard to say whether forcing subclasses learning would be beneficial to UniDA. To investivage this, I think it should have a solid analysis to the problem.\n\n2. The proposed method introduces too many hyper-parameters to the leanning process, inlcuding $N$, $S$, $K$, $\\lambda$, $\\lambda_1$, $\\lambda_2$, and $\\lambda_3$, etc., and there have not sufficient studies to investigate those hyper-parameters for different datasets or tasks. Note that this is important in UniDA since there is no validation set for model selection. Therefore, it is hard to say whether the effectiveness of the method may come from hyper-parameters tunning.\n\n3. Abalation studies are also not enough to understanding the effectiveness of different loss terms in Equation (8). Although improvements have shown when comparing to the DCC method, but to my knowledge with the CLIP models,  a simple baseline of standard training on source data only may already outperform the proposed method. However, this is not compared in the experiments.\n\n4. The results reported in the ResNet50 are meaningless since the proposed method do not run on this backbone. This is also a limitation of the proposed method. \n\n5. The experiments to verify the effectiveness of the proposed idea only conduct on the DCC method, which is not enough.\n\nThe authors claim that the proposed method could make interpretability from Figure 3, but I do not know how it works for the explainability since reconstruction does not imply interpretability. A random noise could also reconstruct the input.\n\nThe loss of $\\mathcal{L}_{cdd}$ is not illustrated in the paper. It is a bad way to let readers to understand it from other papers as it is not popular. \n\nSome typos exist in the paper, and please carefully check if some formulas are presented correctly, e.g., Equations (2), (6).\n\nQUESTIONS:\nAll weaknesses listed above should be well addressed to improve the paper.\n\nLIMITATIONS:\nThe authors have shown some limitations of the proposd method, but more should consider other that the method itself.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to exploit the intrinsic structures for each class, where sub-prototypes are devised to associate domain-common knowledge for universal domain adaptation. Specifically, MemSPM employs a memory module to mine sub-class information, and a corresponding reconstruction module to derive task-oriented representations. Experiments on representative benchmarks are conducted to verify the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1, This paper is generally well-written and easy to follow, and neat figures are presented to enable a more intuitive understanding. \n\n2, The motivation for decoupling with subclass structures seems reasonable.\n\n3, The technical details are well explained.  \n\n4, Surpassing previous methods with noticeable margins, justifying its effectiveness.\n\nWEAKNESSES:\nI think the main drawback of this paper lies in its presentations:\n\n1, Motivations of some designs are not well explained, i.e., why sub-prototypes benefits the universal scenario？ \n\n2, Some technical details seem missing. \n\nThe details of these concerns are presented in the ‘Questions’ part. \n\nMinors: \nPage 5 Line 179: missing space ''[17]that''\n\nQUESTIONS:\n1, Why can sub-prototypes benefit the universal domain adaptation scenario? \nI understand that, even within a domain, samples from the same class can be grouped into sub-classes. But, a critical part is missing why this helps the cross-domain association of common classes. which is the core problem for universal domain adaptation. An explanation or empirical justification is needed here, i.e., what is the pattern of retrieved sub-prototypes for common samples and private ones? \n\n2, Some technical details are not comprehensive enough. \n1) Is the memory learnable parameters? How to initialize them? This can be basic knowledge for people familiar with this, but it is still necessary to briefly detail this. \n2) After reading sec 3.5,  it is still unclear to be how the sub-prototypes help align the embeddings \\hat{Z}. \n\n3, In Fig. 1 (c), does this method assume the sub-class of two domains can be matched? This seems unrealistic under the distribution shift.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on Universal Domain Adaptation (UniDA), a practical DA setting that does not make any assumptions on the relation between source and target label sets. The goal is to adapt a classifier from source to target domain such that both source and target domains may have their own private classes apart from shared classes. The paper claims that existing UniDA methods overlook the intrinsic structure in the categories, which leads to suboptimal feature learning and adaptation. Hence, they propose memory-assisted sub-prototype mining (MemSPM) that learns sub-prototypes in a memory mechanism to embody the subclasses from the source data. Then, for target samples, weighted sub-prototype sampling is used before passing the embedding to a classifier, which results in reduced domain shift for the embedding. They also propose an adaptive thresholding technique to select relevant sub-prototypes. Finally, they adopt the cycle consistent matching loss objective from DCC [24] along with an auxiliary reconstruction loss for training. They show results on UniDA, Partial DA, and Open-Set DA using standard benchmarks like Office-31, Office-Home, VisDA, and DomainNet.\n\nSTRENGTHS:\n* The motivating ideas for the approach are interesting and intuitive. Further, the technical contributions are novel as well as effective.\n\n* It is intriguing that the auxiliary reconstruction task provides interpretability, which is usually not possible in existing DA solutions.\n\n* The paper is fairly easy to follow (with the exception of some equations and many typos and grammatical errors, see Weaknesses).\n\n* With their method and the advantages of a CLIP-pretrained ViT model, they achieve large improvements over existing ResNet-based methods. While they also show small improvements over some existing methods using the CLIP-pretrained model, this can serve as a new strong baseline for future UniDA work.\n\nWEAKNESSES:\n* The paper claims that existing UniDA works overlook the internal intrinsic structure in the categories. \n    * However, [W1] aims to resolve the same problem. [W1] proposes to learn lower-level visual primitives that are unaffected by the category shift in the higher-level features. And, in their proposed word-prototype-space, different visual primitives can be shared across domains and classes (including unknown classes).\n    * There is a significant overlap in the motivation given by this paper and that of [W1]. Consequently, the high-level conceptual novelty of this paper is overclaimed. However, I do believe that these conceptual ideas are interesting as well as important for UniDA.\n    * Please discuss the similarities and differences (both in terms of motivation and the actual approach) of this paper w.r.t. [W1].\n    * Another paper with similar conceptual ideas is [W2].\n\n* This paper lacks some mathematical rigor.\n    * Eq. 1, 2: $\\hat{Z}=W\\cdot M$ is shown as matrix multiplication (I assume that it is not element-wise multiplication since dimensions of $W$ and $M$ are different), but the expansion of this matrix multiplication contains an arg-max over the elements of $W$. Then, it does not make sense for the overall computation to be a standard matrix multiplication.\n    * Eq. 1, 2: the text mentions that $s_i$ is the index of sub-prototypes in the $i^\\text{th}$ item but Eq. 2 implies that $s_i$ is a particular dimension found with arg-max. This seems contradictory and is confusing.\n    * Eq. 2: Use $\\mathop{\\arg\\max}_{j}$ instead of using `dim=1` since it is a mathematical equation and not the code implementation.\n    * Eq. 5: It is unclear which dimension is used for top-$k$\n    * Eq. 6: It should be $\\max(... , 0)$ instead of just $\\max(...)$.\n\n* The requirement of a CLIP-pretrained backbone is very restrictive since the method cannot be extended to other settings (like medical imaging) where the CLIP-pretraining may be suboptimal. While the paper shows comparisons where prior methods use the CLIP-pretrained model, it should also show comparisons when starting from a random initialization as well as the more widely used ImageNet initialization.\n    * The paper claims that a CLIP backbone is needed to retrieve sub-prototypes in early iterations. Why not start retrieving sub-prototypes after a few epochs of normal training?\n\n* L135: “eliminates the domain-specific information from the target domain”. This is a very strong claim which does not seem to be backed by evidence. Performing “domain alignment” is not the same as “eliminating” domain-specific information. Further, as we can see from Fig. 3, the sub-prototypes seem to be retaining domain-specific information.\n\n* There are no sensitivity analyses for the several loss-balancing hyperparameters $\\lambda_1, \\lambda_2, \\lambda_3$ (not even in the Supplementary). While the paper claims to have borrowed them from DCC, this approach is vastly different from DCC, and we need to check for sensitivity to these hyperparameters. Further, DCC does not have a reconstruction loss, so it is unclear how that hyperparameter is selected.\n\n* There is no ablation study for the adaptive threshold $\\lambda$. It should be compared to various fixed thresholds and the value of the adaptive threshold should also be plotted over the course of training to obtain more insights into its working.\n\n* Other UniDA works, like OVANet [40] and [W1], study the sensitivity of their methods to the degree of openness (i.e. the number of shared/private classes) which changes the difficulty of the UniDA problem. This analysis is missing in this paper. This should be shown for a better understanding of the capabilities of the proposed method.\n\n* Some more related work [W3-W4] on Open-Set DA and UniDA (apart from [W1, W2]) that is not discussed in this paper.\n\n* Minor problems (typos):\n    * L53: “adaption” → “adaptation”\n    * L59: “shifts” → “shift”\n    * L92: use `unknown’ i.e. use a backquote in LaTeX for it to properly render the opened and closed quotes like in L102. \n    * L119: use math-mode for K in top-$K$.\n    * L124: “varies” → “vary”\n    * L126, 179: add space between text and \\cite{...}\n    * L134: “differenciates $\\hat{Z}$ with” → “differentiates $\\hat{Z}$ from”\n    * L151: “max” → “maximum”\n    * L166: “only the $K$” → “only the top-$K$”\n    * L181: “$max$” → “$\\max$”\n    * L244: “fellow” → “following”\n\n* Minor problems (grammatical errors):\n    * L32: “aims” → “aiming”\n    * L40: “Since such kind” → “Since this type”\n    * L41: “almost happens in all the” → “occurs in almost all of the”\n    * L59: “embedding give into” → “embedding is passed to” \n    * L125: “sometimes is” → “is sometimes”\n\n### References\n\n[W1] Kundu et al., “Subsidiary Prototype Alignment for Universal Domain Adaptation”, NeurIPS22\n\n[W2] Liu et al., “PSDC: A Prototype-Based Shared-Dummy Classifier Model for Open-Set Domain Adaptation”, IEEE Transactions on Cybernetics, Dec. 2022\n\n[W3] Chen et al., “Evidential Neighborhood Contrastive Learning for Universal Domain Adaptation”, AAAI22\n\n[W4] Garg et al., “Domain Adaptation under Open Set Label Shift”, NeurIPS22\n\nQUESTIONS:\nPlease see the weaknesses section. \n\nOverall, the technical contributions seem to be novel and intuitive. However, there are significant concerns regarding missing discussions on highly relevant work [W1], lack of mathematical rigor, missing sensitivity analyses and ablation studies, and the restrictiveness of requiring a CLIP-pretrained backbone. Hence, my rating is “4: borderline reject” at this time but I am willing to update my rating based on the rebuttal and discussion.\n\nLIMITATIONS:\nI appreciate that the paper provides both limitations and broader societal impact discussions in the Supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes a Memory-Assisted Sub-Prototype Mining (MemSPM) method that can learn the differences between samples belonging to the same category and mine sub-classes when there exists significant concept shift between them.

STRENGTHS:
The writing of the article is very good. Graphical expressions such as t-SNE are very clear. The method have achieved relatively high classification H-score.

WEAKNESSES:
Some training details need to be explained, such as the selection of hyperparameters. How to adjust the N, S and lambda, and what criteria are based on? If it is based on the final experimental effect, it also indirectly depends on the label information of the target domain.
The scalability of the method is relatively poor. If the data set is large and there are many categories, will there be many prototypes required, and how will the method perform? It is crucial to have the Domainnet dataset in the experiments.

QUESTIONS:
mainly of the weaknesses.

LIMITATIONS:
This paper has no limitation sections.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of universal domain adaptation by focusing on the intra-class structure within categories, which is often overlooked by existing methods.\n\nThe main contribution is the proposed Memory-Assisted Sub-Prototype Mining (MemSPM) method, which learns the differences between samples belonging to the same category and mines sub-classes in the presence of significant concept shift. By doing so, the model achieves a more reasonable feature space that enhances transferability and reflects inherent differences among samples.\n\nExperimental evaluation demonstrates the effectiveness of MemSPM in various scenarios, achieving state-of-the-art performance on four benchmarks in most cases.\n\nSTRENGTHS:\nS1 : The primary contribution of this work is the introduction of sub-prototypes, learned from samples within the same category but exhibiting significant concept shift.   The utilization of sub-prototypes allows for a more fine-grained adaptation process, which is an intuitive and an interesting idea.  The ablation experiment Figure 3 (graph), supports the notion that mining sub-prototypes is indeed advantageous, as increasing the number of sub-prototypes (S) leads to a substantial performance improvement, from approximately 62% (with one sub-prototype per category) to around 80% (with 40 sub-prototypes per category). \n\nS2: The results presented in Table 2 and Table 3 demonstrate significant performance improvements compared to previous works, with increases of +4.5% and +6.4% in H-score on DomainNet and Office-31 datasets for UniDA scenario. Additionally, there is a +1.6% improvement in H-score on the Office-Home dataset. It should be noted that the comparisons are not entirely apples-to-apples, as discussed in the weaknesses section.\n\nWEAKNESSES:\nW1: The utilization of CLIP-based embedding as mentioned in line 126 offers semantic capabilities that generalize across various domains (as shown by works such as [1, 2, ..] that build on top of CLIP). However, the importance of using CLIP-based embedding is not clearly demonstrated in the ablation analysis. A comparison between CLIP-based embedding, learned embedding (without pre-training), and ViT-B/16 (pre-trained on ImageNet) would provide valuable insights. Additionally, the lack of utilization of CLIP's semantic capabilities in prior works raises concerns about the apples-to-apples comparison of the results presented in Table 2 and Table 3.\n\nW2: From the experiment section, the impact of different losses, such as cross-entropy (L_ce), domain alignment loss (L_cdd), and auxiliary reconstruction task (L_rec), on model performance is not clearly explained in the experiment section. Understanding the contribution of each loss would enhance the understanding of the paper.\n\nW3: The sensitivity of hyperparameters across different scenarios, such as Open-Set Domain Adaptation (OSDA) and UniDA, is not adequately addressed in this section. Investigating the sensitivity of hyperparameters would provide valuable insights into their impact on model performance.\n\nW4: Section 3.3.3 discusses the \"Adaptive Threshold Technique for More Efficient Memory,\" but there is a lack of experimental details showcasing the memory efficiency of this technique. Without such evidence, it becomes challenging to fully appreciate the technical contribution.\n\nW5: While the motivation and the main idea of mining sub-prototypes are novel, it is worth noting that memory-based prototype mining was explored earlier in works like [3]. This observation slightly diminishes the overall technical contribution..  \n\nW6: Supplementary material Figure 1 reveals that a significant portion (>60%) of the sub-prototype visualizations are not interpretable. This undermines the contribution of interpretability in this work. \n[1] Rinon Gal and Or Patashnik and Haggai Maron and Gal Chechik and Daniel Cohen-Or StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators, ACM Transactions on Graphics\n[2] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, René Ranftl, Language-driven Semantic Segmentation, ICLR 2022\n[3]Tarun Kalluri , Astuti Sharma, Manmohan Chandraker.\\ MemSAC: Memory Augmented Sample Consistency for Large Scale Domain Adaptation, ECCV 2022\n\nQUESTIONS:\nPlease refer the weaknesses section for the related questions that need more clarification.\n\nLIMITATIONS:\nA notable limitation of the study is the lack of clarity regarding the contribution of various components of the proposed method to the overall performance. Specifically, the impact of CLIP-based embedding, which has demonstrated generalizable capabilities even in zero-shot scenarios across domains, needs to be thoroughly understood to fully appreciate the proposed components. Gaining insights into the individual contributions of different components would provide a deeper understanding of their influence on the overall performance. Further investigations or additional analyses focusing on these aspects would enhance the comprehensiveness and rigor of the study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}