Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrieval
Authors: Lixu Wang
Xinyu Du
Qi Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Retrieval
Domain Adaptation
Primary area: machine_vision
TL;DR: We identify an unexplored but important problem called Universal Unsupervised Cross-Domain Retrieval, and propose a two-stage semantic feature learning framework to address it.

Abstract: Cross-domain retrieval (CDR) is finding increasingly broad applications across various domains. However, existing efforts have several major limitations, with the most critical being their reliance on accurate supervision. Recent studies thus focus on achieving unsupervised CDR, but they typically assume that the category spaces across domains are identical, an assumption that is often unrealistic in real-world scenarios. This is because only through dedicated and comprehensive analysis can the category composition of a data domain be obtained, which contradicts the premise of unsupervised scenarios. Therefore, in this work, we introduce the problem of **U**niversal **U**nsupervised **C**ross-**D**omain **R**etrieval (U^2CDR) for the first time and design a two-stage semantic feature learning framework to address it. In the first stage, a cross-domain unified prototypical structure is established under the guidance of an instance-prototype-mixed contrastive loss and a semantic-enhanced loss, to counteract category space differences. In the second stage, through a modified adversarial training mechanism, we ensure minimal changes for the established prototypical structure during domain alignment, enabling more accurate nearest-neighbor searching. Extensive experiments across multiple datasets and scenarios, including close-set, partial, and open-set CDR, demonstrate that our approach significantly outperforms existing state-of-the-art CDR methods and other related methods in solving U^2CDR challenges.

Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrieval
Lixu Wang
Xinyu Du
Northwestern University, IL. USA
General Motors Global R&D, MI,USA xinyu. du@gm. com
edu
Zhu
Northwestern University, IL. USA gzhu@northwesteenn.c edu
Abstract
Cross-domain retrieval (CDR) is finding increasingly broad applications across various domains. However, existing efforts have several major limitations, with the most critical being their reliance on accurate supervision. Recent studies thus focus on achieving unsupervised CDR, but they typically assume that the category spaces across domains are identical, an assumption that often unrealistic in real-world scenarios. This is because only through dedicated and comprehensive analysis can the category composition of a data domain be obtained, which contradicts the premise of unsupervised scenarios. Therefore, in this work, we introduce the problem of Universal Unsupervised Cross-Doman Retrieval (U²CDR) for the first time and design two-stage semantic feature learning framework to address it. In the first stage, a cross-domain unified prototypical structure established under the guidance of an instance- prottttter-ttpee--ee.eoo mixedeiixeeeeeee contrastive loss and a semantic-enhanced loss, to counteract category space differences. In the second stage, through a modified adversarial training mechanism, ensure minimal changes for the established prototypical structure during domain alignment, enabling more accurate nearest-neighhor searching. Extensive experiments across multiple datasets and scenarios, including close-set, partial, and open-set CDR, demonstrate that our approach significantly outperforms existing state-of-the-art CDR methods and other related methods in solving U2CDR challenges.
Introduction
In real-world application, cross cross-domain retrieval (CDR) finds extensive utility across diverse domains, such image search [I]. product recommendations 2 and artistic creation 3 4
However, the
efficacy of current CDR methods relies heavily on accurate and sufficient supervision 5 to provide categorical cross-domain pairing labels. The acquisition of such information demands costly efforts and resources. Hence, there is an urgent need to develop unsupervised CDR techniques.
For the regular Unsupervised CDR (UCDR) problem 7 there are two data domains with semantic similarity but distinct characteristics: the query domain and the retrieval domain. Despite the absence of category labels, regular UCDR typically assumes that the label spaces of both domains identical. However, in real-world applications
[5], the categorical composition of an unlabeled data domain is usually uncertain, which is hard to acquire without detailed analysis and dedicated expertise. In this work, we focus on extending UCDR to more universal scenarios, which allow for the possibility of disparate category spaces across domains. The objective of this Universal UCDR CDR) problem is to retrieve samples from the retrieval domain that share the same category label with a query sample
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Intra-Domain Samantic Learning
Cross-Domann Domain Semantic Semantic-Mtshhed Learning
Cross
A
Doran D
Farau
Marped
Mamory Bank
Stared
Reprevertatio
Figure 1: Overview of our proposed UEM semantic feature learning framework for CDR. In the first stage of Intra-Domain Semantic-Enhane across domains, which is driven and enhanced by instance- pawancce-rooooe-mmiiiie semantic-enhanced loss. In the second stage of Eross-Domain Semantic-Matched Learning, Semantic- Preserving Domain Alignment aligns domains while preserving the built prototypical structure, and Switchable Nearest Neighboring Match achieves more accurate cross-domain categorical pairing.
Learning, UEM establishes a unified prototypical structure
contrastive loss and
from the query domain. Naturally, in the case of private categories exclusive to the query domain, the retrieval result should be null.
Two issues must be addressed in solving traditional UCDR: 1) effectively distinguishing data samples in each domain, and 2) achieving alignment across domains for samples of the same category. For the first issue, self-supervised learning [ [0] (SSL) is employed independently within each domain. For the second, many nearest-neighbo searching algorithms may apply. However, for UºCDR, applying these existing methods introduces
challenges. First, the prevailing SSL methods, particularly contrastive learning 9 11 10], are highly influenced by the category label space [2],, which means different label spaces lead to distinct semantic structures. Second, existing nearest-neighhor searching algorithms 13 14 7 overlook the presence of domain gaps. We found that only by first addressing the domain gap can the nearest neighbor searching become reliable and accurate.
Thus, to effectively address the above challenges in solving U²CDR, we propose a two-stage Unified, Enhanced, and Matched (UEM) semantic feature learning framework, in Figure | In the first stage, we establish a cross-domann unified prototypical structure with an eestance-prototypee contrastive loss, accompanied by semantic-enhanced loss (SEL). In the second stage, before con- ducting cross- doos--domain category alignment, we incorporate Semantic- -reuntic-reeerving Domain Alignment (SPDA) diminish the domain gap while ensuring minimal changes for the established prototypical structure. As the domain gap diminishes, we propose Switchable Nearest Neighhoring Match MSN'M), to select more reliable cross-domain neighbors based the relationship between instances and prototypes. Extensive experiments and ablation studies on popular benchmark datasets demonstrate that our method can substantially outperform state-of-the-ars methods from UCDR and other related problems, In addition, we also theoretically analyze the principles behind the major challenges of CDR and the design intuition of UEM. In summary, this work made the following contributions:
(IPM)
We are the first to identify and solve an important problem when employing UCDR in practice- Universal UCDR (UCCDR), where the category spaces of different domains are distinct.
We propose a two-stage Unified, Enhanced, and Matched (UEM) semantic feature learning framework to solve C²DDR. In the first stage, UEM establishes a unified prototypical structure across domains, ensure consistent semantic learning under category space differences. In the second stage, UEM achieves more effective domain alignment and cross- noos-domain pairing.
We conduct extensive experiments on multiple benchmark datasets, with settings including Close- set, Partial, and Open-set UCDR. The results demonstrate that UEM can substantially outperform state-of-the-an methods of UCDR and other potential solutions in all settings.
2
Related Work
Cross-Domain Retrieval (CDR) is not very difficult to achieve if there are categorical labels [5] However, in real-world applications, such categorical labeling information is hard to acquire, thus

---

focus on achieving unsupervised CDR (UCDR). CDS 14 proposes a contrastive learning-basee based cross-domain pre- pre-training to align different domains. PCS incor- porates prototype contrastive learning [1] into the cross-domain pre-training. Recent studies also search ways like clustering [7]. pseudo-labeling [8]. classifier mixup [7, and data augmentation to achieve more advanced CDR. However, all these UCDR works assume that the query and retrieval domains share the same category space. Although there is a study [19] that can achieve CDR with distinct categories, its effectiveness relies on accurate and sufficient data labels.
more recent works 7 1 17
Universal Cross-Domain Learning. Cross-domain learning consists of domain adaptation (DA) and domain generalization (DG) [20]. Regular DA and DG also only consider the scenario where the label space of the target domain the same as the source label space, which is termed Close- set DA/DG. Recently, more studies have realized that the target label space may be subset of
the souree one (i.e., Partial DA/DG) [21
22 or contain some private labels that other domains 24 To deal with more universal setups, UniDA (25
do not have (i.e., Open-set DA/DG) 23
unifies entropy and domain similarity to quantify sample transferability across domains. CMU 26 extends transferability quantification into entropy, consistency, and confidence. More recent works search ways like clustering [27 28 and nearest neighbor matching [29 30) to achieve universal DA/DG. In addition, some studies appear to achieve unsupervised DG where the source domain is also unlabeled [1 B2), which is similar to the setup of UCDR. However, these studies consider the classification task and cannot effectively work in image retrieval, especially in completely unsupervised cases.
3 Methodology
3.1 Problem Formulation
In the problem of CDDD we assume there are two domains characterized by N and ' unlabeled instances, which are denoted DA= domains are provided unlabeled data without category labels, we assume their label spaces y 7 consist of CA and CB different categories, and there is a relationship that C + C ): n n of )A U Without losing generality, if we regard domain A as the query domain, while domain B is the retrieval domain, the objective of U2 CDR is to retrieve correct data from domain B that belongs to the same categories the query data provided by domain A. To achieve this objective, it is required to train a valid feature extractor fo : X + R that can map both these two domains from the input space X' feature space R. Then the retrieval process R(fo. xx²) is shaped like for particular query instance æ; with the label y from domain A, the representation distance between all instances in domain and æ:A needs to be calculated to form set, i.e., S ((r)))))) (a)))) where d(-) a particular distance metric (e.g., Euclidean Distance), and we have
= ( J and DB = {=:">>" y respectively. Although these two
null, yf E )' \ (S) k], otherwise,
R(fo, )²)
where sort, (.) means ascending order sorting, and [1 ki] denotes the first k elements of a set.
To solve UCDR,, we propose Unified, Enhanced, and Matched (UEM) semantic feature learning framework that consists of two stages - Intra-Domain Cemantic-Enhancel (IDSE, Section 3.2p Learning and Cross-Domale Semantic-Matche Matched (CDSM, Section 3.3) Learning, which is shown in Figure I IDSE can help the feature extractor fe to extract categorical semantics and ensure unified semantic structure across domains at the same time, which is achieved by instance- prototype-mixed (IPM, Section 1 contrastive learning and a novel Semantic-Fnhanced Loss Domain Alignment (SPDA,
Method Overview.
(SEL, Section 3.2.2p. After IDSE, CDSM conducts Section 3.3.) to minimize the domain gap while preserving the semantic structure learned by IDSE. With the minimization of the domain gap, more accurate nearest-neighhor searching can be achieved by our Switchable Nearest Neighboring Match (SX"M, Section 3.3.2
3.2 Intra-Domain Semantic-Fnhancee Learning
To achieve effective cross-domain retrieval, feature extractor fo needs to learn consistent cross- domain features to differentiate data categories. Instance Discrimination [9] is usually employed to
3

---

achieve discriminative feature learning, but directly applying it in U²CDR has four fundamental issues that hinder the possibility of accurate cross moss-domain categorical matching later:
1) Instance discrimination tends to extract semantics that separate domains rather than categories,
d(f ), (=])) < d(f(a(); f(=))), uf = # v
(2)
2) Instance discrimination cannot characterize categorical semantics in the feature space,
d( a(f(=)), of = = # (,
(3)
d(a)
3) The randomness introduced by stochastic data augmentations results in evident changes in learned categorical semantic structures during training, i.e.,
d(C(PA)) G(PA))
min ((G(P)), PA-PA-
(4)
where G(-) corresponds to graph constructed by the input vectors, and P denotes the set of categorical prototypes for a domain. The subscript denotes different training iterations, while H represents the hypothesis space of possible categorical prototype sets for a particular domain. d(.)
here is measurement for graph difference, e.g., graph edit distance (33
4) Distinct label spaces make instance discrimination learn different categorical semantic structures:
Theorem 3.1 (Geometry Distinctness). Suppose data distributions of two domains (A and B) have muiually disjoint supports, and they are uniform over these supports. Assuming the support sets of domains A and B are not identical, the optimal feature extractors f* that minimize the instance discrimination loss of different domains present distinct geometric feature spaces.
3.2.1 Instance-Proooyyp--Miiell Contrastive Learning.
To fix the above issues, we adopt a slowly momentum-updatte contrastive learning algorithm MoCo [TO). to handle the third issue reflected by Eq. (4) Moreover, the MoCo-based instance discrimination is conducted separately for each domain, which encourages fo to focus less on learning domain semantics (for the first issue, Eq. (2). Besides, we accompany MoCo with
prototypical contrastive loss, to enhance the mapping of categorical semantics from the input space to the feature space (for addressing the second issue, Eq. (3). With well-crafted prototype update mechanism, this prototypical contrastive loss can also help build a unified semantic structure across domains (for the last issue, Theorem 3.I). Then let introduce IPM contrastive learning in detail. First of all, two memory banks M and AM are maintained for domains A and B, which store historical features m of data samples
MA A | x .B = [m m R where m, + Bm, + (1 - ())((a,) (5)
Here m, is initialized by the feature of æ; extracted by the initial fo and updated in momentum, where 3 controls the momentum speed, and we set it as popular value 0.99. With these two memory banks, MoCo builds the positive pairs the pair of each instance and its historical feature, while the negative ones are pairs of each instance and the historical features of all other instances:
exp (Jo(ws) m./T)
CINCE Σ log
(6)
Σラ exp (fa(z:) m,/r)"
exp
where B is the batch size and T is temperature factor that is set as 0.07.
As for the design of our prototypical contrastive loss, K-Means is applied on MA and M to construct prototypes as cluster centers P = ( In our problem, the cluster number C is unknown, thus we apply the Elbow approach [34] to estimate as Ć. Then, for each instance x; it belongs the e,-th cluster, the prototypical contrastive loss LPNCE shapes like.
B
exp (fe(æ.) Pc,/T)
LPNCE
log
Ee _e_exp exp (fa(xx)) po/T)
Until here, the first three issues can be fixed by mixing LINCE and LINCE, but the last issue, Theorem 3.1 is the bottleneck of CDDDR Next, let us introduce how we build unified prototypical

---

structure for LPNCE to address the last issue. Specifically, after obtaining the prototype sets PA pn of domain A and B, if take domain an example to illustrate the prototypical structure building process, the prototype set PB of domain B will be translated to domain A as,
PB-A Toーーー=p
M C"
Jcl:
(8)
where M denotes the average vector of all vectors in M. Next, each prototype p PA searches its closest pă-AA E pDB+A (we use Hungarian algorithm to search the closest cross-domain prototypes) for the opportunity of merging, which needs to satisfy the condition,
d(p PA) min min d(p.. Pj), min d(p, PiP..P))
PLPPEEPA
PL,PEEPB
where -(:,-) computes the Euclidean distance. If we use the symbol + to identify pro- totypes that satisfy this merging condition, the final prototypical structure for domain is pA = (pA (pr.) (PB-A ( U (PA. @ pBBBA. where (PA.0 @ PB-+A.8) =
Then the computation of LINCE Eq. 77) for domain A is conducted { B-A + p /2 c=1
on the newly-built PAA, and all these operations are same for domain B.
However, establishing the semantic cluster-like structure requires time, it is unreasonable to conduct prototype contrastive learning from the beginning of training. Therefore, we conduct instance discrimination at the beginning and progressively incorporate LPNCE. In this case, not only are the constructed cluster centers more reliable, but the Elbow approach also provides more accurate cluster number estimations. Specifically, we use a coefficient a that is scheduled by a Sigmoid function to control the incorporation weight of LPNCE, ie.,
LIPM = LINCE + aLPNCE, where a = 1 + exp (0.5E e)*
(10)
E and here are the overall training epochs and the current epoch for IDSE.
3.2.2 Semantic-Enhanced Loss.
For the IPM contrastive learning, it is arbitrary to allocate a data instance æ;; to a single cluster when the preferred semantic prototypical structure cannot be learned in advance. As result, to speed up the cructure-buildiig process, we propose a novel with the prototypes better. Specifically, instead of assigning data instances with single cluster, SEL considers potential semantic relationships between instances with all clusters, which are measured by the Softmax probability. Moreover, as both PA and p are obtained by the Euclidean distance-based K K-Means, we directly minimize the Euclidean distance between samples and prototypes:
Ennhacce Loss (SEL) to align data instances
1 LSEL
exp(So(x:; Po/T)
(11)
B
Σ exp(folæ;) Pc/T)
where Ö denotes the number of prototypes after merging, e.g., C is the number of elements in
PAA. By taking all potential semantic correlations into account, SEL alleviate the impact of the noise within the K-Means clustering results and further guide the model to lear more distinguishable semantic information in terms of Euclidean distance. Certainly, such SEL benefits also rely on high-quality semantic prototypical structures. As a result, we also apply the progressive coefficient SEL, then the final optimization objective for IDSE is
LIDSE (Cft + CPPM) a + L.
(12)
3.3 Cross-Domain Semantic-Matcher Learning
3.3.1 Semantic-Preserving Domain Alignment.
Domain invariance is another requirement for the extracted features UEM, However, is difficult to effectively align feature clusters across domains when category label nor correspondence
5

---

Standard Adversarial Learning
Semandic Preserving Domain Alignment
Incorrect NN before Domain Alignment
V.S.
Domain Alignment
Figure 2: Comparison of nearest neighbor searching before or after domain alignment.
Figure 3: Comparison between standard adversarial learning and our semantic- semantic-preserving domain alignment in terms of semantic structure changes.
annotation can be utilized in CDD Instance matching 32 [3] is proposed to match an instance x; another instance x;) in the other domain with the most similar features. However, due the domain gap, instances can be easily mapped mistakenly. For example, if there is an instance in one domain that is extremely close the other domain, will be determined as the nearest neighbor for all instances in the other domain [13. shown in Figure 2 As result, before conducting instance matching, the domain gap needs to be diminished. Existing works usually leverage discrepancy minimization [35] or adversarial learning (36] to achieve domain alignment. However, these methods provide inferior performance due to semantic categorical structure changes, i.c., the semantic correlations among instances within domains change great deal during domain alignment, shown in Figure X
To achieve more effective domain alignment, especially with semantic preservation, we propose Domain Alignment (SPDA). Similar to the standard domain adversarial learning, SPDA happens on two parties, one is the feature extractor fa, and the other is a domain classifier ga parameterized The domain classifier tries to distinguish the representations of two domains, while the feature extractor tries to fool the domain classifier. Thus, the training is shaped like bi-level optimization in terms of the domain classification task on & and w, shown follows,
2B
DDLL-
Σ 1=1
y4 log Iw (fo(z:)) - (1 -y) loogll1 gw (fo
(13)
where y here denotes the domain label, e.g., if we regard y for domain A, the domain label of domain B is y O. After sufficient adversarial training, the feature extractor captures nearly domain-invariant features, thus achieving domain alignment.
To prevent the semantic structure from being changed, first make a copy for the model trained by IDSE and denote it as fg. Then for a mini-batch of a particular domain, we feed all instances to fg and calculate pair-wise cosine similarity and Euclidean distance. In this way, all these instances constrain and influence each other, which means that when the correlation a particular instance pair changes, will affect the correlations of other related pairs. Then, apply pemeeric-resserving regulation into domain adversarial learning to make the pair-wise correlations unchanged,
B
fo(a,) fo(a,)
(a for (xx)
CSPR B
ΣΣ1
fo(z:))lfo(z)
(x,) for (x,
(14)
+ [d(fo(z)), fe( - difo (xx)) for (xx;)))
Recall IPM Contrastive Learning. Actually, aligning two domains together without any semantic structure change is impossible. As result, there is need for dedicated design to alleviate the impact of such unavoidable changes. Our solution is to strengthen the instances' semantic correlations by enhancing the cluster's inner density and inter-separability. For the final convergence of IPM contrastive learning, each instance is optimized to get as close to its corresponding cluster prototype as possible, and as far other cluster prototypes as possible:
Theorem 3.2 (Convergence of IPM). Suppose the data distribution of domain has mutually disjoint supports, and it is uniform over these supports. Simplex Equiangular Tight Frame (ETF) representations (37) minimize the Destance-Protoyyee Mixed Loss of this domain.

---

3.3.2 Switchable Nearest Neighboring Match.
With SPDA, the domain gap could be effectively minimized to enable more accurate cross- domain instance matching. However, existing instance matching approaches 7 13 32 lack the capability of measuring the matching reliability between instance and its nearest neighbor, which allows us to conduct cross-domain matching with different weights. For example, an instance is located the joint boundary of multiple categories, which indicates that the current feature extractor cannot extract sufficiently distinguishable semantic features for this instance, we are supposed to lay less emphasis on this case. To fix such issues, propose the Switchable Nearest Neighboring Match (SN"M).
The principle behind SNAM is that prototypes are more convincing and reliable. Specifically, for particular sample 2² in domain A, we first determine its inner nearest cluster prototype p in domain A. We can also search for the nearest instance z²A.B.B in domain B. Both these two searching processes are based on the product of a modified cosine similarity and Euclidean distance,
fo(wxt p [fo(
A = arg min Pa
-((o(aa))
(15)
fo(xxt) fo(a))) [Fo(at))lfo({{1)
((lo(m)))
2 arg min
(16)
After obtaining x; A.B SN searches for its inner nearest prototype Pa; A.B' PR (which has been merged with the translated prototype set pA-B from domain A) and two cases allows us to measure the reliability of A.A. Before introducing these two cases, the inner nearest prototype P of x needs to be translated to domain В and checked whether should be merged to follow condition Eq. (9). and we denote the translated prototype p.. Then the first potential case A.B' exactly identical to pa which means, z; is convincing since it shares the same prototype correlations with æ across two domains. Then the pair of A and æ; should be viewed positive pair in the contrastive loss. Otherwise, p A.B' is different from P it may be located at the intersection region of multiple clusters. In this case, the pair of A and x; is not supposed to be treated as positive pair. However, it does not mean SM does nothing for these unreliable cases, instead, SN leverages modified prototype contrastive loss match the pair of x. and P The modification is to incorporate cross-domain instance- wise negative comparison,
B
CSNTM-
1
Σ 1=1
log
B
ÖB Een exp(So(x()) /T) En exp(Jo(x)) So(aBB/T)
Na
(17)
exp(Je (m= p /T) if A.B'
where A =
exp(Ja (a
/T) (fo((f((=))) (a fo( )/T), otherwise.
Finally, the overall optimization objective of CDSM follows
LCDSM LDAL SSR LB + C
SN SNPM
(18)
4 Experiments
The datasets, experimental settings, and comparison baselines are introduced below. More implemen- tation details, experiment results, and source codes are provided in the Supplementary Materials.
Datasets. Office-31 38 includes three domains with 31 classes: Amazon (A), DSLR (D), Webcam (W). Office-Home Home 39 contains four different domains: Art (A), Clipart (C), Product (P), Real (R). And each domain has 67 data categories. DomainNer [40] is the most challenging cross-domain dataset our best knowledge, which includes six domains: Quickdraw (Qu), Clipart (CI), Painting (Pa), Infograph (In), Sketch (Sk) and Real (Re). DomainNet originally class-imbalanced, thus we follow [7] to select data classes that contain more than 200 samples.
Experiment Settings. For fair comparison, we apply ResNet-50 4 pre-trained with ImageNet in MoCov2 [10] as the feature extractor. The domain classifier consists of two fully-connected layers.
7

---

The SGD optimizer with momentum of 0.9 is adopted with an initial learning rate of 0.0002 that is scheduled to zero by a cosine learning strategy. The batch size is 64. The training epochs of IDSE are 100 for Office-31 and Office-Home, and 200 for DomainNet. The epoch number of CDSM is 50 for all three datasets. Following 17,, we adopt mean average precision on all retrieved results (mAP@ All) to measure the performance. All experiments are run repeatedly 3 times with seeds 2024, 2025, and 2026, and we report the mean performance and standard deviation.
Comparison Baselines. Our proposed method is compared with a comprehensive set of state-of -ftte- art works from Cross-Domain Representation Learning (CDS [4]. PCS (13), Unsupervised Domain Generalization (DARL [31] DN2A (32) ). and Unsupervised Cross-Domain Retrieval (UCDIR 77.
CoDA 17. DGDIR 8 We follow their default settings and only conduct compulsory customization.
Table Performance comparison (mAP@ All) between ours and other baseline methods on Office-31 and DomainNet in Close-set Unsupervised Cross Dosss-D Retrieval. We blue and bold the best performance, and bold the second best, same for all tables.
Methods A-D A-W D-*A D-W W-A W-+ Avg.
Qu-~CI CI-+Pa Pa->In In-SSk Sk-+R Avg.
CDS
66.7=1.1 20.9 70.9=1.0 90.0 64.4 88.4 73.8
19.2 35. 35.
24,4 25,5 32.3 27. 23..
PCS
72.7=2.a 70.7.0.7 75.32144 88.513. 71.2 89.2228 77.91111
22.22.0
36.0g1..27.77:
28.020. 33.0go.7 29.410.
DARL
65.5_a.5 70.2 73.4ga.0 86.622 69.0 83.7
74.7
25.9
27.2
32.5
28. 28.3
DN2A
71.121.0 22.4 72.5g1.4 85.82o.7
71.2 90.0= 77.2...
23.33
35.00g.
26.0
27.7=22
33.0g.1. 29.0
UCDIR 73.7-1.6 69.9+s.1 74.6+0.4
91.4+1.1
o 90.2==.000.. 78.8+1.2
25.8
11.8++..36.6:1..
28.0+o.5
27.5=1.
+4.1 30.4+0.4
CoDA
71.4223 24. 91. 91.4 20 90.2
78.8
26.0
34,9
29.2
2.
33.8 30.4g...
DGDIR
73.5-2.1 71.2=1.4 75.1 91.7 74.0 90.5 79.3
27.7
35.001..
30.0
27.8
33.6=2.0 30.8
Ours
76.2 27. 75.6 g0 78.9
21.0
81.9
31.9
39.4
35.0 29.8 35.7 34.420.5
Query Sample:
Retrioval Roulss of UENE
Retrioval Result of DGDR:
Painting
Figure 4: Retrieval results of UEM and DGDIR on Office-31 (A (A-D), Office-Home (A+C), and DomainNet (CI-Pa) in Close-set Unsupervised Cross-Domaie Retrieval. Green and red rectangles denote correct and incorrect retrieval results. Best viewed in colors.
4.1 Effectiveness of UEM When Solving C²CDR
Retrieval. For the close-set setting, the label space of the query domain is identical to the retrieval domain, All domain pairs of Office-31 and Office-Home are tested, and 5 pairs of DomainNet. The results for Office-31 and DomainNet are shown in Table T (Office-Home can be found in the Appendix). According to these results, we can observe that UEM significantly outperforms other baselines in all cases. Specifically, we can achieve mAp@ All improvement of 2.6% compared to the best baseline method on Office-31, and such improvement is even larger on DomainNet with average of 3.6%. Figure 4 also shows the retrieval results of our approach on Office-31, Office- Home, and DomainNet, and we can observe that all results are correct.
Close-set Unsupervised
Dooss-Domain Retrieval. To establish the partial setting, the query domain contains only half of the label space of the retrieval domain, and the query label space is randomly selected. As shown in Table 2 we can observe that UEM exceeds all other baseline methods significantly in mA mAP@All, which is much more substantial than close-set UCDR. For example, UEM outperforms the second-best with average of 12.4% on Office-Home. Similar improvements
Partial Unsupervised
8

---

on Office-31 and DomainNet can be observed (results are provided in the Appendix). Besides, we can easily observe that existing state-of-the-an of-tthe-rt studies are nearly incapable of dealing with the label space difference in partial UCDR (see Theorem 3.T while our UEM can work effectively.
Open-set Unsupervised Dooman Retrieval. As for the open-set setups, we ensure that the label space of the retrieval domain is half of the query label space. The experiment results for DomainNet (results for Office-31 and Office-Homi Home are in the Appendix) are presented in Table 3 with two metrics mAP @ All for the shared label set, and detection accuracy for the private (open-set) query labels (please refer to Appendix for how to detect the private query labels). According to these results, we easily observe that our approach substantially exceeds other baseline methods in both metrics. Similar trends can also be found for Office-31 and Office-Home All these results strongly validate the effectiveness of UEM in open-set UCDR.
Table 2: Performance comparison (mAP@ All) between ours and other baseline methods on Office- Home in Partial Unsupervised Dooss-Dommin Retrieval.
Methods A-C
A-P
A-R
C-+A
C-P
C-R
P-+A
P-C
P-R
R-A
R-C
R-P Avg.
CDS
22.0_1.1 20. 32.5 26.521. 25.610: 27.9 30.0 31.8111 40.5g2. 32,3 25.5 6..6 30.3
PCS 24.5_o.1 36.5g1.a 38.8 8120 24.9 28.8g1.1 29.0=1.0 28.6.2.1 35.3 41.72114
37,55220 26.921. 40.0
32.710.s
DARL 25.5
29.
25.0g1.
=3.921:27,5.11
26.8
31.92..
40.0gg.
35.5 27.7=a. 40.0gn.5 30.7
DN2A 37.02...7.0:.. 29.5
25.2
2
35.721.7
28.0za. 41.0z1. 31.720.5
UCDIR 28.7-2....7++.222 31.0
26.0+1.
22.0+1.1
23,5=1.
31,1+111111
30.4
40.24..
2..9
36.8 29.7+0.7
CoDA 22.5-1.2 34.2 35.7
25.0
29.5
30.0=o.
30.7
32.0
43.2+
35.2 28.5
41.3 32.3
DGDIR 24.4 30.9 41.0
27.2
30.5
30.42...
33.2411
37.1
30.9 42.0a1. 33.6.0.3
Ours 40.5 45.8
35.1
39.2
41.1 52.423.0
46.0
55.0
29.0
43.1
56.721.2 46.0
Table 3:
Performance comparison (mAP@ All for shared-label set, detection accuracy for open-label set) between ours and other baseline methods on Domain- Net in Open-set Unsupervised Cross-Domain Retrieval.
Table 4: Ablation studies of UEM Office-31, Office-Home, and DomainNet in Open-set Unsupervised Retrieval. The average values of Shared- set mAP@ All and Open-set Acc for all domain pairs are reported here.
Dross-
Methods Qu-=CI
CI-Pa
Pa-
In-+Sk
Sk-tRe
Avg.
Shared-set mAP@ All
Open-set Ace
CDS
22.4 58.9
25,5
60.7
33.7
28.9
61.8
PCS
23,3 57.8
34.2 67.8
24.9 60.5
27.8 65.4
34.7 66.9
29.0 63.7
Variations
Office-31
Office-Home DDeeea-HommeDemanNNNN
Net
DARL
21.9 52.5 60.2
22.0 53.9
26.6 60.6
32.3 62.8
27.1 58.4
Shared-sct mAP@ All Open-set Acc
DN2A
22.7 56.6
33,4
60.7
21.9 55.2
24.8
57.8
34.0 61.2
27.4 58.3
UCDIR
24.4 59.0
34.4
67.0
26.7 62.9
25.6
63.4
35.5 68.2
29.3 64.1
Ours w/o P.M.
668.3
78.8
45.7
72.6
330.9
70.2
CoDA
24.2 58.8
35.6 66.6
27.0 61.1
24,9
58.0
34.6
62.9
29.3 61.5
Ours
72.2
88.3
47.5
81.7
33.0
76.4
DGDIR
23,5 57.5
36.0 68.3
25.8 60.6
25.8
59.1
35.0 64.3
29.2 62.0
Ours w/o SPDA
62.2
61.9
39.0
60.8
24.4
60.5
Ours 3M
75.0
89.2
48.8
82.8
33.3
80.5
Ours
30.3 72.9
40.2 88.1
36.0 81.5)31.1
78.2
36.6
36.6
83.0
34.8 80.9
Ours
77.4
92.5
50.2
86.7
34.8
80.9
4.2 Ablation Study
All the ablation studies are carried out in open-set UCDR on three datasets, and the average metrics (Shared-set mAP@ All and Open-set Acc) for all domain pairs of single dataset are reported here.
Effectiveness of Prototype Merging, When evaluating the effectiveness of a unified prototypical structure, we do not use prototype merging in IDSE. According to the results of 'Ours w/o P.M.' in Table there is non-negligible performance drop in both shared-set mAP@ @ All and open-set accuracy. This validates the importance of building a unified prototypical structure across domains.
Effectiveness of SEL. When evaluating SEL, we detach it during the model training. According to the results of 'Ours w/o SEL' in Table 4 there also an evident performance drop compared to the full UEM. This validates that SEL is vital as it can help prepare a better base model for CDSM.
Effectiveness of SPDA. We replace our SPDA with the standard domain adversarial learning for the ablation study. As shown in Table E there is a significant performance difference between 'Ours w/o SPDA' and the full UEM, which illustrates the importance of semantic preservation during domain alignment, as well as indirectly verifying the necessity of SPDA.

---

Effectiveness of SN We also replace SN with the nearest neighboring search approach leveraged by UCDIR. By comparing the results of "Ours w/o SN and 'Ours', we can conclude that SN is more compatible with UEM and able to achieve more accurate cross-domain categorical matching.
5 Conclusion
In this work, we focus on two major challenges when conducting cross-domain retrieval (CDR) in real-world scenarios: one is that the category space across domains usually distinct, and the other is that both the query and retrieval domains are unlabeled. To tackle these challenges, we propose Unified, Enhanced, and Matched (UEM) semantic feature learning framework that can establish unified semantic structure across domains and preserve this structure during categorical matching. Extensive experiments cases including close-set, partial, open-set unsupervised CDR on multiple datasets demonstrate the effectiveness and universality of UEM, which are reflected in the substantial performance improvement over state-of-the-ae studies from Cross-Domain Representation Learning. Unsupervised Domain Generalization, and Unsupervised CDR.
6 Limitations and Future Work
To solve the real-world challenges when employing cross-domain retrieval, especially considering the category distinctness across unsupervised data domains, we propose the UEM semantic feature learning framework in this work. Although extensive empirical evaluation and theoretical analysis have validated the effectiveness of UEM, some minor limitations still need more exploration. First, the current UEM framework is composed of two stages, and we empirically determine the switching point. In the future, we need achieve real end-to-end UEM by changing the training stage automatically. Besides, there is a lack of theoretical analysis for the second stage (CDSM). In future efforts, we should theoretically prove the semantic preservation of SPDA and the reliability of SN'M. Lastly, the general applicability of UEM also needs testing. For instance, cross-person generalization in wearable devices (42) and property analysis of material retrieval. Therefore, we should test UEM on other modalities like time series and graph data.
4 or molecular 44 structures require cross-domain
Acknowledgments
We gratefully acknowledge the support of a grant from General Motors.
References
[1] Benjamin Klein and Lior Wolf. End-to End-to-end supervised product quantization for image search and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5041-50500 2019.
[2] Muhammad Murad Khan, Roliana Ibrahim, and Imran Ghani. Cross domain recommender systems: a systematic literature review. ACM Computing Surveys (CSUR), 50(3):1-34, 2017.
[3] Min Jin Chong, Wen-Sheng Chu, Abhishek Kumar, and David Forsyth. Retrieve in style: Unsupervised facial feature transfer and retrieval. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 3887-3896, 2021.
[4] Bojana Gajic and Ramon Baldrich. Cross-domain fashion image retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1869-1871, 2018.
[5] Xiaoping Zhou, Xiangyu Han, Haoran Li, Jia Wang, and Xun Liang. Cross- image retrieval: methods and applications. International Journal of Multimedia Information Retrieval, 113))199-2188 2022.
[6] Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image re- trieval with dual attribute-aware ranking network. Proceedings of the IEEE international conference computer vision, pages 1062-1070, 2015.
10

---

[7] Conghui Hu and Gim Hee Lee. Feature representation learning for unsupervised cross domsan image retrieval. In European Conference on Computer Vision, pages 529-544. Springer, 2022.
[8] Conghui Hu, Can Zhang, and Gim Hee Lee. Unsupervised feature representation learning for Jomain-generalin cross-domain image retrieval. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pages 11016-11000 11025, 2023.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMIR, 2020.
[10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[11] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference Learning Representations, 2020.
[12] Cuie Yang, Yiu-Ming Cheung, Jinliang Ding, Kay Chen Tan, Bing Xue, and Mengjie Zhang. Contrastive learning assisted-alignment for partial domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[13] Xiangyu Yue, Zangwei Zheng. Shanghang Zhang, Yang Gao, Trevor Darrell, Kurt Keutzer, and Alberto Sangiovanni Vincentelli. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13844, 2021.
[14] Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A Plummer, Stan Sclaroff, and Kate Saenko. Cds: Cross-domain self-supervised pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9123-9132, 2021.
[15] Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image re- trieval with dual attribute-aware ranking network. In Proceedings of the IEEE international conference on computer vision, pages 1062-1070, 2015.
[16] Xiaoping Zhou, Xiangyu Han, Haoran Li, Jia Wang, and Xun Liang. Cross-domain image retrieval: methods and applications. International Journal of Multimedia Information Retrieval, 11(3):199-218, 2022.
[17] Xu Wang. Dezhong Peng, Ming Yan, and Peng Hu. Correspondence-free domain alignment for unsupervised cross-domain image retrieval. arXiv preprint arXiv:2302.06608/ 2023.
[18] Mingyuan Ge, Jianan Shui, Junyu Chen, and Mingyong Li. Pseudo-label based unsuper- vised momentum representation learning for multi-domain image retrieval, In International Conference on Multimedia Modeling, pages 369-380. Springer, 2024.
[19] Kaipeng Fang, Jingkuan Song, Lianli Gao, Pengpeng Zeng, Zhi-Qi Cheng, Xiyao Li, and Heng Tao Shen. Pros: Prompting-te srrriinngtt---mmmenee to-simulatee generalized knowledge for universal cross- domain retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[20] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generaliza- tion. IEEE Transactions on Knowledge and Data Engineering, 2022.
[21] Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European conference computer vision (ECCV), pages 135-150, 2018.
[22] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference Machine Learning, pages 11455-11472 1472. PMLR, 2022.

---

[23] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE international conference on computer vision, pages 754-763, 2017.
[24] Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang. and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE/CVF conference on computer vixion and pattern recognition, pages 2927-2936, 2019.
[25] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720-2729, 2019.
[26] Bo Fu, Zhangjie Cao. Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In Computer Vision-FCCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 567-583. Springer, 2020.
[27] Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9757-9766, 2021.
(28) Kuniaki Saito and Kate Saenko. Ovanet: One-vs-all network for universal domain adaptation. In Proceedings of the ieee/cyf international conference on computer vision, pages 9000-9009,
2021.
[29] Liang Chen, Qianjin Du, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. Mutual nearest neighbor contrast and hybrid prototype self-training for universal domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6248 6248-6257, 2022.
[30] Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. Evidential neighborhood contrastive learning for universal domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2258-6267, 2022.
[31] Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4910-4920, 2022.
[32] Yuchen Liu, Yaoming Wang, Yabo Chen, Wenrui Dai, Chenglin Li, Junni Zou, and Hongkai Xiong. Promoting semantic connectivity: Dual nearest neighbors contrastive learning for unsupervised domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3510-3519, 2023.
[33] Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. A survey of graph edit distance. Puttern Analysis and upplications, 13:1 13-1 2010.
[34] Fan Liu and Yong Deng. Determine the number of unknown targets in open world based on elbow method. IEEE Transactions Fuzzy Systems, 2955)988699 2020.
[35] Mahsa Baktashmotlagh, Mehrtash Har, Mathieu Salzmann, al. coaributiio-matcciin em- bedding for visual domain adaptation. Journal of Machine Learning Research, 17(108):1300, 2016.
[36] Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael I Jordan. Conditional adversarial domain adaptation. Advances neural information processing systems, 31, 2018
[37] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40):24652-2466) 2020.
[38] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pages 213-226. Springer, 2010.
12

---

[39] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[40] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE/CVF international conference computer vision, pages 1406-1415, 2019.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[42] Chin-Chia Michael Yeh, Huiyuan Chen, Xin Dai, Yan Zheng, Junpeng Wang, Vivian Lai, Yujie Fan, Audrey Der, Zhongfang Zhuang, Liang Wang, et al. An efficient content-basee based time series retrieval system. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4909 4915, 2023.
[43] Tahoura Mosavirik, Mohammad Hashemi, Mohammad Soleimani, Vahid Nayyeri, and Omar M Ramahi. Accuracy-improve and low-cost material characterization using power measurement and artificial neural network. IEEE Transactions Instrumentation and Measurement, 70:1-9,
2021.
[44] Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anand- kumar. Retrieval- controllable molecule generation. In The Eleventh International Conference on Learning Representations, 2022.
[45] Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning? In International conference on machine learning, pages 1101-1116. PMLR, 2022.
13

---

Appendix
This Appendix includes additional details for the paper" Feature Learning for Universal Unsupervised Cross-Domain Retrieval", including theoretical proofs (Section[A) implementation details (Section B). additional experiment results (Section C). and broader impact (Section D
A Theoretical Proofs
Geometry Distinctness). Suppose data distributions of two domains (A and B) have mutually disjoint supports, and they are uniform over these supports. Assuming the support sets of domains and are not identical, the optimal feature feature extractors f that minimize the instance discrimination loss of different domains present distinct geometric feature spaces.
Theorem 3.1
Proof. Suppose data distribution is made of mutually disjoint supports and distributed uniformly over these supports, there is a theorem demonstrating the representation geometry learned by instance discrimination in a research work [45], i.e..
Theorem A.1. Assuming a data distribution has mutually disjoint supports and is uniform over these supports, any Simplex ETF representation extracted by minimizes LNCE for any convex and non-increasing loss function 1. Moreover, if is strictly convex (e.g., logistic loss), then Simplex ETF representations are the only minimizes of (NC
The Simplex ETF representations are defined as follows,
Definition A.2 (Simplex ETF). A simplex ETF is collection of equal-length and maximally equiangular vectors. We call a P x K matrix M an ETF if satisfies
K K-
M
K-1
1
(19)
where is non-zero scalar, is the identity matrix and 1g is all-ones vector.
In representation learning, ETF representations mean that samples from different categories (æi, V:), (æej My) satisfy
fo(x;) fo( -1 C-1
Vy + Vj [o (();;;)]l((=
(20)
With the above preparations, we apply proof by contradiction to prove that different domains hold distinct geometric feature spaces when their respective instance discrimination loss achieves minimization. Specifically, we assume domains A and B share C' categories and there is only one category exclusively owned by domain B, i.e.,
Y yn = (E') J y CHIBB = +1.
(21)
This assumption satisfies that the support of domains A and B not identical. Then suppose that the geometric structures of domains A and B are identical when the instance discrimination loss has been minimized, according to Theorem A.II both domains A and B present Simplex ETF representations. In this case, if we randomly select two categories y" y E YA n y from the domain-shared category set, and their samples within each domain have the following relation,
Vy".8 y VA nBB fo(xp.A). (a(
-1
fo( (x fo
(22)
fo(x). flfo CA-I1' ( ([loo(2
C CB-
According to Eq. (21. Eq. (2) implies that the same category pair across domains has different cosine similarities, which contradicts the assumption identical geometry across domains.
Convergence of IPM). Suppose the data distribution of a domain has mutually dis- joint supports, and it is uniform over these supports. Simplex Equiangular Tight Frame (ETF) representations minimize the
Theorem 3.2
Loss of this domain.
Proof. As shown in Eq. 1T), the Instance BaeaanceProootyp--Miiss loss is composed of instance discrimina- tion and prototype contrastive loss. The simplex ETF representations have been proven to minimize instance discrimination Theorem A. Then we only need prove that the prototype contrastive loss satisfies the convex and non-increasing properties,
14

---

Property A.3. For a loss function defined on a set V = {uili= with the size of t. it holds for all subsets of index S S {1, ........ t that
1(V8+)), where V. VSt : Us if Uj otherwise.
((V)= 2 S| jES
(23)
To prove the non-increasing property of prototype contrastive loss, we can view LPNCE built on the set V fue fa(x),) P fo (z;) Pe/T]em then LPNCE (x;) (V) log(1
E. expl - ). ). We can leverage the concavity of the log function (Jensen's 's inequality) and denote T Lnes exp(- exp(-ey), and we have
(V) - loggTTT +
Σ CES
exp(-uu)
S
log(T +
=
S
Log (2S5
CES
CES
(24)
Therefore, the prototype contrastive loss also holds the non-increasing property, which proves that simplex ETF representations minimize the Instance-Prototyye Prototype-Mixed Loss.
B
Implementation Details
B.1 Open-set Query Label Detection
In open-set unsupervised cross-domain retrieval (UCDR) settings, the query domain has some private categories that not included in the retrieval domain. In this case, the retrieval results of query samples for such private query categories should be null. As aforementioned, for a query sample a;A, the retrieval process needs to calculate the distance between all samples in the retrieval domain and wF Then the most similar retrieval samples are supposed to be those located as close to æ;A as possible. Intuitively, if æ belongs to the private query categories, the nearest sample in the retrieval domain should be relatively distant. Therefore, there a need for a threshold that allows us to determine whether the closest retrieval sample of query sample is located too far to be a similar sample. Next, let us introduce how our UEM detects and identifies whether a query sample belongs to private query categories.
In our UEM framework, the crucial design is build unified prototypical structure across domains, which also shapes the private label detection strategy. Specifically, after the training of CDSM, we apply K-Means the query and retrieval domain datasets again construct the prototype sets PA and PB. Then the prototypes of the retrieval domain are translated to the query domain for potential merging. The detailed prototype translation and merging have been introduced Section B.2.1 After the prototype merging, both PA and PB are divided into two groups by satisfying the merging condition (Eq. (9)) or not. For the prototype pairs that satisfy the merging condition, we record the maximum samr-e distance among their clusters
(1-
fo(=))) fo(a)) [fo( (1fo(((]]
Doo
max MAEE 2 EXBB
(25)
XAA
Sa arg min (((o(();))))) p
A.
xB ={
{
arg min
(1) p
B.e
pd
pB
(26)
Then for any query sample x;A there are two cases for its belonging. One is x belongs to the clusters of prototypes unmerged, i.e., PA pAle In this case, ax, is supposed to come from private query labels with high confidence. The other case is that the closest prototype of æ² satisfies the merging condition, i..., arg mings [d(.fo (x+)), p>}] po
A.B PA.a in which we should compare
the recorded D and the minimal distance between all samples in the retrieval domain and ær i.e.,
( (1- (o((
fo(m;)) fo(x))
DA
min n ED!
(27)
15

---

Dee < DA we have faith that there is no sample similar enough in the retrieval domain, which means æ. should belong to private query categories. Otherwise, 2; comes from the shared label set, and we should conduct the normal retrieval operation.
B.2 Comparison Baseline Implementation
In our evaluation process, we implement a number of state-of-the-ar baseline methods to compare our proposed UEM in UDDDR. For fair comparison, ensure two principles for all these used baselines one is that the training data consists of at least two domains, and the other is that the training data is unlabeled. Following these two principles, in addition to unsupervised cross-domain retrieval studies 77 8 17,, two other problems share similar setups: cross-domain representation learning (CDRL) [3 and unsupervised domain generalization (UDG) [1 32) For CDRL, the objective is to learn -eneringennealiizaale representations that provide somain-transferabbe knowledge for any downstream task. One of the most typical downstream tasks of CDRL is cross-domain retrieval. As for UDG, in addition to achieving effective cross-domain representation learning, domain- generalizable classifiers are also needed. However, in our evaluation, the retrieval process does not require any classifier, thus we omit all techniques related to classifier training for the used UDG approaches.
For the specific setups of our experiments, we consider close-set, partial, and open-set UCDR. The close-set UCDR assumes that the label spaces of the query and retrieval domains are identical, which is the benchmark setup of other baseline methods. In this case, follow the default settings of these baseline methods to evaluate their performance in close-set UCDR. As for the partial UCDR, the label space of the query domain is half of the retrieval label space, where most baseline methods can work normally without any modification. But some baseline approaches (PCS [13. UCDIR (7]. DGDIR () especially those based on prototype learning, require the knowledge of category numbers for domains, therefore, we suppose the category numbers are known to these approaches. The last open-set UCDR supposes the retrieval label space is half of the query label space. In this setup, the query domain has private categories, and if we conduct retrieval for samples from these categories, the retrieval results should be null. To detect private categories, we follow the strategy leveraged by UEM (Section I).I) to employ a similar one for the used baseline methods. Specifically, we divide all baseline methods into two groups by whether there is a dedicated nearest neighboring searching algorithm. For those (DARL (31) that don't have the nearest neighboring search, we attach the searching algorithm used by UCDIR 77 After conducting all training and operations of any baseline method, we use the nearest neighboring search to pair samples from the query and retrieval domains. Moreover, we conduct K-Means in the query domain to build the prototype set PA (for CoDA (I7), we leverage its auxiliary classifiers to construct the prototype set). Then for each cluster in the query domain, we record the maximum inter-sample distance D Note that the distance measurement here is different from the product of minus cosine similarity and Euclidean distance (Eq. (25)), and different approaches use diverse measurement, e.g., UCDIR 7 uses cosine similarity while DN2A 32 leverages Euclidean distance.
Table 5: Performance comparison (mAP@ All) between ours and other baseline methods on Office- Home in Close-set set Unsupervised Cross-Domain Retrieval.
Methods
A-+C
A-P
A-R
C-+A
C-+P
C-R
P-+A
P-C
P-R
R-A
R-C
R-P Avg.
CDS
33.0
44.5 $1.42.a 32.4 40.3 41.8=2.
45.3 41.5
60.8
42.0=
58.8 45.2
PCS
34.3_1.1
46.3
51.6 32.3
40.6
47.0 41. 6
42.8
60.1 45.9
DARL
32.4-1. 40.9
50.5 33.0 33.001033..7.0r
36.7
47.0
40.9+1.4
59.0
43.0=2a
60.8 44.7+1.
DN2A 35.5-o. 42.8 52.9+2. 34.0 35.7
42.0=
43.2:2 43.2+
48.0
59,8,14 49.0
44.7
56.
45.3
UCDIR 36.1 46.5 55.9 34.0
44.1
52.
CoDA
34.72o. 53.2209
33.2
42.922
44.7
50.42..
45.2
65.221.
65221
DGDIR 36.0_1.0 50.1 55.5
34.1 44.9 a 43.0 51.5
45.5 66.6
53.0
44.5
67.0 49.3
Ours
38.5 52.6 19.0
34.2 47.5 49.0
55.2 49.0+1.1
69.5 56.9+2.1
48.7
68.2 52.40.7
C Additional Experiments
Here we provide the additional experiment results including the close-set UCDR on Office-Home 5),, partial UCDR on Office-31 and DomainNet (Table and open-set UCDR on Office-31 (Table 7 and Office-Home (Table R). According to these results, we can obtain similar observations and conclusions to the main paper. First, our UEM can achieve the best performance in close-set
16

---

Table 6: Performance comparison (mAP@ All) between ours and other baseline methods on Office-31 and DomainNet in Partial Unsupervised Cross-Domain Retrieval.
Methods
A-D
A-W
D>+A
D-W
W-+A W-+1
Avg.
Qu-+CI
CI-+Pa Pa-+In
In-+Sk
Sk-Re
Avg.
CDS
42.5-...2 39.7
45.4 68.8 20.6 40.7 55.8 48.8
91.7
22.0 21. 29.6.2.s 24.4
PCS
41.5+.. 41.5+..44..99 47.9+1.s
47.3400
50.5=o.:5055::::
18.0
31.551.
23.4:1822.1#0:: 22.1 27.9 24.6+0.4
DARL
41.5g1.0
38.9 49.0
65,
45.721
53.622.1
49.021a
19.0g1.1
30,512.
28.7 24.911.2 23.3g1.6 23.0g22
DN2A 42.5=1.0
20.0
g0.
66.7
46.520.
55.0=1a55.3..1
18.7
31.1
25.0go.7
24.4g1.4
30.7 26.0gn.9
UCDIR
46.0_..1 41.822. 50.3 62.9 47.0g1 56.9 50.8
19.3
29.82o.s
24.4g1.4 23.1 29.0g1. 25.1
CoDA
45.1-2.0 40.7 49.3+s..
66.0.14
47.0 a 57.8 0.a 51.0
20.2
30,9
31.0 26.3 25.2 24,0=0a
DGDIR 45.0 39.2 49.7
64.4
g 55.2 1.3 50.3
18.8
30.4
25.0go.2
24.5 31.2 26.0go.5
Ours 64.4 22.0 59.4 76.9 2..2 61.5 a 65.0 63.0 28.2 34,9 32.720.7 26.6 34.0 31.320.5
Table 7: Performance comparison (mAP@ All for shared-label set, detection accuracy for open-label set) between ours and other baselines on Office-31 in Open-set Unsupervised
Retrieval.
Methods
A
A
D D-
W-
DD
Avg
Shared-sct MAP@ All/ Open-sct Acc
CDS
20.7
76.2
80.2
62.9
81.3
80.7
90:222.
79.0_a.
[0..
89.7
56
66.9
82.8 82.8
PCS
62.5+1.4
78.9+22
67.3+..
84.4=1.
65.8e1.1
82,9.3.
82.9 90.8=1a 63.3-0a
79.5-1.0
81.6-1
91.1+23
70.6+11
84,6+Ls
DARL
60.9+u2
78.0=2.4
60.5g..
77.2a1.9
62.6
78.0+1a
78.3+1 884430
60.0-1.
79.4-2.
78.1
87.9 66.7
81.5
DN2A
61.7204 76.91..
60.821.1
75.329a
62.5g0 78.0110
80.219a 90.45.
61.12r 76.7_a
78.82o7
90.0g14
67.5
81.2
UCDIR
62.9
78.1
65.5
82.4g1.6
62.61e. 80.4147
19.9 86.9
64.4_aa
84.3 1.7
80.5
91. 69.3
13.9
CoDA
60.9211
74.622.0
62.4g1.
78.0g1.9
66.62o
82.221
83.521a
90.0gaa
70.3
85.0-1.1
78.5210
89.0ga4
70.4g0
83.12
DGDIR
68.0109
81.1
62.3
80.0gaa
64.010.7
79.911.
80.91.1
87.519.
66.8_o.
79.4_ 1.a
[2..2
91.01aa
70.211
83.2gg.
Ours
70.613.2 90.7 10.8 93.0 72.4 89.8139 87.7 95.5 74.0_..1 90.2 (82.9
95.8,33
77.4 92.5
UCDR, which is reflected by the average performance improvement of 3.1% on Office-Home. Moreover, UEM can exceed all other baseline methods much more substantially in both partial and open-set UCDR. Specifically, our methods outperform the best baseline with margin of 5.0% on DomainNet and 12.0% on Office-31 in partial UCDR. As for open-set UCDR, UEM exceeds other baseline methods with range of 6.8% and 7.9% on Office-31 for shared-set mAp@ detection accuracy respectively, and such improvement is much higher on Office-Home in 15. % for shared-set mAP@ All and 13.1% for open-set detection accuracy. In addition, if we compare the results of the same domain pairs between close-set and partial/open-se UCDR for baseline methods, we observe that the performance drops lot, which also validates the existence of geometry distinctness (Theorem 3.T). J. In particular, such geometry distinctness originating from the category space difference can incur performance drop of up 29% for DGDIR on Office-31 between close-set and partial UCDR.
@ All and open-set
D Broader Impact
The research development for solving Universal Unsupervised Cross-Domain Retrieval (UºCDR) has the potential to make a significant positive impact across various sectors. By enabling more accurate and flexible retrieval of information across different domains without the need for supervision and the concern about semantic category distinctness, our proposed UEM framework can enhance user experiences product recommendation systems, leading to more personalized and relevant suggestions. In the realm of artistic creation, UEM can facilitate novel connections and inspirations by retrieving cross-domain artistic elements, fostering creativity and innovation. Beyond these applications, UEM can also be beneficial in healthcare, finance, and scientific research with certain adaptive modifications. In healthcare, it can improve diagnostic tools and personalized treatment plans by integrating diverse data sources. In finance, it can enhance risk assessment and fraud detection by analyzing cross-domain financial data. In scientific research, it can accelcrate discoveries by connecting insights from different fields. While UEM has substantial benefits, it is crucial that its deployment adheres to existing privacy and intellectual property protection regulations and policies. Ensuring that data used in cross-domain retrieval respects user privacy and intellectual property rights is essential to prevent misuse and maintain public trust. Thus, applying this research in the real world should consider ethical issues ensure responsible and fair use, thereby aiming for a positive societal impact without any negative social consequences.
17

---

Table 8: Performance comparison (mAP@ All for shared-label set, detection accuracy for open-label set) between ours and others on Office- Home in Open-set Unsupervised Cross-Domaa Retrieval.
Mathods
A-+C
A-P
A-R
C-+A
C-P
C-R
P-A
P--
P-R
R-+A
R-C
R-P
Avg.
Shared-set
mAP
Open-eet
CDS
26.9 60.2
34.7
68.7
33.9
27.9
66.2
28.8
65.5
30.0 66.2
32.1
68.9
32.5
44,2
74,5
32.5
69.0
26.7
65.1
40.0
76.0
32.5
68.2
26.6
72.0
76.6
65.4
69.9
31.5
37,9
42.6
77.4
38,9
72.8
66.6
78.0
DARL
26.9 64.0
36,9
70.1
28.5
66.4
26.7 65.0
22.0 60.6
29.4 68.2
25.9
61.3
31.0
89.1
42.5 79.5
35.0
72.2 28.0
65.1
4.22
78.0
31.4
68.5
DN2A
63.3 38.2
14,4
29.0
68.8
26.8 65.4
28.9 67.5
14.0 72.4
29.0 66.8
33.0
71.5
42.0 80.3
35.5
72.6
29,4
68.8
41.5
794
32,9
70.9
UCDIR
24.7 62.2
30.1
64.4
26.8 64.0
26.5 65.2
21.5 62.0
25.5 67.0
32.0
71.9
31.1 70.8
43.0
78.9
37.6
76.6
28.0
69.2
38.5
77.4
30.4
69.1
CoDA
24.9 63.0
15.5
74.3
56.0 74.5
25.5 65.9
M1.1 70.8
12.5
72.3
90.5
71.1
33.7
74.4
44.4 81.5
37.0
78.
366
76.0
42.1
80.9
34.2 73.6
DGDIR
25.5
29.0
31.0
33.0
45.8
68.0
82.2
71.3
Ours
40.6 80.8
49.0
87.7
(55.4
92.0
92.0
33.7
72.2
72.2
45,0
45.3
|5,9 90.2)
48.7
87.8|64.4
90.0
52.2
90.3
47.7 86.7
86,7
89.5
50.2 86.7
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are addressed in Sections and of the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section of the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
18

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The main theoretical insights are discussed in Section and detailed proofs are given in Section of the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the implementation details to reproduce the results are given in Section of the main paper and Section B of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
19

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are checking the relevant regulations of our institutions to release the source code. All the datasets used for this work are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guides wwwee//Ssssesssssiooooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc http/uullcgggguugee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All the experiment details given in Section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
20

---

Answer: [Yes]
Justification: All the experiment results are supplemented with standard deviations as the error bar resulting from 3 independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The Appendix discusses all relevant computing requirements in detail.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:t/////// http///wwrrr...cepuuuueee/eeetaa
Answer: [Yes]
Justification: Our work adheres all the ethical guidelines outlined by NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
21

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Section of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does pose any explicit risks applying our proposed methods to other areas needs dedicated modifications.
public datasets, and
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All dataset details and original authorship are cited in Section
Guidelines:
22

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwprswipeeaaaeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
23

---

Justification: Our paper does not involve crowdsoureing nor research with human subjects, hence do not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "XvBaornhmb", "reviewer": "Reviewer_gZ1R", "summary": "This paper proposes Universal Unsupervised Cross-Domain Retrieval for the first time and designs a two-stage semantic feature learning framework to address it.", "questions": "See Weaknesses section", "limitations": "no", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper proposes a new approach in universal unsupervised domain adaptation, with sufficient experiments to verify its motivation.", "weaknesses": "1. In unified unsupervised domain adaptation, there is no handling of instances that are not common categories. Isn't this necessary?\n\n2. From the perspective of innovation, the proposed unified prototype structure is interesting, and the rest is mostly incremental work, such as semantic structure preservation and adjacent feature matching in domain adaptation. From the visualization results, the author failed to prove the above contributions.\n\n3. This paper should reflect the difference between universal domain adaptation and unsupervised domain adaptation.\n\n4. This article does not have a better way to state the method, especially in cross-domain prototype conversion and close neighbor matching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Any handling of instances belonging to uncommon categories\n\nIn the first stage of our UEM framework, we aim to build a unified prototypical structure across domains via the IPM loss. The IPM loss is a combination of instance and prototype contrastive losses. Given that the IPM loss is computed separately in each domain, we design a novel prototype merging strategy to merge highly potential common categories across domains. Then for the prototype contrastive loss, each instance searches for and tries to approach its closest prototype. **There is no difference between instances corresponding to unmerged (uncommon category) or merged prototypes, as all prototypes should be treated equally to form the final prototypical structure -- otherwise, the uncommon category detection in partial and open-set cross-domain retrieval becomes much harder.** As for the second stage, each instance of each domain searches for and decides whether to approach the nearest neighbor in the other domain. For instances belonging to merged prototypes (common category), there is a much higher possibility of being optimized to approach their nearest neighbors. **While for instances that are not common categories, they usually shouldn't approach their cross-domain neighbors. However, we still allow these instances to approach their domain-translated prototypes, which means that we do have the handling for instances that are not common categories.** We believe this is necessary to achieve more effective cross-domain matching. \n\n> Novelty of semantic structure preservation and adjacent feature matching\n\nPlease refer to the Global Response for a detailed description of the novelty. To further validate the novelty of semantic structure preservation and adjacent feature matching, we carry out experiments with the replacement of another state-of-the-art binary graph-based semantic preservation approach (SPPA [1]) and nearest neighbor searching algorithm (CD^2NN [2]). SPPA preserves the instance-level cosine similarity within each cluster and the prototype-level Euclidean distance across clusters. CD^2NN determines cross-domain nearest instance pair by seeking neighboring instance consistency. The experiment results in open-set CDR (in each cell, the left number is the Shared-set mAP@All, while the right is the Open-set detection accuracy) are shown below, **which demonstrate that UEM's semantic structure preservation and adjacent feature matching are more effective.**\n\n|         | Office-31  | Office-Home | DomainNet  |\n|---------|------------|-------------|------------|\n| UEM w/ SPPA  | 76.0, 88.9| 47.2, 84.4  | 31.5, 75.7   |\n| UEM w/ CD^2NN | 76.4, 89.4 |   46.5, 85.0 | 32.1, 76.4 |\n| UEM     | 77.4, 92.5 |  50.2, 86.7  | 34.8, 80.9 |\n\n[1] Lin, et al. Continual semantic segmentation via structure preserving and projected feature alignment. ECCV 2022\n\n[2] Liu, et al. Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023\n\n> Difference between universal and unsupervised domain adaptation\n\nWe suppose you are indicating the difference between universal and unsupervised cross-domain retrieval (CDR) rather than domain adaptation. First, the CDR problem considers two semantically distinct but similar domains and aims to retrieve data samples from one domain belonging to the same category as a query sample from the other domain. The unsupervised CDR specifies that these two domains only contain unlabeled data. **Regular unsupervised CDR studies assume that the category spaces of these two domains are identical, while universal CDR focuses on scenarios where the category space across domains is distinct.** In our work, we believe the assumption of identical cross-domain category space is unreasonable for many real-world applications, as the categorical composition of an unlabeled data domain is hard to acquire without detailed analysis and dedicated expertise.\n\n\n> Better statement of cross-domain conversion and close neighbor matching\n\nPlease refer to the Global Response for better descriptions of cross-domain conversion and close neighbor matching.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IvytY2tyfs", "reviewer": "Reviewer_tjhd", "summary": "This paper tackles the problem of unsupervised cross-domain retrieval. This is the problem where the query and retrieval domains are distinct. For example, in sketch to real retrieval, the system must retrieve the most relevant real images to a query sketch. \"Unsupervised\" refers to the fact that no labels are available during training, but the images from both domains are available. The authors claim to be the first to investigate the \"universal\" version of this problem, where the query and retrieval domains are allowed to have disjoint labels spaces. For this problem, the authors propose a two-stage optimization procedure. In the first stage, three losses are used: (1) an instance-wise contrastive loss (2) a cluster-wise contrastive loss and (3) a semantic enhanced loss. In the second stage, the embeddings between domains are aligned with three losses: (1) an adversarial domain alignment loss (2) a contrastive loss and (3) a nearest neighbor matching loss.", "questions": "(1) Why do we need two stages of training? Is it really necessary to have two completely different sets of novel loss functions in each stage?", "limitations": "Adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The method is theoretically motivated.\n\n(2) The paper follows a logical orders.\n\n(3) Experiments appear to be complete.", "weaknesses": "(1) The method is clearly described and seems to be theoretically motivated. However, it is hard to understand intuitively why each loss is necessary. In particular, why we must use six different versions of the contrastive loss across two stages? (IPM, INCE, PNCE, SEL, SPR, SN2M). The theory only seems to justify the IPM loss. \n\n(2) In my opinion, even for someone well versed in metric learning, this method is hard to grasp. Some examples:\n\n - In line 148, the method applies k-means with a variable number of clusters determined by the \"Elbow approach\" and a contrastive loss on top of the cluster centroids. Just this one paragraph requires the person implementing the algorithm to reference another paper and implement a clustering algorithm.\n\n- The argument, starting at line 152, explaining the IPM loss is hard to understand, mostly because of the unusual notation (arrows and xor symbols).\n\n- The argument for the SN2M loss, starting at line 235 is unclear to me. \n\n(3) Overall, the method reads like a series of steps that do not follow one central motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The necessity illustration of each loss and the theory justification\n\nFirstly, we did not use six versions of contrastive loss. **IPM** combines INCE and PNCE with the intuitive goal of performing categorical semantic learning on unlabeled domain data. **INCE** forms the basis of unsupervised semantic learning. However, relying solely on INCE results in unclear boundaries between categories. Fortunately, introducing **PNCE** can enhance the distinction between categories, but determining the optimal timing to introduce PNCE requires careful consideration. We use a sigmoid function to weight the INCE training and then introduce varying degrees of PNCE accordingly. Additionally, we design a prototype conversion strategy to provide a cross-domain unified prototypical structure for PNCE. However, this IPM suffers from single prototype allocation mistakes and errors. To compensate for this issue, we propose **SEL**, which considers the relationship between a single instance and all prototypes, thereby alleviating erroneous optimization caused by single prototype allocation mistakes. **IPM and SEL constitute the first stage of UEM optimization.**\n\nIn the second stage of UEM, we recognize that the primary reason for the inaccuracy of existing cross-domain matching work is the neglect of the domain gap. However, regular domain alignment methods are not suitable for our UEM framework as they severely destroy the prototypical structure learned in the first stage. Therefore, we modified standard domain adversarial training by introducing **semantic preservation constraints (SPR)**. We also design a more accurate cross-domain matching algorithm that aligns with the unified prototypical structure, which is **SN^2M**. SN^2M assesses the reliability of cross-domain pairs by the consistency of the nearest prototypes in their respective domains and then decides whether to treat the pair as a positive pair in contrastive learning. Accordingly, **the second stage of UEM optimization is guided by SPR and SN^2M.**\n\nIn our current work, we have only conducted theoretical analysis on IPM. Due to the high complexity of entanglement between feature geometry distribution and semantics in high-dimensional space, theoretical analysis of SPR and SN^2M is challenging in supervised learning. Considering unsupervised cross-domain scenarios makes it even more difficult. Thus, we will attempt to provide theoretical explanations for this part in future work.\n\n> Detailed description of clustering and Elbow approach\n\nThanks for your suggestion. We will incorporate the following description of clustering and the Elbow approach into our future revision. The Elbow approach requires a pre-set maximum cluster number, which is 100 in our implementation. Then we repeatedly apply K-Means to the memory bank of each domain with the cluster number increasing from 2 to 100. For each run, we record two metrics. One is the within-cluster sum of squares (WCSS) which measures the sum of squared distances between each data point and its assigned centroid, reflecting the compactness of the clusters. The other is the silhouette score (SS) which measures how similar a data point is to its own cluster compared to other clusters. Then we draw two curves with the cluster number as the x-axis and the metric value as the y-axis. To determine the elbow point, we select the farthest point below the line passing through the first point and the last point of the curve w.r.t Euclidean distance as the estimated cluster number. After obtaining the respective estimated cluster numbers from the WCSS and SS curves, we select the larger one as the final estimation. \n\n\n> More explanation of the IPM\n\nPlease refer to the explanation and description of cross-domain prototype conversion in the Global Response.\n\n\n> More explanation of the SN2M\n\nPlease refer to the explanation and description of cross-domain instance matching in the Global Response.\n\n> The central motivation of the two-stage training\n\nAs mentioned in the introduction, the successful achievement of cross-domain retrieval (CDR) relies on solving two problems: 1) effectively distinguishing data samples in each domain, and 2) achieving alignment across domains for samples of the same category. Most existing methods adopt a two-stage processing strategy, where self-supervised learning (SSL) is first used for categorical semantic learning on unlabeled data, followed by nearest neighbor searching for cross-domain categorical matching. Our UEM framework also employs this two-stage strategy and introduces several novel designs and algorithms during the training phases to address the U^2CDR problem.\n\nWe considered whether it might be possible to solve the U^2CDR problem with a single-stage, end-to-end framework. The answer is no because we cannot introduce additional constraints during the first stage of SSL, such as domain alignment or cross-domain categorical matching. **These constraints would significantly affect the original optimization goals and directions of SSL, deviating the model from learning categorical semantics, which is the basis of everything.** To validate our thinking, we carry out experiments with a single-stage design by incorporating domain alignment (from the second stage of UEM) and SN^2M into the first stage. Below are the experiment results for close-set, partial, and open-set UCDR (in each cell of open-set UCDR, the left number is Shared-set mAP@All, and the right is Open-set detection accuracy). As shown in the table, **the performance of the single-stage process is indeed much poorer compared to the two-stage design.**\n\n|||Office-31|Office-Home|DomainNet|\n|-|-|-|-|-|\n|Close-set UCDR|Single-stage UEM|64.7|40.4|24.9|\n||UEM|81.9|52.4|34.4|\n|Partial UCDR|Single-stage UEM|45.0|30.5|24.4|\n||UEM|63.0|46.0|31.3|\n|Open-set UCDR|Single-stage UEM|57.7, 65.3|39.8, 60.9|25.2, 59.8|\n||UEM|77.4, 92.5|50.2, 86.7|34.8, 80.9|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcOW8GR45G", "reviewer": "Reviewer_ssYY", "summary": "This paper introduces the problem of Universal Unsupervised Cross-Domain Retrieval (U2CDR) and proposes a two-stage semantic feature learning framework to address it. The framework includes a cross-domain unified prototypical structure established through an instance-prototype-mixed contrastive loss and a semantic-enhanced loss in the first stage, and a modified adversarial training mechanism to ensure minimal changes during domain alignment in the second stage. Extensive experiments demonstrate that this approach significantly outperforms existing state-of-the-art CDR methods in solving U2CDR challenges.", "questions": "While this paper introduces a new problem, where exactly is the novelty in the methodology section?", "limitations": "No limitations or negative impacts have been identified in this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a new problem, namely Universal Unsupervised Cross-Domain Retrieval, and proposes an initial solution.\n2. The paper first formulates the problem and then introduces the proposed method in a hierarchical manner, which is clear and well-structured.\n3. The ability to perform U2CDR has broad implications for various applications, such as image search, product recommendations, and artistic creation.", "weaknesses": "1. The main effort of the paper seems to be on designing an optimization method. However, the optimization methods involved appear to be mostly existing ones. The authors should enhance the description of the novelty.\n2. Although the paper uses $L_{SPR}$ to maintain the semantic structure within domains, how to maintain the relationship between the positive pairs across domains should be emphasized.\n3. The analysis related to the Ablation Study seems insufficient. It would be beneficial to analyze the reasons for the experimental results in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Description of the methodology novelty\n\nPlease refer to the novelty illustration in the Global Response. In addition to the ablation study, To further validate the novelty of semantic structure preservation and cross-domain matching, we carry out experiments with the replacement of another state-of-the-art binary graph-based semantic preservation approach (SPPA [1]) and nearest neighbor searching algorithm (CD^2NN [2]). SPPA preserves the instance-level cosine similarity within each cluster and the prototype-level Euclidean distance across clusters. CD^2NN determines cross-domain nearest instance pair by seeking neighboring instance consistency. The experiment results in open-set CDR (in each cell, the left number is the Shared-set mAP@All, while the right is the Open-set detection accuracy) are shown below, demonstrating that UEM's semantic structure preservation and cross-domain matching are more effective.\n\n|         | Office-31  | Office-Home | DomainNet  |\n|---------|------------|-------------|------------|\n| UEM w/ SPPA  | 76.0, 88.9| 47.2, 84.4  | 31.5, 75.7   |\n| UEM w/ CD^2NN | 76.4, 89.4 |   46.5, 85.0 | 32.1, 76.4 |\n| UEM     | 77.4, 92.5 |  50.2, 86.7  | 34.8, 80.9 |\n\n[1] Lin, et al. Continual semantic segmentation via structure preserving and projected feature alignment. ECCV 2022\n\n[2] Liu, et al. Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023\n\n> How does UEM maintain the relationship between positive pairs across domains\n\nThank you for your suggestion. In fact, **cross-domain positive pairs are maintained through the construction of that unified prototypical structure.** During the first stage of our UEM framework, all instances in each domain approach their nearest prototype. These prototypes undergo a prototype conversion process, making them cross-domain unified, meaning that prototypes of the same category have the same geometric relationship relative to other categories within each domain. **In this scenario, cross-domain positive pairs will be close to the prototype of the same category in their respective domains.**\n\nThe relationship of cross-domain positive pairs is preserved during the second stage of UEM. As the prototypical structures across domains are unified, **the inner-domain relationship preservation achieved by our semantic-preserving domain alignment actually equals to preserving inter-domain relationships, which definitely includes the relationship of cross-domain positive pairs. Besides, the relationship of cross-domain positive pairs is further reinforced in SN^2M,** which is a primary objective of cross-domain matching.\n\n> More analysis of the ablation study\n\nThanks for your suggestion. We provide more analysis and reasoning for the ablation study as follows, and will incorporate them into the future revision.\n\nThe performance degradation caused by not using Prototype Merging may be attributed to the geometry distinctness of instance discrimination learning within each domain (Theorem 3.1). The high-level intuition is that the same categories across domains are forced to distinct geometry locations due to distinct contrastive comparisons with different category spaces. In this case, there are plenty of mismatches and misalignments in cross-domain matching. Therefore, the UEM performs poorly without using Prototype Merging. \n\nAs for the performance loss due to SEL, we think it originated from the unavoidable errors of the arbitrary prototype allocation in the prototype contrastive loss. Different from this single prototype allocation, SEL considers the potential relationship between the instance and all prototypes. This design provides the possibility of error correction and can effectively compensate for the prototype contrastive loss.\n\nAccording to Table 4, there is the largest performance gap between the 'Ours w/o SPDA' approach and the full 'Ours' approach. This indicates the semantic structure destruction during standard domain adversarial training. The semantic structure learned by the first stage is the basis of cross-domain matching, thus causing such a large performance drop.\n\nTo test the effectiveness of SN^2M, we replace it with the neighbor searching used in UCDIR, which searches for the nearest cross-domain instance and prototype in terms of cosine similarity distance to approach. Apparently, this search strategy has a lot of errors, which is worse if there is a domain gap. By contrast, our SN^2M can measure the reliability of the nearest cross-domain instance and then decide whether to approach it. Moreover, with the prototype translation and merging, the nearest cross-domain prototype is more accurate and reliable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response

We would like to thank all the reviewers for their constructive comments and suggestions. In the global response below, we respond to some common questions and present more visualization in the attached PDF.

> [For Reviewers gZ1R and ssYY] Methodology novelty
+ **Unified Prototypical Structure**: We identified the challenges of applying standard contrastive learning (CL) to U^2CDR, with the most significant being the cross-domain geometry distinctness. To tackle this issue, we innovatively perform cross-domain translation on each domain’s prototypes, followed by a unified merging strategy. This ensures a unified prototypical structure during prototype CL. To our knowledge, no prior work has employed such cross-domain prototype conversion as we have.
+ **Single Prototype Allocation Compensation**: In the prototype CL, the single prototype allocation can often be inaccurate, especially in the early stages of training. To compensate for this inaccuracy, we design a semantic-enhanced loss (SEL) that considers the relationship of an instance with all prototypes and directly optimizes the geometric distances between them. We have not seen this specific design in existing research.
+ **Optimal Timing for IPM**: Practically, we need instance CL to gradually learn the semantic prototypical framework of a domain, followed by prototypical CL and SEL to further stabilize and refine this framework. Determining the optimal timing to combine these losses is a challenging issue. For example, UCDIR divides the process into three stages, using different weights to combine instance and prototype CL at each stage. Our UEM resolves this issue by utilizing the Sigmoid function for weighting throughout the training process.
+ **Semantic-preserving Domain Alignment**: We discovered that the primary reason for the inaccuracy of existing cross-domain matching algorithms is the neglect of the domain gap. However, standard domain alignment methods are not suitable for our UEM framework as they severely destroy the prototypical structure learned in the first stage. Therefore, we modified the standard domain adversarial training by introducing semantic preservation constraints. These constraints utilize cosine similarity and Euclidean distance to associate all instances pairwise. This pairwise association is simple yet effective as any change in an instance affects the cosine similarity and Euclidean distance of all other instances. Although geometric structure preservation has been explored in continual learning using multi-node graphs based on graph neural networks [1], our UEM’s semantic preservation approach is much simpler and unseen before.
+ **Mismatch Filtering in Nearest Neighbor Searching**: We found that existing cross-domain retrieval work neglects to handle the inevitable mismatches in nearest neighbor searching, which usually occur at cluster boundaries. Thus, we designed SN^2M to evaluate the consistency of cross-domain neighboring instance pairs relative to their prototypes in each domain and determine whether to optimize the instance pair to be closer. Experimental results show that our SN^2M is highly accurate and perfectly aligns with the previously mentioned prototype translation and merging, distinguishing it completely from existing cross-domain matching algorithms.

[1] Yu, Da, et al. Contrastive correlation preserving replay for online continual learning. IEEE TCSVT 2023

> [For Reviewers gZ1R and tjhd] More explanation about cross-domain prototype conversion and cross-domain instance matching

We elaborate on these two aspects below and will incorporate them in the revision of the paper. To simplify the explanation, we avoid using mathematical symbols here but will do so in paper revision.

+ **Cross-Domain Prototype Conversion**: We design the cross-domain prototype conversion strategy to unify the prototypical structures of different domains. This strategy involves the following four steps:
    + As an example, for prototype conversion of domain A, we first translate all prototypes of domain B along the vector connecting the centers of the two domains to domain A.
    + Next, all prototypes in domain A use the Hungarian algorithm to find the nearest translated domain B prototypes.
    + For each prototype pair determined by the Hungarian algorithm, we check if they satisfy the merging condition (Eq.9 in the paper). If they do, the prototypes are merged by averaging; if not, they remain unchanged.
    + Finally, the prototype set for domain A is composed of the merged prototypes, the unmerged original domain A prototypes, and the unmerged translated domain B prototypes.

We visualize this prototype conversion as Figure 1 in the attached PDF.

+ **Cross-Domain Matching**: Our SN^2M can assess the reliability of cross-domain instance pairs and then decide whether to include them as positive pairs in contrastive learning. The specific steps are as follows:
    + The prototypes of domains A and B are transformed according to the previously described prototype conversion strategy.
    + For an instance $x^A$ in domain A, we find its nearest domain A prototype $p_{x^A}^A$ based on the product of cosine similarity and Euclidean distance. We also find $x^A$'s nearest domain B instance $x_{x^A}^B$ using the same criteria (refer to Eq.15 and Eq.16 in the paper).
    + For the nearest domain B instance $x_{x^A}^B$, we then find its nearest domain B prototype $p^B_{x_{x^A}^B}$.
    + If $p^B_{x_{x^A}^B}$ matches $p_{x^A}^A$ across domains (i.e., if we translate $p_{x^A}^A$ from domain A to B, the translated $\widetilde{p_{x^A}^A}$ is the same as $p^B_{x_{x^A}^B}$), we consider $x^A$ and $x_{x^A}^B$ as a positive pair in contrastive learning (Eq.17). Otherwise, we only consider $x^A$ and $\widetilde{p_{x^A}^A}$ as a positive pair in contrastive learning.

We visualize the SN^2M process as Figure 2 in the attached PDF.

Meta-review: The paper introduces the problem of Universal Unsupervised Cross-Domain Retrieval  for the first time and designs a novel two-stage semantic feature learning framework to effectively address it.  All reviewers have acknowledged the technical novelty and importance of tackling the “universal” problem. The rebuttal effectively addressed most concerns raised by the reviewers, leading to final ratings of Borderline Accept, Weak Accept, and Accept. Overall, the proposed framework offers a robust solution for cross-domain retrieval without supervision. Thus, the AC agrees with the reviewers and recommends "accept".
Decision: Accept (poster)