Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrieval

Authors: lixu  wang
xinyu  du
qi  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 09:45:40

Abstract: Cross-domain retrieval (CDR) is finding increasingly broad applications across various domains. However, existing efforts have several major limitations, with the most critical being their reliance on accurate supervision. Recent studies thus focus on achieving unsupervised CDR, but they typically assume that the category spaces across domains are identical, an assumption that is often unrealistic in real-world scenarios. This is because only through dedicated and comprehensive analysis can the category composition of a data domain be obtained, which contradicts the premise of unsupervised scenarios. Therefore, in this work, we introduce the problem of **U**niversal **U**nsupervised **C**ross-**D**omain **R**etrieval (U^2CDR) for the first time and design a two-stage semantic feature learning framework to address it. In the first stage, a cross-domain unified prototypical structure is established under the guidance of an instance-prototype-mixed contrastive loss and a semantic-enhanced loss, to counteract category space differences. In the second stage, through a modified adversarial training mechanism, we ensure minimal changes for the established prototypical structure during domain alignment, enabling more accurate nearest-neighbor searching. Extensive experiments across multiple datasets and scenarios, including close-set, partial, and open-set CDR, demonstrate that our approach significantly outperforms existing state-of-the-art CDR methods and other related methods in solving U^2CDR challenges.

Main paper: 
 introduction: 
 In real-world applications, cross-domain retrieval (CDR) finds extensive utility across diverse domains, such as image search [1] , product recommendations [2] , and artistic creation [3, 4] . However, the efficacy of current CDR methods relies heavily on accurate and sufficient supervision [5, 6] to provide categorical or cross-domain pairing labels. The acquisition of such information demands costly efforts and resources. Hence, there is an urgent need to develop unsupervised CDR techniques.For the regular Unsupervised CDR (UCDR) problem [7, 8] , there are two data domains with semantic similarity but distinct characteristics: the query domain and the retrieval domain. Despite the absence of category labels, regular UCDR typically assumes that the label spaces of both domains are identical. However, in real-world applications [5] , the categorical composition of an unlabeled data domain is usually uncertain, which is hard to acquire without detailed analysis and dedicated expertise. In this work, we focus on extending UCDR to more universal scenarios, which allow for the possibility of disparate category spaces across domains. The objective of this Universal UCDR (U 2 CDR) problem is to retrieve samples from the retrieval domain that share the same category label with a query sample more recent works [7, 8, 17] focus on achieving unsupervised CDR (UCDR). CDS [14] proposes a contrastive learning-based cross-domain pre-training to align different domains. PCS [13] incorporates prototype contrastive learning [11] into the cross-domain pre-training. Recent studies also search ways like clustering [7] , pseudo-labeling [18] , classifier mixup [17] , and data augmentation [8] to achieve more advanced CDR. However, all these UCDR works assume that the query and retrieval domains share the same category space. Although there is a study [19] that can achieve CDR with distinct categories, its effectiveness relies on accurate and sufficient data labels.Universal Cross-Domain Learning. Cross-domain learning consists of domain adaptation (DA) and domain generalization (DG) [20] . Regular DA and DG also only consider the scenario where the label space of the target domain is the same as the source label space, which is termed Closeset DA/DG. Recently, more studies have realized that the target label space may be a subset of the source one (i.e., Partial DA/DG) [21, 22] or contain some private labels that other domains do not have (i.e., Open-set DA/DG) [23, 24] . To deal with more universal setups, UniDA [25] unifies entropy and domain similarity to quantify sample transferability across domains. CMU [26] extends transferability quantification into entropy, consistency, and confidence. More recent works search ways like clustering [27, 28] and nearest neighbor matching [29, 30] to achieve universal DA/DG. In addition, some studies appear to achieve unsupervised DG where the source domain is also unlabeled [31, 32] , which is similar to the setup of UCDR. However, these studies consider the classification task and cannot effectively work in image retrieval, especially in completely unsupervised cases. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are discussed in Section 6 of the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 In the problem of U 2 CDR, we assume there are two domains characterized by N A and N B unlabeled instances, which are denoted asD A = {x A i } N A i=1 and D B = {x B i } N B i=1, respectively. Although these two domains are provided as unlabeled data without category labels, we assume their label spaces Y A , Y B consist of C A and C B different categories, and there is a relationship thatC A ̸ = C B , Y A ∩ Y B ̸ = Y A ∪ Y B .Without losing generality, if we regard domain A as the query domain, while domain B is the retrieval domain, the objective of U 2 CDR is to retrieve correct data from domain B that belongs to the same categories as the query data provided by domain A. To achieve this objective, it is required to train a valid feature extractor f θ : X → R that can map both these two domains from the input space X to a feature space R. Then the retrieval process R(f θ , x A i ) is shaped like for a particular query instance x A i with the label y A i from domain A, the representation distance between all instances in domain B and x A i needs to be calculated to form a set, i.e., S = {d(f (x B j ), f (x A i ))} N B j=1 where d(•) is a particular distance metric (e.g., Euclidean Distance), and we haveEQUATIONwhere sort ↑ (•) means ascending order sorting, and [1 : k] denotes the first k elements of a set.Method Overview. To solve U 2 CDR, we propose a Unified, Enhanced, and Matched (UEM) semantic feature learning framework that consists of two stages -Intra-Domain Semantic-Enhanced (IDSE, Section 3.2) Learning and Cross-Domain Semantic-Matched (CDSM, Section 3.3) Learning, which is shown in Figure 1 . IDSE can help the feature extractor f θ to extract categorical semantics and ensure a unified semantic structure across domains at the same time, which is achieved by instanceprototype-mixed (IPM, Section 3.2.1) contrastive learning and a novel Semantic-Enhanced Loss (SEL, Section 3.2.2). After IDSE, CDSM conducts Semantic-Preserving Domain Alignment (SPDA, Section 3.3.1) to minimize the domain gap while preserving the semantic structure learned by IDSE.With the minimization of the domain gap, more accurate nearest-neighbor searching can be achieved by our Switchable Nearest Neighboring Match (SN 2 M, Section 3.3.2).To achieve effective cross-domain retrieval, feature extractor f θ needs to learn consistent crossdomain features to differentiate data categories. Instance Discrimination [9] is usually employed to achieve discriminative feature learning, but directly applying it in U 2 CDR has four fundamental issues that hinder the possibility of accurate cross-domain categorical matching later:1) Instance discrimination tends to extract semantics that separate domains rather than categories,EQUATION2) Instance discrimination cannot characterize categorical semantics in the feature space,EQUATION3) The randomness introduced by stochastic data augmentations results in evident changes in learned categorical semantic structures during training, i.e.,EQUATION)) ≫ minEQUATIONwhere G(•) corresponds to a graph constructed by the input vectors, and P denotes the set of categorical prototypes for a domain. The subscript t denotes different training iterations, while H represents the hypothesis space of possible categorical prototype sets for a particular domain. d(•) here is a measurement for graph difference, e.g., graph edit distance [33] .4) Distinct label spaces make instance discrimination learn different categorical semantic structures:Theorem 3.1 (Geometry Distinctness). Suppose data distributions of two domains (A and B) have mutually disjoint supports, and they are uniform over these supports. Assuming the support sets of domains A and B are not identical, the optimal feature extractors f * that minimize the instance discrimination loss of different domains present distinct geometric feature spaces.To fix the above issues, we adopt a slowly momentum-updated contrastive learning algorithm -MoCo [10] , to handle the third issue reflected by Eq. ( 4). Moreover, the MoCo-based instance discrimination is conducted separately for each domain, which encourages f θ to focus less on learning domain semantics (for the first issue, Eq. ( 2)). Besides, we accompany MoCo with a prototypical contrastive loss, to enhance the mapping of categorical semantics from the input space to the feature space (for addressing the second issue, Eq. ( 3)). With a well-crafted prototype update mechanism, this prototypical contrastive loss can also help build a unified semantic structure across domains (for the last issue, Theorem 3.1). Then let us introduce IPM contrastive learning in detail. First of all, two memory banks M A and M B are maintained for domains A and B, which store historical features m of data samples x:M A = m A 1 , ..., m A N A , M B = m B 1 , ..., m B N B , where m i ← βm i + (1 -β)f θ (x i ). (5)Here m i is initialized by the feature of x i extracted by the initial f θ and updated in momentum, where β controls the momentum speed, and we set it as a popular value 0.99. With these two memory banks, MoCo builds the positive pairs as the pair of each instance and its historical feature, while the negative ones are pairs of each instance and the historical features of all other instances:EQUATION)where B is the batch size and τ is a temperature factor that is set as 0.07.As for the design of our prototypical contrastive loss, K-Means is applied on M A and M B to construct prototypes as cluster centers P = {p c } C c=1 . In our problem, the cluster number C is unknown, thus we apply the Elbow approach [34] to estimate it as C. Then, for each instance x i , if it belongs to the c i -th cluster, the prototypical contrastive loss L PNCE shapes like,EQUATION)Until here, the first three issues can be fixed by mixing L INCE and L PNCE , but the last issue, Theorem 3.1, is the bottleneck of U 2 CDR. Next, let us introduce how we build a unified prototypical structure for L PNCE to address the last issue. Specifically, after obtaining the prototype sets P A , P B of domain A and B, if we take domain A as an example to illustrate the prototypical structure building process, the prototype set P B of domain B will be translated to domain A as,EQUATIONwhere M denotes the average vector of all vectors in M. Next, each prototype p A c ∈ P A searches its closest p B→A c ′ ∈ P B→A (we use Hungarian algorithm to search the closest cross-domain prototypes) for the opportunity of merging, which needs to satisfy the condition,EQUATIONwhere d(•, •) computes the Euclidean distance. If we use the symbol ⊕ to identify prototypes that satisfy this merging condition, the final prototypical structure for domain A isP A ′ = P A \ P A,⊕ ∪ P B→A \ P B→A,⊕ ∪ P A,⊕ ⊕ P B→A,⊕ where P A,⊕ ⊕ P B→A,⊕ = p B→A,⊕ c ′ + p A,⊕ c /2 C ⊕ c=1. Then the computation of L A ′ PNCE -Eq. ( 7) for domain A is conducted on the newly-built P A ′ , and all these operations are same for domain B.However, as establishing the semantic cluster-like structure requires time, it is unreasonable to conduct prototype contrastive learning from the beginning of training. Therefore, we conduct instance discrimination at the beginning and progressively incorporate L PNCE . In this case, not only are the constructed cluster centers more reliable, but the Elbow approach also provides more accurate cluster number estimations. Specifically, we use a coefficient α that is scheduled by a Sigmoid function to control the incorporation weight of L PNCE , i.e.,EQUATIONE and e here are the overall training epochs and the current epoch for IDSE.For the IPM contrastive learning, it is arbitrary to allocate a data instance x i to a single cluster when the preferred semantic prototypical structure cannot be learned in advance. As a result, to speed up the structure-building process, we propose a novel Semantic-Enhanced Loss (SEL) to align data instances with the prototypes better. Specifically, instead of assigning data instances with a single cluster, SEL considers potential semantic relationships between instances with all clusters, which are measured by the Softmax probability. Moreover, as both P A and P B are obtained by the Euclidean distance-based K-Means, we directly minimize the Euclidean distance between samples and prototypes:EQUATIONwhere C denotes the number of prototypes after merging, e.g., C A is the number of elements inP A ′. By taking all potential semantic correlations into account, SEL can alleviate the impact of the noise within the K-Means clustering results and further guide the model to learn more distinguishable semantic information in terms of Euclidean distance. Certainly, such SEL benefits also rely on high-quality semantic prototypical structures. As a result, we also apply the progressive coefficient α to SEL, then the final optimization objective for IDSE isEQUATION)3. annotation can be utilized in U 2 CDR. Instance matching [7, 32, 13] is proposed to match an instance x A i to another instance x B j in the other domain with the most similar features. However, due to the domain gap, instances can be easily mapped mistakenly. For example, if there is an instance in one domain that is extremely close to the other domain, it will be determined as the nearest neighbor for all instances in the other domain [13] , shown in Figure 2 . As a result, before conducting instance matching, the domain gap needs to be diminished. Existing works usually leverage discrepancy minimization [35] or adversarial learning [36] to achieve domain alignment. However, these methods provide inferior performance due to semantic categorical structure changes, i.e., the semantic correlations among instances within domains change a great deal during domain alignment, shown in Figure 3 .To achieve more effective domain alignment, especially with semantic preservation, we propose Semantic-Preserving Domain Alignment (SPDA). Similar to the standard domain adversarial learning, SPDA happens on two parties, one is the feature extractor f θ , and the other is a domain classifier g ω parameterized on ω. The domain classifier tries to distinguish the representations of two domains, while the feature extractor tries to fool the domain classifier. Thus, the training is shaped like a bi-level optimization in terms of the domain classification task on θ and ω, shown as follows,EQUATIONwhere y here denotes the domain label, e.g., if we regard y = 1 for domain A, the domain label of domain B is y = 0. After sufficient adversarial training, the feature extractor captures nearly domain-invariant features, thus achieving domain alignment.To prevent the semantic structure from being changed, we first make a copy for the model trained by IDSE and denote it as f ′ θ . Then for a mini-batch of a particular domain, we feed all instances to f ′ θ and calculate pair-wise cosine similarity and Euclidean distance. In this way, all these instances constrain and influence each other, which means that when the correlation of a particular instance pair changes, it will affect the correlations of other related pairs. Then, we apply a semantic-preserving regulation into domain adversarial learning to make the pair-wise correlations unchanged,EQUATION)Recall IPM Contrastive Learning. Actually, aligning two domains together without any semantic structure change is impossible. As a result, there is a need for a dedicated design to alleviate the impact of such unavoidable changes. Our solution is to strengthen the instances' semantic correlations by enhancing the cluster's inner density and inter-separability. For the final convergence of IPM contrastive learning, each instance is optimized to get as close to its corresponding cluster prototype as possible, and as far to other cluster prototypes as possible:Theorem 3.2 (Convergence of IPM). Suppose the data distribution of a domain has mutually disjoint supports, and it is uniform over these supports. Simplex Equiangular Tight Frame (ETF) representations [37] minimize the Instance-Prototype-Mixed Loss of this domain.With SPDA, the domain gap could be effectively minimized to enable more accurate cross-domain instance matching. However, existing instance matching approaches [7, 13, 32] lack the capability of measuring the matching reliability between an instance and its nearest neighbor, which allows us to conduct cross-domain matching with different weights. For example, if an instance is located at the joint boundary of multiple categories, which indicates that the current feature extractor cannot extract sufficiently distinguishable semantic features for this instance, we are supposed to lay less emphasis on this case. To fix such issues, we propose the Switchable Nearest Neighboring Match (SN 2 M).The principle behind SN 2 M is that prototypes are more convincing and reliable. Specifically, for a particular sample x A i in domain A, we first determine its inner nearest cluster prototype p A ci in domain A. We can also search for the nearest instance x A,B i in domain B. Both these two searching processes are based on the product of a modified cosine similarity and Euclidean distance,EQUATIONx A,B i = arg minEQUATIONAfter obtaining x A,B i , SN 2 M searches for its inner nearest prototype p A,B ′ ci in P B ′ (which has been merged with the translated prototype set P A→B from domain A) and two cases allows us to measure the reliability of x A,B i . Before introducing these two cases, the inner nearest prototype p A ci of x A i needs to be translated to domain B and checked whether should be merged to follow condition Eq. ( 9), and we denote the translated prototype asp A ci . Then the first potential case is p A,B ′ ci is exactly identical to p A ci , which means, x A,Bi is convincing since it shares the same prototype correlations with x A i across two domains. Then the pair of x A,B i and x A i should be viewed as a positive pair in the contrastive loss. Otherwise, if p A,B ′ ci is different from p A ci , it may be located at the intersection region of multiple clusters. In this case, the pair of x A,B i and x A i is not supposed to be treated as a positive pair. However, it does not mean SN 2 M does nothing for these unreliable cases, instead, SN 2 M leverages a modified prototype contrastive loss to match the pair of x A i and p A ci . The modification is to incorporate cross-domain instance-wise negative comparison,EQUATIONFinally, the overall optimization objective of CDSM followsEQUATION)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The main theoretical insights are discussed in Section 3 and detailed proofs are given in Section A of the Appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 The datasets, experimental settings, and comparison baselines are introduced below. More implementation details, experiment results, and source codes are provided in the Supplementary Materials.Datasets. Office-31 [38] includes three domains with 31 classes: Amazon (A), DSLR (D), Webcam (W). Office-Home [39] contains four different domains: Art (A), Clipart (C), Product (P), Real (R).And each domain has 67 data categories. DomainNet [40] is the most challenging cross-domain dataset to our best knowledge, which includes six domains: Quickdraw (Qu), Clipart (Cl), Painting (Pa), Infograph (In), Sketch (Sk) and Real (Re). DomainNet is originally class-imbalanced, thus we follow [7] to select 7 data classes that contain more than 200 samples.Experiment Settings. For fair comparison, we apply ResNet-50 [41] pre-trained with ImageNet in MoCov2 [10] as the feature extractor. The domain classifier consists of two fully-connected layers.The SGD optimizer with a momentum of 0.9 is adopted with an initial learning rate of 0.0002 that is scheduled to zero by a cosine learning strategy. The batch size is 64. The training epochs of IDSE are 100 for Office-31 and Office-Home, and 200 for DomainNet. The epoch number of CDSM is 50 for all three datasets. Following [17] , we adopt mean average precision on all retrieved results (mAP@All) to measure the performance. All experiments are run repeatedly 3 times with seeds 2024, 2025, and 2026, and we report the mean performance and standard deviation.Comparison Baselines. Our proposed method is compared with a comprehensive set of state-of-theart works from Cross-Domain Representation Learning (CDS [14] , PCS [13] ), Unsupervised Domain Generalization (DARL [31] , DN2A [32] ), and Unsupervised Cross-Domain Retrieval (UCDIR [7] ,CoDA [17] , DGDIR [8] ). We follow their default settings and only conduct compulsory customization. (Office-Home can be found in the Appendix). According to these results, we can observe that UEM significantly outperforms other baselines in all cases. Specifically, we can achieve mAP@All improvement of 2.6% compared to the best baseline method on Office-31, and such improvement is even larger on DomainNet with an average of 3.6%. Figure 4 also shows the retrieval results of our approach on Office-31, Office-Home, and DomainNet, and we can observe that all results are correct.Partial Unsupervised Cross-Domain Retrieval. To establish the partial setting, the query domain contains only half of the label space of the retrieval domain, and the query label space is randomly selected. As shown in Table 2 , we can observe that UEM exceeds all other baseline methods significantly in mAP@All, which is much more substantial than close-set UCDR. For example, UEM outperforms the second-best with an average of 12.4% on Office-Home. Similar improvements on Office-31 and DomainNet can be observed (results are provided in the Appendix). Besides, we can easily observe that existing state-of-the-art studies are nearly incapable of dealing with the label space difference in partial UCDR (see Theorem 3.1), while our UEM can work effectively.Open-set Unsupervised Cross-Domain Retrieval.As for the open-set setups, we ensure that the label space of the retrieval domain is half of the query label space. The experiment results for DomainNet (results for Office-31 and Office-Home are in the Appendix) are presented in Table 3 with two metrics -mAP@All for the shared label set, and detection accuracy for the private (open-set) query labels (please refer to Appendix for how to detect the private query labels). According to these results, we can easily observe that our approach substantially exceeds other baseline methods in both metrics. Similar trends can also be found for Office-31 and Office-Home. All these results strongly validate the effectiveness of UEM in open-set UCDR. All the ablation studies are carried out in open-set UCDR on three datasets, and the average metrics (Shared-set mAP@All and Open-set Acc) for all domain pairs of a single dataset are reported here.Effectiveness of Prototype Merging. When evaluating the effectiveness of a unified prototypical structure, we do not use prototype merging in IDSE. According to the results of 'Ours w/o P.M.' in Table 4 , there is a non-negligible performance drop in both shared-set mAP@All and open-set accuracy. This validates the importance of building a unified prototypical structure across domains.Effectiveness of SEL. When evaluating SEL, we detach it during the model training. According to the results of 'Ours w/o SEL' in Table 4 , there is also an evident performance drop compared to the full UEM. This validates that SEL is vital as it can help prepare a better base model for CDSM.Effectiveness of SPDA. We replace our SPDA with the standard domain adversarial learning for the ablation study. As shown in Table 4 , there is a significant performance difference between 'Ours w/o SPDA' and the full UEM, which illustrates the importance of semantic preservation during domain alignment, as well as indirectly verifying the necessity of SPDA.Effectiveness of SN 2 M. We also replace SN 2 M with the nearest neighboring search approach leveraged by UCDIR. By comparing the results of 'Ours w/o SN 2 M' and 'Ours', we can conclude that SN 2 M is more compatible with UEM and able to achieve more accurate cross-domain categorical matching.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All the implementation details to reproduce the results are given in Section 4 of the main paper and Section B of the Appendix. 
conclusion: 
 In this work, we focus on two major challenges when conducting cross-domain retrieval (CDR) in real-world scenarios: one is that the category space across domains is usually distinct, and the other is that both the query and retrieval domains are unlabeled.  
limitations and future work: 
 To solve the real-world challenges when employing cross-domain retrieval, especially considering the category distinctness across unsupervised data domains, we propose the UEM semantic feature learning framework in this work. Although extensive empirical evaluation and theoretical analysis have validated the effectiveness of UEM, some minor limitations still need more exploration. First, the current UEM framework is composed of two stages, and we empirically determine the switching point.In the future, we need to achieve a real end-to-end UEM by changing the training stage automatically.Besides, there is a lack of theoretical analysis for the second stage (CDSM). In future efforts, we should theoretically prove the semantic preservation of SPDA and the reliability of SN 2 M. Lastly, the general applicability of UEM also needs testing. For instance, cross-person generalization in wearable devices [42] and property analysis of material [43] or molecular [44] structures require cross-domain retrieval. Therefore, we should test UEM on other modalities like time series and graph data. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The broader impacts are discussed in Section D of the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not pose any explicit risks as we use public datasets, and applying our proposed methods to other areas needs dedicated modifications.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All dataset details and original authorship are cited in Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release any new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects, hence do not require IRB approval. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Theorem 3.1 (Geometry Distinctness). Suppose data distributions of two domains (A and B) have mutually disjoint supports, and they are uniform over these supports. Assuming the support sets of domains A and B are not identical, the optimal feature extractors f * that minimize the instance discrimination loss of different domains present distinct geometric feature spaces.Proof. Suppose a data distribution is made of mutually disjoint supports and distributed uniformly over these supports, there is a theorem demonstrating the representation geometry learned by instance discrimination in a research work [45] , i.e., Theorem A.1. Assuming a data distribution has mutually disjoint supports and is uniform over these supports, any Simplex ETF representation extracted by f minimizes L NCE (f ) for any convex and non-increasing loss function l. Moreover, if l is strictly convex (e.g., logistic loss), then Simplex ETF representations are the only minimizes of L NCE (f ).The Simplex ETF representations are defined as follows, Definition A.2 (Simplex ETF). A simplex ETF is a collection of equal-length and maximally equiangular vectors. We call a P × K matrix M an ETF if it satisfiesEQUATIONwhere α is a non-zero scalar, I is the identity matrix and 1 K is an all-ones vector.In representation learning, ETF representations mean that samples from different categories (x i , y i ), (x j , y j ) satisfyEQUATION)With the above preparations, we apply proof by contradiction to prove that different domains hold distinct geometric feature spaces when their respective instance discrimination loss achieves minimization. Specifically, we assume domains A and B share C categories and there is only one category exclusively owned by domain B, i.e.,EQUATIONThis assumption satisfies that the support sets of domains A and B are not identical. Then suppose that the geometric structures of domains A and B are identical when the instance discrimination loss has been minimized, according to Theorem A.1, both domains A and B present Simplex ETF representations. In this case, if we randomly select two categories y p , y q ∈ Y A ∩ Y B from the domain-shared category set, and their samples within each domain have the following relation,EQUATIONAccording to Eq. ( 21), Eq. ( 22) implies that the same category pair across domains has different cosine similarities, which contradicts the assumption of identical geometry across domains.Theorem 3.2 (Convergence of IPM). Suppose the data distribution of a domain has mutually disjoint supports, and it is uniform over these supports. Simplex Equiangular Tight Frame (ETF) representations minimize the Instance-Prototype-Mixed Loss of this domain.Proof. As shown in Eq. ( 10), the Instance-Prototype-Mixed loss is composed of instance discrimination and prototype contrastive loss. The simplex ETF representations have been proven to minimize instance discrimination in Theorem A.1. Then we only need to prove that the prototype contrastive loss satisfies the convex and non-increasing properties, Property A.3. For a loss function l defined on a set V = {v i } t i=1 with the size of t, it holds for all of index S ⊆ {1, ..., t} thatEQUATIONTo prove the non-increasing property of prototype contrastive loss, we can view L PNCE is built on the setV = {v c = f θ (x i ) • p ci /τ -f θ (x i ) • p c /τ } C c=1 , then L PNCE (x i ) = l log (V) := log(1 + c exp(-v c )). We can leverage the concavity of the log function (Jensen's inequality) and denote T := 1 + j / ∈S exp(-v j ), and we haveEQUATIONTherefore, the prototype contrastive loss also holds the non-increasing property, which proves that simplex ETF representations minimize the Instance-Prototype-Mixed Loss.In open-set unsupervised cross-domain retrieval (UCDR) settings, the query domain has some private categories that are not included in the retrieval domain. In this case, the retrieval results of query samples for such private query categories should be null. As aforementioned, for a query sample x A i , the retrieval process needs to calculate the distance between all samples in the retrieval domain and x A i . Then the most similar retrieval samples are supposed to be those located as close to x A i as possible. Intuitively, if x A i belongs to the private query categories, the nearest sample in the retrieval domain should be relatively distant. Therefore, there is a need for a threshold that allows us to determine whether the closest retrieval sample of a query sample is located too far to be a similar sample. Next, let us introduce how our UEM detects and identifies whether a query sample belongs to private query categories.In our UEM framework, the crucial design is to build a unified prototypical structure across domains, which also shapes the private label detection strategy. Specifically, after the training of CDSM, we apply K-Means to the query and retrieval domain datasets again to construct the prototype sets P A and P B . Then the prototypes of the retrieval domain are translated to the query domain for potential merging. The detailed prototype translation and merging have been introduced in Section 3.2.1. After the prototype merging, both P A and P B are divided into two groups by satisfying the merging condition (Eq. ( 9)) or not. For the prototype pairs that satisfy the merging condition, we record the maximum inter-sample distance among their clusters asEQUATIONEQUATIONThen for any query sample x A i , there are two cases for its belonging. One is x A i belongs to the clusters of prototypes unmerged, i.e., P A \ P A,⊕ . In this case, x A i is supposed to come from private query labels with high confidence. The other case is that the closest prototype of x A i satisfies the merging condition, i.e., arg minp A d(f θ (x A i ), p A ) = p A,⊕ c ⊕ ∈ P A,⊕, in which we should compare the recorded D c ⊕ and the minimal distance between all samples in the retrieval domain and x A i , i.e.,EQUATIONIf D c ⊕ < D A→B i , we have faith that there is no sample similar enough in the retrieval domain, which means x A should belong to private query categories. Otherwise, x A i comes from the shared label set, and we should conduct the normal retrieval operation.In our evaluation process, we implement a number of state-of-the-art baseline methods to compare our proposed UEM in U 2 CDR. For a fair comparison, we ensure two principles for all these used baselinesone is that the training data consists of at least two domains, and the other is that the training data is unlabeled. Following these two principles, in addition to unsupervised cross-domain retrieval studies [7, 8, 17] , two other problems share similar setups: cross-domain representation learning (CDRL) [14, 13] and unsupervised domain generalization (UDG) [31, 32] . For CDRL, the objective is to learn domain-generalizable representations that provide domain-transferable knowledge for any downstream task. One of the most typical downstream tasks of CDRL is cross-domain retrieval. As for UDG, in addition to achieving effective cross-domain representation learning, domain-generalizable classifiers are also needed. However, in our evaluation, the retrieval process does not require any classifier, thus we omit all techniques related to classifier training for the used UDG approaches.For the specific setups of our experiments, we consider close-set, partial, and open-set UCDR. The close-set UCDR assumes that the label spaces of the query and retrieval domains are identical, which is the benchmark setup of other baseline methods. In this case, we follow the default settings of these baseline methods to evaluate their performance in close-set UCDR. As for the partial UCDR, the label space of the query domain is half of the retrieval label space, where most baseline methods can work normally without any modification. But some baseline approaches (PCS [13] , UCDIR [7] , DGDIR [8] ), especially those based on prototype learning, require the knowledge of category numbers for domains, therefore, we suppose the category numbers are known to these approaches. The last open-set UCDR supposes the retrieval label space is half of the query label space. In this setup, the query domain has private categories, and if we conduct retrieval for samples from these categories, the retrieval results should be null. To detect private categories, we follow the strategy leveraged by UEM (Section B.1) to employ a similar one for the used baseline methods. Specifically, we divide all baseline methods into two groups by whether there is a dedicated nearest neighboring searching algorithm. For those (DARL [31] ) that don't have the nearest neighboring search, we attach the searching algorithm used by UCDIR [7] . After conducting all training and operations of any baseline method, we use the nearest neighboring search to pair samples from the query and retrieval domains. Moreover, we conduct K-Means in the query domain to build the prototype set P A (for CoDA [17] , we leverage its auxiliary classifiers to construct the prototype set). Then for each cluster in the query domain, we record the maximum inter-sample distance D A c . Note that the distance measurement here is different from the product of minus cosine similarity and Euclidean distance (Eq. ( 25)), and different approaches use diverse measurement, e.g., UCDIR [7] uses cosine similarity while DN2A [32] leverages Euclidean distance. Here we provide the additional experiment results including the close-set UCDR on Office-Home (Table 5 ), partial UCDR on Office-31 and DomainNet (Table 6 ), and open-set UCDR on Office-31 (Table 7 ) and Office-Home (Table 8 ). According to these results, we can obtain similar observations and conclusions to the main paper. First, our UEM can achieve the best performance in close-set UCDR, which is reflected by the average performance improvement of 3.1% on Office-Home. Moreover, UEM can exceed all other baseline methods much more substantially in both partial and open-set UCDR. Specifically, our methods outperform the best baseline with a margin of 5.0% on DomainNet and 12.0% on Office-31 in partial UCDR. As for open-set UCDR, UEM exceeds other baseline methods with a range of 6.8% and 7.9% on Office-31 for shared-set mAP@All and open-set detection accuracy respectively, and such improvement is much higher on Office-Home in 15.1% for shared-set mAP@All and 13.1% for open-set detection accuracy. In addition, if we compare the results of the same domain pairs between close-set and partial/open-set UCDR for baseline methods, we can observe that the performance drops a lot, which also validates the existence of geometry distinctness (Theorem 3.1). In particular, such geometry distinctness originating from the category space difference can incur a performance drop of up to 29% for DGDIR on Office-31 between close-set and partial UCDR.The research development for solving Universal Unsupervised Cross-Domain Retrieval (U 2 CDR) has the potential to make a significant positive impact across various sectors. By enabling more accurate and flexible retrieval of information across different domains without the need for supervision and the concern about semantic category distinctness, our proposed UEM framework can enhance user experiences in product recommendation systems, leading to more personalized and relevant suggestions. In the realm of artistic creation, UEM can facilitate novel connections and inspirations by retrieving cross-domain artistic elements, fostering creativity and innovation. Beyond these applications, UEM can also be beneficial in healthcare, finance, and scientific research with certain adaptive modifications. In healthcare, it can improve diagnostic tools and personalized treatment plans by integrating diverse data sources. In finance, it can enhance risk assessment and fraud detection by analyzing cross-domain financial data. In scientific research, it can accelerate discoveries by connecting insights from different fields. While UEM has substantial benefits, it is crucial that its deployment adheres to existing privacy and intellectual property protection regulations and policies.Ensuring that data used in cross-domain retrieval respects user privacy and intellectual property rights is essential to prevent misuse and maintain public trust. Thus, applying this research in the real world should consider ethical issues to ensure responsible and fair use, thereby aiming for a positive societal impact without any negative social consequences. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.Justification: All the experiment results are supplemented with standard deviations as the error bar resulting from 3 independent runs.Guidelines:• The answer NA means that the paper does not include experiments. Justification: The Appendix discusses all relevant computing requirements in detail.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: Our work adheres to all the ethical guidelines outlined by NeurIPS.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces the problem of Universal Unsupervised Cross-Domain Retrieval  for the first time and designs a novel two-stage semantic feature learning framework to effectively address it.  All reviewers have acknowledged the technical novelty and importance of tackling the “universal” problem. The rebuttal effectively addressed most concerns raised by the reviewers, leading to final ratings of Borderline Accept, Weak Accept, and Accept. Overall, the proposed framework offers a robust solution for cross-domain retrieval without supervision. Thus, the AC agrees with the reviewers and recommends "accept".

Reviews: review: paper_summary: paper_summary: This paper proposes Universal Unsupervised Cross-Domain Retrieval for the first time and designs a two-stage semantic feature learning framework to address it.
strength_weakness: strength_weakness: 1. In unified unsupervised domain adaptation, there is no handling of instances that are not common categories. Isn't this necessary?

2. From the perspective of innovation, the proposed unified prototype structure is interesting, and the rest is mostly incremental work, such as semantic structure preservation and adjacent feature matching in domain adaptation. From the visualization results, the author failed to prove the above contributions.

3. This paper should reflect the difference between universal domain adaptation and unsupervised domain adaptation.

4. This article does not have a better way to state the method, especially in cross-domain prototype conversion and close neighbor matching.
questions: questions: See Weaknesses section
limitations: limitations: no
paper_summary: paper_summary: This paper tackles the problem of unsupervised cross-domain retrieval. This is the problem where the query and retrieval domains are distinct. For example, in sketch to real retrieval, the system must retrieve the most relevant real images to a query sketch. "Unsupervised" refers to the fact that no labels are available during training, but the images from both domains are available. The authors claim to be the first to investigate the "universal" version of this problem, where the query and retrieval domains are allowed to have disjoint labels spaces. For this problem, the authors propose a two-stage optimization procedure. In the first stage, three losses are used: (1) an instance-wise contrastive loss (2) a cluster-wise contrastive loss and (3) a semantic enhanced loss. In the second stage, the embeddings between domains are aligned with three losses: (1) an adversarial domain alignment loss (2) a contrastive loss and (3) a nearest neighbor matching loss.
strength_weakness: strength_weakness: (1) The method is clearly described and seems to be theoretically motivated. However, it is hard to understand intuitively why each loss is necessary. In particular, why we must use six different versions of the contrastive loss across two stages? (IPM, INCE, PNCE, SEL, SPR, SN2M). The theory only seems to justify the IPM loss. 

(2) In my opinion, even for someone well versed in metric learning, this method is hard to grasp. Some examples:

 - In line 148, the method applies k-means with a variable number of clusters determined by the "Elbow approach" and a contrastive loss on top of the cluster centroids. Just this one paragraph requires the person implementing the algorithm to reference another paper and implement a clustering algorithm.

- The argument, starting at line 152, explaining the IPM loss is hard to understand, mostly because of the unusual notation (arrows and xor symbols).

- The argument for the SN2M loss, starting at line 235 is unclear to me. 

(3) Overall, the method reads like a series of steps that do not follow one central motivation.
questions: questions: (1) Why do we need two stages of training? Is it really necessary to have two completely different sets of novel loss functions in each stage?
limitations: limitations: Adequately addressed.
paper_summary: paper_summary: This paper introduces the problem of Universal Unsupervised Cross-Domain Retrieval (U2CDR) and proposes a two-stage semantic feature learning framework to address it. The framework includes a cross-domain unified prototypical structure established through an instance-prototype-mixed contrastive loss and a semantic-enhanced loss in the first stage, and a modified adversarial training mechanism to ensure minimal changes during domain alignment in the second stage. Extensive experiments demonstrate that this approach significantly outperforms existing state-of-the-art CDR methods in solving U2CDR challenges.
strength_weakness: strength_weakness: 1. The main effort of the paper seems to be on designing an optimization method. However, the optimization methods involved appear to be mostly existing ones. The authors should enhance the description of the novelty.
2. Although the paper uses $L_{SPR}$ to maintain the semantic structure within domains, how to maintain the relationship between the positive pairs across domains should be emphasized.
3. The analysis related to the Ablation Study seems insufficient. It would be beneficial to analyze the reasons for the experimental results in Table 4.
questions: questions: While this paper introduces a new problem, where exactly is the novelty in the methodology section?
limitations: limitations: No limitations or negative impacts have been identified in this paper.

Comments: title: Acknowledgement of Rebuttal
A Gentle Reminder of Further Feedback to Reviewer gZ1R
A Gentle Reminder of Further Feedback to Reviewer tjhd
Thanks to Reviewer ssYY
comment: Dear Reviewer tjhd,

Thank you for your positive feedback and insightful suggestions. We appreciate your recognition of our efforts to address your concerns and include more experiments. We are committed to expanding these clarifications and experiments to our next revised version.

We value your input and sincerely hope you consider raising your rating based on the improvements we’re implementing. Your endorsement would greatly enhance the credibility of our work. We will be also glad to answer any further questions.

Best regards,

Authors of Paper 5055
Dear Authors,

I read the rebuttal and appreciate the additional experiments and clarifications.
Dear Reviewer gZ1R,

The conclusion of the discussion period is closing, and we eagerly await your response. We greatly appreciate your time and effort in reviewing this paper and helping us improve it.

Thank you again for the detailed and constructive reviews. We hope our response is able to address your comments related to the handling of uncommon categories, the novelty description of semantic preservation and adjacent feature matching, and clarification of problem settings. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give us.

Best Regards,

Authors of Paper 5055
Dear Reviewer tjhd,

As the rebuttal discussion phase ends soon, we want to express our gratitude for your engagement thus far. We really want to check with you whether our response addresses your concerns during the author-reviewer discussion phase. We have diligently addressed every concern and question you raised during the initial review, and our efforts are aimed at enhancing the clarity and quality of our work.

We genuinely hope our responses have resolved your concerns related to the design intuition behind our methodology, the detailed description of the Elbow approach, IPM and SN2M losses, and the central motivation of the two-stage design. Your thoughtful evaluation greatly aids in our paper's refinement and strength. Again, we sincerely appreciate your dedication and time.

Best regards,

Authors of Paper 5055
Thank you for your positive feedback and insightful suggestions. We appreciate your recognition of our efforts to address your concerns and your rating arising. We will be also glad to answer any further questions. Thank you once again for your time and valuable feedback.
Thanks for the author's response. I decide to raise my score to Accept.