Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap
Authors: Christopher Liao
Christian So
Theodoros Tsiligkaridis
Brian Kulis
Venue: ICLR 2025 Poster (2025)
Keywords: Retrieval
Domain Generalization
Multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Domain generalization (DG) is an important problem that learns a model which generalizes to unseen test domains leveraging one or more source domains, under the assumption of shared label spaces. However, most DG methods assume access to abundant source data in the target label space, a requirement that proves overly stringent for numerous real-world applications, where acquiring the same label space as the target task is prohibitively expensive. For this setting, we tackle the multimodal version of the unsupervised domain generalization (MUDG) problem, which uses a large task-agnostic unlabeled source dataset during finetuning. Our framework does not explicitly assume any relationship between the source dataset and target task. Instead, it relies only on the premise that the source dataset can be accurately and efficiently searched in a joint vision-language space. We make three contributions in the MUDG setting. Firstly, we show theoretically that cross-modal approximate nearest neighbor search suffers from low recall due to the large distance between text queries and the image centroids used for coarse quantization. Accordingly, we propose paired k-means, a simple clustering algorithm that improves nearest neighbor recall by storing centroids in query space instead of image space. Secondly, we propose an adaptive text augmentation scheme for target labels designed to improve zero-shot accuracy and diversify retrieved image data. Lastly, we present two simple but effective components to further improve downstream target accuracy. We compare against state-of-the-art name-only transfer, source-free DG and zero-shot (ZS) methods on their respective benchmarks and show consistent improvement in accuracy on 20 diverse datasets. Code is available: https://github.com/Chris210634/mudg

Published as a conference paper at ICLR 2025
MULTIMODAL UNSUPERVISED DOMAIN GENERALIZA- TION BY RETRIEVING ACROSS THE MODALITY GAP
Christopher Liao Boston University cliao258bu. edu
Christian So Boston University cbso@bu. edu
Thcodoros Tsiligkaridis MIT Lincoln Laboratory ttsiliell.ne mit. mit.edu edu
Brian Kulis Boston University bkulis@bus edu
ABSTRACT
Domain generalization (DG) an important problem that involves learning a model which generalizes to unseen test domains by leveraging one or more source domains, under the assumption of shared label spaces. However, most DG methods assume access to abundant source data in the target label space, requirement that proves overly stringent for numerous real-world applications, where acquiring the same label space as the target task is prohibitively expensive. For this setting, we tackle the multimodal version of the unsupervised domain generalization (MUDG) problem, which uses a large task-agnostic unlabeled source dataset during fine- tuning. Our framework relies only on the premise that the source dataset can be accurately and efficiently searched in joint rision-language space. We make three contributions in the MUDG setting. Firstly, we show theoretically that cross cross-modal approximate nearest neighbor search suffers from low recall due the large dis- tance between text queries and the image centroids used for coarse quantization. Accordingly, we propose paired k-means, simple clustering algorithm that im- proves nearest neighbor recall by storing centroids in query space instead of image space. Secondly, we propose an adaptive text augmentation scheme for target labels designed to improve zero-shot accuracy and diversify retrieved image data. Lastly, we present two simple but effective components to further improve down- stream target accuracy. We compare against state-of-the of-thh-art name-only transfer, source-free DG and zero-shot (ZS) methods on their respective benchmarks and show consistent improvement in accuracy on 20 diverse datasets. Code is available: https:///ithuie com www.///rr/cc///0/0 m/ddq
INTRODUCTION
Domain generalization (DG) is widely studied in the computer vision literature because the train and test image data distributions often differ for many applications. However, traditional DG methods assume access labeled task-specific source data, which is expensive for many real-world applications. Consequently, more recent studies have tackled the unsupervised DG (UDG) problem, where source labels are not used during finetuning (Zhang et al. 2022 Narayanan et al. 2022 Harary et al. 2022). Unfortunately, this experimental procedure is fairly restrictive and impractical, since still assumes that the source and target label spaces are identical. To address this shortcoming, we propose studying more realistic multimodal UDG (MUDG) setting, where the source data is both unlabeled and "task-agnossic"" i.e. the source dataset is not specifically designed for the target task. Instead, we only require that the source dataset contains the target visual concepts, and that these target visual concepts aligned with the corresponding target language concepts in the
Domain Generaltaaioon
Unuggrndsed Domain Seneralisatin (UDO)
Domain Scneralization (MUDS)
Multimodal Unsupervised
Traning Domain
Traiiin Domain
Uniabeled Traiting
Training Gemain
Tranning Domais
PN
Labeel
Labels
alephant
Texs Domain
Test Gemann
Text Boman
Comparison of Problem Settings. DG assumes labeled source data for the target task. UDG assumes the same data, but without labels. Our MUDG setting assumes a superset of the unlabeled source data, most of which is irrelevant to the target task. The core challenge of MUDG is constructing a task-specific dataset for model finetuning.
Figure 1:

---

Published as a conference paper at ICLR 2025
Task aynossic Taskospecilic Unlabeled
Target
Setting
Source
Source
Target
Label Names
DA
Sun al. 2016
Saito al.
2019
Acuna et all
2021
laheled
SFDA
ZS Menon and Vondrick
2022
Prait
2023
Roth
2023
Novack et 2023
DG
Cha etall 2022
Min
2012 Sha el
2124
Khallak
2023
laheled
UDG
Zhang
2022
Narayanan
Harary
unlabeled
SFDG (Clo et al N25
MUDG (Ulandardd
(our work)
Table 1: Comparison to related works based on the information available at training time. DA
domain adaptation; DG domain generalization; source free; ZS zero-shot. UDG unsupervised DG. MUDG multimodal UDG is our setting.
CLIP embedding space used for indexing. Under these relaxed assumptions, we can leverage publicly available large-scale image datasets to improve DG accuracy by building a subset of images relevant to the target task. Figure compares our proposed MUDG with the most relevant problem settings.
Multimodal Unsupervised Domain Generalization (MUDG) In order to leverage publicly avail- able unlabeled image data, such as LAION (Schuhmann et al., (022). YFCC100M Thomee et al. (2016). WIT Srinivasan et al. (202T). ) and CC12M Changpinyo et al. (202T), we study MUDG, a generalization of UDG classification. "Multimodal" refers to the requirement for the source dataset to be accurately and efficiently searchable in a joint vision- lasion-languag space using a pretrained CLIP model. Using this searchable index, our goal is build pseudo-labeled subset of the source data to train a model for a given target classification task. Table positions our problem setting relative to related works. Our work can be viewed as an extension of the recent source-free domain generalization (SFDG) problem (Cho et al., (2023), which only uses the target label names during finetuning. Compared to SFDG, finding suitable subset of the source data poses an interesting challenge, and our results show that the margin for accuracy improvement is much larger under our MUDG setting. MUDG is most similar the "name-only transfer" problem posed by SuS-X
Udandarao al. 2023), but our work is more in line with the DG literature.
To illustrate the practical benefits of MUDG over UDG, consider set of e-commerce classification tasks (e.g. "casual" vs. "formal", "modem"" vs. "traditional"". In the traditional UDG setting, dataset must be curated for each category in each classification task. In contrast, MUDG allows for the use of one universal unlabeled dataset of general product images. We accomplish this by selecting a relevant and representative subset of the general dataset using the target class names, and then training customized model for each task. This approach is more practical and scalable, since it eliminates the need for handcrafted task-specific datasets.
Cross-modal Retrieval with Paired k-means
Accurate and Efficient Retrieval The core challenge of MUDG is accurate approximate nearest neighbor search for finding images rel- evant to the target task. Similar to retrieval aug- mentation (Blattmann et al. 2022] Gur al.
Image Festure culorry
Querry Spore
2021 Long et al.) [022; Iscen et al. (023). we propose constructing a subset of images re- trieved from the source dataset using the text query "a photo of a <label)". Existing works off-the-shel inverted feature list (IVF), which organizes images into buckets based on the closest feature centroid, as shown in Figure During deployment, the index calculates simi- larity scores between the query and images only in the bucket corresponding to the closest cen- troid. This simple search algorithm works well when the probability of the query and its nearest neighbor residing in the same Voronoi cell is high. However, we show theoretically that this probability is low when the query belongs to a different modality due to the well-known modal-
Airtenieteo Carrr
Momrott
bocood Inloor of
X fecorrott rosiical
Meddality
Modalty Cop
Poglo coss mns soooch centride
centro heyssacce
Figure 2: Illustration of Paired k-means. this ex- ample, image samples are grouped using k-means, and the search is limited to the group whose cen- troid is closest to the query. Left: Due to the large distance between the query and image sam- ples, the true nearest neighbor is unlikely to reside within the Voronoi cell of the closest image cen- troid. Right: We correct this issue by maintaining the centroids of the Voronoi cells in query space.
ity gap (Liang et al. 2022 Oh et al. 20224) Shi
et al. 2023 Ming and Li 2024
We empirically
2

---

Published as a conference paper at ICLR 2025
confirm that cross-modal approximate nearest neighbor search using IVF index has lower recall than in-modal search. In other words, a query may return images that belong to a different label, leading to low downstream target accuracy. To mitigate this issue, we propose paired k-means, clustering algorithm that maximizes the probability of a text query and its closest image sample belonging to the same Voronoi cell by updating the centroids to be in the query distribution. We show empirically that paired k-means converges and leads to better cross- cross-modal recall under the same latency constraint.
Diversified Retrieval On the other hand, ac- curate retrieval is not sufficient for high target accuracy, since a training dataset that covers only the small, high-confidence region of the tar- get image distribution is undesirable. To intro- duce diversity, we must augment the text query, e.g. "a photo of a chicken, (descriptor). "* Pratt et al. (2023) and Menon and Vondrick 2022) use LLM-generated descriptors to augment the query, but al. (2023) suggest that these LLM descriptors achieve the same zero-shot accuracy as random text augmentations. Intu- itively, querying with descriptors that already achieve high zero-shot accuracy should lead to better target performance after finetuning. Fol- lowing this intuition, we design an unsuper- vised heuristic to select good label augmenta- tions adaptively based on the target classification task, without an LLM or image data. Our heuristic favors augmentations that do not reduce the variance between target text features (see Figure 3 We show that our adaptive descriptor selection achieves state-of-the-ant zero-shot accuracy across 10 standard datasets, and that this translates additional gains in downstream target accuracy.
Unsupervised Objective for Adoptive Label Augmentation
Saarsae cle wootocttoonooo probler
Cormple poodesgmence
Roce clastiiaatio probieon
Peatere proserved
wish beeter
AN
Figure 3: Adaptive Label Augmentation. Starting with a base classification problem, our goal is to find label augmentations that preserve the variance of the classifier prototypes. Intuitively, maintaining feature variance enhances zero-shot accuracy by better capturing the underlying data distribution.
Finally, we introduce two additional components that makes our method more robust to irregularities in the source data and the target task: (1) Sample selection by clustering: cluster image embeddings into & clusters within each label group and randomly select one sample from each cluster. The purpose of this step is twofold: to build balanced dataset, with samples per label; and to ensure that no two images are semantically similar. (2) Diversity preserving loss: regularize the KI divergence between current and initial soft predictions on training samples for every augmentation to avoid collapse of textual representations.
Overall, we make two core contributions related to retrieval in the context of MUDG:
We identify a fundamental limitation of cross-modal approximate nearest neighbor search caused by the modality gap. We investigate this challenge theoretically, and propose a paired k-means clustering algorithm for building an index with better cross-modal recall.
*
We develop an unsupervised adaptive label augmentation scheme for diversified retrieval.
2 RELATED WORK
Multimodal foundational models Multimodal foundational models (Radford et al., 2021 Jia et al., 2021 Li al. 2022b Yu al. (022) use separate image and language encoders embed the two modalities into a joint space. Once pretrained, these embeddings can be used to create database searchable by both image and text (Schuhmann al. (022). Large-scale efficient search enabled by approximate nearest neighbor search libraries such as FAISS (Douze et al., 2024). A recent work (Gadre al., 2023) achieved the latest state-of-the-art on ZS ImageNet by cleaning LAION-5I with a teacher CLIP model. Another recent work (Sun et al., (0233 uses large CLIP model and unpaired web-crawled data to train a smaller foundational model
distillation-inspree manner. The above works focus on generalist pretraining from scratch, which remains out-of-reach of most academic researchers. We focus instead on task-specific finetuning using a constructed dataset of up to 100K samples. React (Liu et al.) 2023) tackles the so-called "model customization" problem; in comparison, our work is more focused on the source subset construction portion of the finetuning pipeline, and consequently, retrieved dataset. Our problem setting is most similar to SuS-X (Udandarao et al., (202)). which retrieves support set from LAION-5B, but they focus the training-free regime. Many recent
achieve similar accuracy improvements React with 100x smaller
3

---

Published as a conference paper at ICLR 2025
ean image ambodd rgs from lorge source datser, and
Diversifeed Resieeaa Aupinnn tndt Calclitee emaediings reighton.
Ronk Pacudoo labeling Poouds lase rtteoeelioogsstf figher
Gusterin: Outter Image embeessings Then, randem select
Trankeg udlctee
AP
AugreetFande
A
LE
Aagmari Eagreniiiinaaa.......
Resreserationall of jurt image text crobediinn space
Ingrevl
W test eriboidi imaggenntedlinng
Figure 4: Illustration of Our Training Pipeline. In step 1, we select good label augmentations according to Fig. and retrieve their nearest neighbors using the procedure shown in Fig. [ In step 2, we pseudo-label the retrieved images with the closest label feature. In step 3, we de-duplicate the retrieved dataset by clustering and then selecting one image from each cluster. Finally, we finetune the CLIP classifier on the dataset.
works strive to understand and tackle the modality gap (Liang et al. 2022 Oh et al. 2024, Shi et al.) 2023 Ming and Li 2024) in the context of model transfer; unlike these works, we study the modality gap's implications on cross-modal search. |Mig and Li (2024): [scen et al. (2023): Liu et al. (2023) work around the cross- cross-modal retrieval problem by additionally performing in-modal search, which is not possible for every application; we work towards more effective cross-modal search.
Flavors of domain generalization Table Tis is a non-exhaustive summary of variations generaliza- tion settings studied in recent literature. Domain adaptation (Sun et al.) 2016 [Saioo et al. (0119] Acuna et al. 202T) aims to leverage out-of-distribution (OOD) but task sask-spe source data in conjunction
with unlabeled target data. Traditional DG (Muandet et al., 2013 Cha el al. 2022 Min et al. 2022)
trains on OOD task-specific source data from multiple domains, without knowledge of target data. A more recent flavor of DG (Shu et al. 2023 Khattak et al. (203) trains on generic labeled source data (e.g. ImageNet) with the goal of generalizing to any classification task, by leveraging transferability of the image-text alignment in CLIP. Unsupervised DG (Zhang et al. [2022] Narayanan et al.) 2022
Harary et al.] 2022) trains on unlabeled task- specific source data. Source-free DG (SHDG) (Cho et al. 2023) aims to increase pretrained accuracy with only the target task information, but the improvement over ZS methods is not consistent empirically. ZS methods (Menon and Vondrick, 2022 Pratt et al., 2023 Roth et al. 2023 Novack et al. 2023) improve accuracy by ensembling multiple text features. Our problem setting, multimodal UDG, takes advantage of plentiful unlabeled Non-task-specinfe image data, which offers more leverage than the SFDG setting, while not relying on any task-specific or labeled images contrary to the DA and DG studies.
Webly supervised, open world, and open set The webly supervised literature (Chen and Gupta
2015 Li et al. 2020) focuses on learning from a noisy web-crawled dataset (Li et al. 201 Sun 2021 and is very closely related to the large body of work on noisy supervised learning, see survey (Song et al. (2022). These works focus on the finetuning algorithm given a dataset, rather than the construction of the training data, unlike our work. Another popular research direction focuses on generalization to unseen classes given certain set of (possibly related) training classes; these works
al.
fall under open-set (Saito et al. 2021;
: Du et al. 2023 Panareda Busto and Gall, 2017) open-wort
(Bendale and Boult 2015 Boult et al. 2019 Cao et al.) (022) or base-to-novel (Zhou et al.) [022]
Khattak et al.] [2233 Kan el al. 2023) semi-supervised learning. Finally, some works selectively
retrieve from unlabeled data pool to expand smaller set of labeled training samples (Sener and
Savarese [017] Killamsetty et al. 2021 Kim et al.; (2023), referred to as core-set sampling. Contrary to these works, we assume no labeled data of any kind at training time. Retrieval augmentation
(Blattmann et al. 2022 Gur et al. 2021 Long et al. [2022] [scen et al. (002)) is a related line of work
which requires retrieval system test time, adding substantially heavier evaluation overhead.
METHOD
Figure illustrates our method. Concisely, we use augmented copies of the target label names to query a large source dataset and then finetune the student CLIP model on retrieved images, with the ultimate objective of high target accuracy. Toward achieving this objective, sub- goals: building a search index with good cross rrosll recall and designing a label augmentation scheme with high 7S accuracy. Section 3.1 tackles the first sub-goal with novel cross-modal
identify two necessary
4

---

Published as a conference paper at ICLR 2025
197
1.0
10
query tery
image
0.0
query ge
Coop
1.0
6
Rapl with
0.4
toxt query -oo galeery
image query - image gallery
4
text query mage gallery
09 0.04
-02
0.0
0.2 0.4
0.5
0.5
10
32
66
128
0.4
0.6
Santy
intty
Figure 5: Left: Empirical confirmation of the modality gap; cross moosl similarity scores are lower than in-modal similarity scores. Middle: Cross-modal nearest neighbor search suffers from lower recall than in-modal search. Right: Empirical verification of Theorem I; queries that are farther away from the closest centroid have lower recall.
indexing scheme for accurate and efficient retrieval; Section 3.2 solves the second sub-goal with an unsupervised heuristic to select good descriptors for augmenting label names; Section 3.3 finishes with diversity preserving loss for model finetuning.
3.1 MORE ACCURATE CROSS
RETRIEVAL.
Background For this paper we will consider two-level IVF indexing scheme used by Udandarao
et al. (023); Liu et al. 2023); : Iscen et al. 2023). The first level is a coarse quantization consisting
of k buckets obtained by k- k-means clustering, the index stores the coordinates of the centroids and list of sample IDs belonging to each bucket along with their residual features. The second level is a fine quantization scheme used to reduce disk storage. We will consider only the coarse quantization scheme. During deployment, the index sorts the centroids by decreasing similarity with the query and searches through the first riprobe buckets for its nearest neighbor. Assuming that each bucket contains a similar number of samples, the query speed is proportional to Pprobbe. The quality of the index can be measured by the percentage of queries where the true nearest neighbor among all gallery samples is retrieved, "R @1) 1", for constant Neprobe-
Motivational Issue Figure 5 Middle shows that the R@] I for text-to-image searches is about 30% lower than in-modal queries for small Mprrobe This is a concern for many multimodal applications, since cross moosal retrieval exhibits far worse recall-latency tradeoff than in-modal retrieval using existing technology. We hypothesize that this drop in recall is caused by the modality gap (Liang al. 2022] Oh et al. 20024 Shi et al.) 2023 MMig and Li 2024), illustrated in Figure[5]Lefe Text queries tend to be far away from the image centroids. Moreover, on a closed space, points far away from centroids are closer to the boundaries of the Voronoi cells, and the neighbors of boundary points are more likely to reside in neighboring cells.
) drawn uniformly from the unit sphere Sá : fx e Rd |||||2 = 1). Consider k additional points(c,, ...; C drawn uniformly from St We refer to these points as "centroids"; k << n. The Voronoi cell around centroid is the set of all points closer to that centroid than all other centroids, i.e. Vor(c) == {x e S ||x - clla S ||x cilla Ve, ~ (ci...., cx..,.,,)) (c). We assume that Vor(c) is a strict subset of the hemisphere centered at c. Theorem 1 (Decreasing recall on closed space) Under Assumptions 1, Vc E [cI,,..,ckk c.pp p Vor(c):
Assumptions 1 Consider n points (xl,...,
Ie (p) Pr
arg min (x).xx........
||x;-D0 - E Vor(c) S Dc (p) +
where e == 1 p(s'); p(cos(0)) L ( ( Lo(() is the regularized incomplete beta function; and ge (p) is a function defined over Vor(c) which satisfies the following properties:
Io(.))
1. (c(c) = p(s'), where, s' :- cos ( cos maxx,e(e....... (e c:)).
2. De (b) + e = 2 for all points b on the boundary of set Vor(c).
3. ge is non-increasing in all directions from e in the following sense:
Be (projsa (au + c)) < ge (prois=(In + c)), Va > b > U
given that all inputs to function Je remain within Vor(c). projs denotes L.2normalization.
The proof follows from the convexity of Voronoi regions, Appendix A.I. Note that p(cos(0)) denotes the surface area of a spherical cap with angle 0 as a fraction of the unit sphere's 's surface area
[Li 2010), and p(s') is close to 1. In plain words, Theorem states that under Assumptions 1, the
5

---

Published as a conference paper at ICLR 2025
Theorem PDF, lilll
Theorem
||p||
Emeans
paired k-means
20
0.4
1.2
10
Emmans
0.2
1.0
paited k-meenss
0.0
0.8
0.6
2
0.6
-2
-2
30 15
10
15
Iterstiion
keration
Figure 6: Left: Plots of the PDF in Eq. 2 of Theorem 2 for d = 2, illustrating both the small variance regime when p in-distribution and the large variance regime when p out-of-distribution. Right: Convergence of the two objectives in Eq. S note that the paired k-means algorithm is better at minimizing Leress Leress-modal, modal, the rate of cross-modal search failures on training data.
bounds on the probability that the nearest neighbor resides in the same Voronoi cell as the query decrease monotonically in all great circle directions from the centroid. This probability is equivalent R@! I with Ppoobe 1. The assumptions are somewhat stringent, but Figure Right empirically verifies this behavior with 20M subset of LAION-2B.
Theorem 1 partially explains the empirical observations in Figure Middle, and text queries are certainly far away from their image centroids. However, Theorem assumes that the query belongs to the same distribution as the gallery set. This is clearly not true for cross-modal retrieval. To understand the drop in recall when the query is not in the support of the gallery distribution, need another set assumptions. Theorem will show that as a query moves away from a Gaussian distributed gallery distribution, the probability that the closest gallery sample and the closest centroid are close decreases. In fact, the distribution of the closest gallery sample approaches a Gaussian distribution in all except one dimension in the limit, i.e. if a query is far away, its position provides little information about the location of the closest gallery sample.
Assumptions Consider r points drawn uniformly from (O Ia). the standard normal distribution in Rdd Denote as (x1, ..., (x,, ). Let (p)) arg (xx ||x; pll2-
Theorem 2 Under Assumptions 2, the probability density function of the closest point to query p is:
Pr(q(p) x] = (1 Pr[x; E (,(p)))" 4/2 exp (- (-2 1-1)) r: ||x pll (2)
where Pr[x; E (p)) indicates the probability that a single point drawn from N(0, Ia) resides in the Rdd ball of radius centered at p. Corollary 2 The probability density function derived in Theorem 2 satisfies the following:
(Small variance when ||a||a small)
Pr[l]q(p) - ppll > r] (1
7
exp - + r)) )))))
(3)
2d/2f
( + 1)
2. (Large variance when ||l||2 large).
lim Pr(q(p) = x] = n (Ø (Iprrojp (x)1l2)) (2T) d/2 exp |0
(- ||x
where (|projp(x) (x) denotes the length of x projected onto p, and & denotes the CDF of the standard normal distribution in ID.
Proofs are in Appendix A.2. Theorem states the probability density of the closest gallery sample q(p) in terms of the location of p, and Corollary 2 interprets the density function by splitting it into small variance and large variance regime. When ||]|| is small, the blue term in Eq dominates and Pr(q(p) = x] looks like Dirac delta, see Fig. Left. In other words, the nearest nei neighbor is likely to be a small region; Eq. states this formally. When ||p|| is large, the red term in Eq. 2 dominates and Pr(q(p) x] looks like Gaussian with the same variance the sample distribution in all directions except for p. see Fig. 6 Middle Left. Clearly, this implies that a query that is far away from the gallery distribution is unlikely to belong the same Voronoi cell as its nearest neighbor. We sketch a geometric argument for this implication using the boundary of Voronoi cells in Appendix A.4 but do not give formal proof.

---

Published as a conference paper at ICLR 2025
Grd Nade Sedeementaie Jereerss
WAPPOLL -ROOCC DisriEas
Yead bestepau
663
1m0
10 Eesonss
Autitige Polneso Coxtre Simdarity
Number Augmenaatonss
Figure 7: Left: Improvement of nearest neighbor recall with paired k-means at various latency settings and fine quantization levels. Middle: Correlation between variance of text features and 7S accuracy. Right:
Intermediary 7S accuracy result with adaptive label augmentation.
Algorithm 1 Paired k-means
Algorithm MUDG
Input: Source dataset X. Al Am...mm Dnnighhors ki, pretrained findex and fandent, {tl, .l.y.. be). Step 1: Let ② = {iindex, (t,)),)) Vi = 1 c,j 1 : m) denote the query set. gor E Q. retrieve Nacighbors closest samples in X'. Combine retrieved images from all queries; denote as X1. Step 2: For q E 2, sort x E X, by decreasing cosine similarity between q and findex, imago (x). Denote rank of x relative Assign each x E X1 the label corresponding to the closest ranked query, i.e. arg min, rank(x. g). Denote the labeled set as X2. Step 3: Initialize
1: Input: Image samples {x,, Xn J. text sam- ples (Pi,... }. number of clusters k.
2: Initialize cluster centroids {Ci, .... )+
3: Calculate the nearest image sample to each text sample, (q(P)), ..., q(pm)). Note that there can be redundancies.
4: for fixed number of iterations do
5: Assign each image sample in centroid.
q(P.)) to the nearest
as rank(x, q) 1.
6: Update each centroid c in {ci, CR to be the mean of text features paired with image features assigned to the cluster:
empty labeled dataset Xg. For each label E 1 find the subset of X₂ with label y. Cluster into yı clusters, using
Σ
[Vor(c)]
P means. Randomly select one sample from each cluster and append to X'3. X3 contains ck1 samples. Step 4: Finetune faudennt on X3 for N iterations, using Liain (Eq. 7 Output: finetuned fausdent
PE Pr. H Ha(p))
7: Normalize centroids.
8: end for
9: Output: cluster centroids {c1, (c,,,.,ccc) Ck)-
Paired k-means The fundamental issue causing the degradation in cross-modal recall is that the image centroids and queries are far away from each other in feature space. Consequently, the nearest centroid to query does not provide much information about the location of the true nearest neighbor. To resolve this issue, we modify the k-means algorithm to update the centroids with the average of text features instead of image features. See Algorithm 1. This algorithm is an attempt at simultancous minimization of the following two objectives heuristically:
Lameuns Σ
Σ
Σ |x - call2. Leross DE(PI....DM.)
l[c(q()) + c(p) (5)
i=1 xEVor(e;)
where {pi. Pn) E Sd denotes a set of n text queries, and c(p) == arg min,, ..... ||: pl denotes the closest centroid to query p. The first objective is the k-means objective. The second objective is the fraction of text queries p whose nearest centroid c(p) different from the closest centroid to the nearest image sample c(q(p)). The first objective enforces good clustering, while the second objective forces query features to be mapped to the same Voronoi cell as the nearest gallery feature. We show that both objectives converge empirically in Fig. 6 Right.
Nearest neighbor search results Figure 7 Left shows that an index trained with paired k-means outperforms the standard k-means index in @ for various values of ngrobe and fine quantization levels. The cross-modal recall directly related to the downstream target accuracy, since subsequent steps in our method rely on retrieving images that are relevant to the target task.
7

---

Published as a conference paper at ICLR 2025
3.2 DIVERSIFIED RETRIEVAL WITH ADAPTIVE TEXT AUGMENTATION
There is very little semantic diversity among the nearest neighbors of any single query, which likely leads to severe overfitting during training, see Figures 16 and 17 in the Appendix. To ensure diversity of finetuning data, we propose to search the source dataset with augmented text queries in the format of "a photo of (label), (descriptor)." Previously, the authors of visual descriptors Menon and Vondrick| (2022) proposed to use GPT to generate phrases that describe the label, e.g. "a photo of a chicken, which has two legs". Subsequently, waffleCLIP Roth et al.) (2023) showed that the visual descriptors achieve similar zero-shot accuracies as random text augmentations on diverse datasets, see Figure Right.
We consider two factors when choosing an appropriate augmentation function: (1) the augmented text does not change the label of the original text; and (2) the resulting distribution of augmented queries covers the entire concept of the class. The first requirement can be measured by the zero-shot accuracy of an ensemble of augmented texts. Let {A11... .... Ay denote a set of M text augmentation functions. We aim to select subset of size m << M that does not change the meaning of the labels. We use the heuristic in Eq. to choose the augmentation subset based on the target labels {ti,.... )- First, we cluster the label text features into l2 clusters using k-means. Denote the label clusters as (St.1....,S..
St,ka) and the text encoder as Feet:
l2
arg min (~
1
(fee > tr.tymer.
Σ
Σ (fie fren fren(t))) ty.tyroee
(6)
Intuitively, an augmentation is desireable it does not reduce the variance of the text features within any label cluster. Eq. 6 measures the variance of label features using their average pairwise cosine similarities, and counts the number of clusters where the augmentation A decreases this variance. For example, on ImageNet, the augmentation "a photo of (label), which can be any size or shape" is a good augmentation because it does not reduce the distance between any two ImageNet labels, and the indicator function in Eq. evaluates to 0 for all label clusters. On the other hand "a photo of a (label), which has sharp teeth" is a bad augmentation for ImageNet because it reduces the distance among text features corresponding
animal labels. This reduction degrades the model's ability to discriminate among the labels within the cluster, and the ZS accuracy decreases as a consequence, see Figure 7 Middle. Table iiin in the Appendix provides qualitative examples of augmentations with varying loss values.
We select the m augmentations {AI...., Am } with the lowest loss according to Eq. 6 and construct a dataset with me queries:
m]. fexd denotes the text encoder used for indexing the source dataset Ng. We retrieve the negiihhees nearest neighbors each query in X₂ and remove redundancies, resulting in preliminary dataset size of at most MCNnnigghorss See step of Algorithm 2.
(Aa, Vi = 1ii=:: = j = 1:
3.3 ADDITIONAL TRICKS FOR SAMPLE SELECTION AND FINETUNING
We label each retrieved image sample according to the text feature to which it is ranked the highest, see step of Algorithm 2 and Appendix C.2 for a justification. We then select kn images for each label according to step 3 of Algorithm 2; the detailed procedure is presented in Appendix C.3. Finally, we finetune using the diversity preserving loss presented in Liao et al. (2023):
Cirain
Σ
CE (ÚA, (1 A)y + AÿA.0)
m
Am(A)....Am.
where CE denotes the cross entropy loss, ÚA E A. denotes the soft prediction of the model with augmentation A. y denotes the one-hot encoded labudo-label, and Ú.A.0 denotes the soft prediction of the initial model with augmentation A. À is hyperparameter. Lmain learns the pseudo-labels while simultaneously preserving the diversity present in the initial text encoder.
4 EXPERIMENTS
We experiment with the VIT B/16 and VIT L/14 pretrained weights released by Radford et al. (2021) and available through the Python openclip package (Ilharco al. (022I). The indexing model is ViT L/14; we modify FAISS (Douze et al. 2024) to build a search index for the source dataset, LAION-2B-en (Schuhmann al. 2022). We experiment with two model sizes show that we achieve large gains in target accuracies even when the indexing model and the student model are identical.
8

---

Published as a conference paper at ICLR 2025
Setting ImugeNet Caltech Pets Cars Flowers Food Airerali SUN DTD EuroSAT UCF Mean
Open-A1 CLIP VIT-B/16
CLIP ZS [Radlord et al.
2021
ZS
67.1
93.3
89.0 65.4
71.0
85.7
24.9
63.2
43.6
46.6
67.4
65.2
waffleCI IP [Rolh
75
68.2
93.5
88.1 65.5
72.1
85.9
25.6
66.2
44.3
47.3
68.1
65.9
Random Desc. (Koth al
2023
78
68.
94.3
87.7 65.7
71.7
85.7
25.2
66.2
44.7
47.7
67.3
65.8
Ensemble (Radford
2221
28
68.4
88.8
71.1
86.0
24.9
66.0
43,9
65.6
Vis. Desc:
Menon and
Vondrsc)
2022
ZS
68.6
93.7
89.0 65.1
72.1
85.7
23.9
67.4
43.9
46.4
66.8
65.7
CuPL Pratt 20A6)
69.1
91.7 65.0
73.5
86.0
27.7
68.5
48,9
70.2
SuS-X
2023
MUDG
70.0
93.9
91.6 65.9
73.1
86.1
30,5
67.9
55.3
58.1
66.7
69.0
Nearest beighbbrr
MUDG
69.4
93.9
93.4 70.2
75.8
86.3
27.2
67.4
52.4
41.2
69.9
67.9
Margin Colemmnn et
2019)
MUDG
64.5
92.8 61.6
75.5
86.3
33.2
61.6
51.0
59.6
71.9
68.4
Least Conndcnt (Coleman of
2019
MUDG
62.7
93.9
92.2
68.6
75.3
86.2
33.6
63.3
52.3
58.3
71.2
68.9
Entropy Coleman 0
MUDG
86.4
52.2
Herding Welling 2075
MUDG
69.0
95.1
93.0
74.2
75.9
86.4
32.8
68.6
53.4
59.0
72.0
70.9
K-Ctr Grecuy (Seact and Savarese
2017
MUDG
67.5
94.7
93.3
72.5
75.6
86.4
33.3
68.5
53.6
58.7
71.9
70.5
MUDG (ours)
MUDG
70.4
94.6
92.9
73.8
76.5
86.7
32.8
68.8
53.3
61.3
71.0
71.1
Upper bound
73.9
95.8
95.1
89.9
95.9
87.5
59.2
77.3
73.2
89.0
86.4
83,9
Open-Al CLIP VIT-L/14
CLIP ZS (Raiford et
202
75
73.8
94.6
93.6
76.9
79.4
90.9
32.8
68.0
52.7
56.2
74.7
72.1
waffleCI IP (Roth ot al
2025
78
75.0
96.
93.5
77.1
78.8
90.9
33.6
69.3
54.3
57.7
75.3
72.9
Random Desc.
2023
ZS
93.4
78.5
90.7
33.6
54.5
59.3
75.5
Ensemble (Radford et 2022
ZS
75.6
95.6
94.0 78.1
79.8
91.2
32.7
70.5
54.0
55.2
75.0
72.9
Vis. Desc. Menoa and Vondrick
2022
ZS
75.3
96.7
93.8
77.4
79.3
90.9
34.8
71.0
56.4
62.8
73.9
73.8
CuPL otal, (222)
ZS
76.3
94.2
76.3
79.5
91.1
36.0
72.4
60.0
75.8
Neurest neighbbrr
MUDG
76.2
95.8
95.3
78.0
80.2
91.3
33.3
71.7
56.2
61.6
75.6
74.1
MUDG (ours)
MUDG
76.4
94.9
79.2
79.4
91.3
35.5
72.5
58.2
70.9
76.8
75.6
Table 2: Comparison of our MUDG baseline with 7S baselines and SuS-X on 11 diverse datasets.
Average of three experiments. For MUDG rows, dataset construction and model training is separate for each dataset. "Nearest neighbors" refers simple nearest neighbors retrieval. indicates results reported by the authors; all other results are our reproductions. We finetune the ViT-B/16 model on 16-shot target training data as upper bound.
ImageNet
Office Home
DomainNet
Setting
Sketch
Open-AI CLIP VTT-B/16
CLIP 7S Radford et al.
202
ZS
60.9
46.6
47,2
74.1
57.2
82.6
67.2
88.8
89.6
82.1
$7.6
waffteLLP Koth
20341
ZS
48.5
89.7
90.4
82.9
59.7
Random Desc Koti ot 2023
ZS
61.7
48.8
49.9
76.6
59.2
83.0
69.1
89.5
90.2
83.0
59.6
Ensemble Radford el al
2021
ZS
61.9
48.5
49.2
77.9
59.4
84.3
67.7
89.3
90.2
82.9
60,2
Vis. Desc: Menon Voniriic
2022
ZS
61.8
48.1
48.6
75.2
58.4
PromptStyker Cho et
0175
SHDU
83.8
68.2
91.6
90.7
83.6
59.4
MUDG (ours)
MUDG
50.4
85.7
Open-AI CLIP VIT-L/14
CLIP ZSS Rudford et al.
2021
ZS
68.0
57.9
68.3
85.5
69.9
87.1
74.8
1
93.4
87.1
63.9
wafleCI. Kaah
58.7
78.2
93.8
94.4
88.5
65.4
Random Desc Koth et
7123
ZS
69.2
59.1
70.5
87.1
71.5
88.2
78.4
94.4
94,0
88.7
65.7
Ensemble Radford et al.
2022
ZS
69.9
59.7
70,2
87.8
71.9
88.5
76.9
93.8
94.5
88.4
66.1
Vis. Desc: Menon Vontrrik
2022
ZS
69.4
58.8
69.6
86.4
71.1
PromptStylkr Cho ot
2125
SFDG
89.1
77.6
94.8
94.8
89.1
65.5
MUDG (ours)
MUDG
94.6
Table 3: Comparison of our MUDG baseline with ZS baselines and PromptStyler on DG benchmarks. Average of three trials. Dataset construction and model training is performed once and evaluated on all domains for Office Home, Terra Incognita, and DomainNet; but perform the steps separately for each ImageNet domain, due to differences in label spaces. PromptStyler (Cho et al. 2023) results are those reported by the authors; all other results are our reproductions.
Datasets
We experiment with diverse set of target classification tasks, ImageNet-IK IK Russakovssy
et al.
2015). 1, Caltech-101 (Li et al. 2022a). Oxford-Pet. (Parkhi et al., 2012), Stanford Caanord-CCrs (Krause
et al. 2013). Flowers- 102 (Nilsback and Zisserman] 2008), F01 (Bossard et al.) 2014). FGVC-
Aircraft (Maji et al., 2013). SUN-397 (Xiao et al. 2010), Descrrbbable-exxturs (DTD) (Cimpoi et al. 2013), EuroSAT (Helber et al. 2019) 10F (an action recognition dataset) (Soomro al. 2012)
in Table and ImageNet- V2 (Recht et al. 2019). ImageNet-Skett
(Wang et al. 2019), ImageNet
(natural adversarial examples) (Hendrycks et al. 2022b). and ImageNet-R (Hendrycks et al. 202
in Table are commonly used by zero-shot papers, while Office Home (Venkateswara et al. 2017).
Terra Incognita (Beery et al. 2018), DomainNet (Peng et al. 2019) VL.CS TToralba and Efros

---

Published as a conference paper at ICLR 2025
Paxred kreeare Iosie
CANXAAYS Oegterkat Metheds
Consascioan Loks Fonctore
Seactiee
Comrechicr
48.
LL
Figure 8: Summary of ablation experiments. See Appendix Tables
5 aand for detailed tables.
201T) and PACS (Li et al. results are in Appendix B.
2017a) are common DG and DA datasets. TerraInc, VLCS and PACS
Baselines We compare to ZS (Roth et al. 2023 Radford et al. 2022 Menon and Vondrick 2022
Pratt et al.]
2023), SFDG (Cho et al. (023). ). and MUDG (Udandarao et al. (2023) baselines. We also
include strong random descriptor baseline which ensembles randomly selected visual descriptors
(Roth et al.
2.23). To the best of our knowledge, PromptStyler (Cho et al. (023) is the only current SFDG baseline and SuS-X (Udandarao et al., (023) is the most suitable MUDG baseline. In addition,
we compare against representative coreset se selection baselines (Coleman et al.
2019 [Welling] 2009
Sener and Savarese (017). using the implementations by Guo et al. (2022) We do not compare against supervised DG baselines, such as ERM and MIRO (Cha et al. 2022). since those methods require labeled data for the target task. Ablations We provide ablation studies justifying our paired k-means indexing, adaptive label augmentation, diversity preserving loss, and sample selection schemes in Tables [ 5 and 6 in the Appendix and summarized in Fig. g Hyperparameters are listed in Tables 9 and of the Appendix. An ablation study m, Rneighhons ki and Mprobe is included in Figure 15 of the Appendix. Qualitative results We provide qualitative examples of successful and unsuccessful retrievals in Figures 9 and I0
Sampla
Chease Triniig
Sampls pusseline
Query
Sumple
Chono inseeng
Orans Traving Sample
Figure 9: Qualitative comparison of our retrieval results against the baseline method (Udandarao) al. 2023). Our method retrieves images which are more aligned with the target concepts. Dataset names in order: ImageNet, SUN, Cars, DTD, Flowers, Food, Pets, UCF.
Queey
Sereçkoa
Thtrinng
Sumplee (targ
Figure 10: Example retrieval failure modes. Target concepts of "clean room" and "paddle" are different from the CLIP alignment; the CLIP embedding space not aligned for the action "playing Daf", the texture "lacelike" and the flower "silverbush".
5 CONCLUSION
This work tackled the multimodal unsupervised domain generalization problem, which finetunes a model for a target task using images retrieved from a non-task -ppcific,,,,, unlabeled source dataset. We broke the MUDG problem down into three smaller sub-problems and proposed novel solutions for each sub-problem. First, we introduced a paired k-means clustering approach to build an index with superior cross moddal recall. Second, we designed an unsupervised heuristic to select good label augmentations for diversified retrieval. Finally, we trained the student CLIP model on the retrieved data with a diversity preserving loss to yield promising accuracy improvements across 20 diverse benchmarks.
10

---

Published as a conference paper at ICLR 2025
ACKNOW
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001 Any opinions, findings, conclusions recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
REFERENCES
David Acuna, Guojun Zhang, Mare T Law, and Sanja Fidler. f-domain adversarial learning: Theory and algorithms. In International Conference on Machine Learning, pages 66-75. PMLR, 2021.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pages 4556477,20 2018.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1893-1902, 2015.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval- augmented diffusion models. Advances Neural Information Processing Systems, 35:15309- 15324, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Terrance E Boult, Steve Cruz, Akshay Raj Dhamija, Manuel Gunther, James Henrydoss. and Wal- Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9801-9807, 2019.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In International Conference on Learning Representations (ICLR), 2022.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In European Conference on Computer Vision, pages 440-457. Springer, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceed- ings of the IEEE international conference computer vision, pages 1431-1439, 2015.
Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler:
Prompt-driven style generation for source-free domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15702-15712, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. CoRR, abs/1311.3618, 2013. URL
1311.3618
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, arXiv preprint arXiv: 1129,, 2019.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng. Jeff Johnson, Gergely Szilvasy, Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024.
Pan Du, Suyun Zhao, Zisen Sheng, Cuiping Li, and Hong Chen. Semi-supervised learning via weight-aware distillation under class distribution mismatch. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16410-16420, 2023.

---

Published as a conference paper at ICLR 2025
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXhv:230..14108, 2023.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference Database and Expert Systems Applications, pages 181-195. Springer, 2022.
Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Reiter. Cross- modal retrieval augmentation for multi-modal classification. Findings of the Association for Computational Linguistics: EMNLP 2021, pages 111-123, 2021.
Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, et al. Unsupervised domain generalization by learning a bridge across domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5280-5290, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2277-2226, 2019.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349,, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021b.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL http::ttt////att ott.:////.o.........000 52817 If you use this software, please cite it as below.
zer.odo. 5143773
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhanced contrastive vision-text models. arXiv preprint arXiv:2306.07/99 270.01796, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR, 2021.
Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng, Knowledge- aware prompt tuning for generalizable vision-languan models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15670-15680, 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:1 8661-18677 2020.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in Neural Information Processing Systems, 44:14888-14501, 2021.
Sungnyun Kim, Sangmin Bae, and Se- Young Yun. Coreset sampling from open-set for fine- grained sulf-supe learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7537-7547, 2023.
12

---

Published as a conference paper at ICLR 2025
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine tie-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pages 5542 2017a.
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022a.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009. 07995, 2020.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888- 12900. PMLR, 2022b.
Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics & Statistics, 4(1):66-70, 2010.
Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: /00::::700.00000 02862, 2017b.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
Christopher Liao, Theodoros Tsiligkaridis, and Brian Kulis. Descriptor and word soups: Overcoming the parameter efficiency accuracy tradcoff for out-of-distribution few-shot learning. arXiv preprint arXiv:2311.336/ 2023.
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with serieval-augmentt knowledge. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Reco pages 1118-15158, 2023,
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In Proceedingy of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 6959-6969, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/1306.5151, 2013. URL httphttt///a/ax.orrg///////
3106
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07/11 2022.
Seonwoo Min, Nokyung Park, Siwon Kim, Seunghyun Park, and Jinkyu Kim. Grounding visual representations with texts for domain generalization. European Conference on Computer Vision, pages 37-53. Springer, 2022.
Yifei Ming and Yixuan Li. Understanding aetreeeal-uummented task adaptation for vision-language models. arXiv preprint wrXi::2405.01468. 2024.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning, pages 10-18, PMLR, 2013.
Vaasudev Narayanan, Aniket Anand Deshmukh, Urun Dogan, and Vineeth N Balasubramanian. On challenges in unsupervised domain generalization. In NeurlPS 202/ Workshop on Pre-registration in Machine Learning, pages 42-58. PMLR, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing, doi: 10. 10.10CC9PPP............
pages 722-729, 2008.
13

---

Published as a conference paper at ICLR 2025
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning, pages 10342-26362. PMLR, 2023.
Changdae Oh, Junhyuk So, Hoyoon Byun, YongTaek Lim, Minchul Shin, Jong-June Jeon, and Kyungwoo Song. Geodesic multi-modal mixup for robust fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
Pau Panareda Busto and Juergen Gall. Open domain adaptation. In Proceedings of the IEEE international conference on computer vision, pages 754-763, 2017.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs, In 2012 IEEE conference on computer vision and pattern recognition, pages
IEEE, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1415- 2019.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15701, 20233
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovic. Hubs in space: Popular nearest neighbors in high-dimensional data, Journal of Machine Learning Research, 11(sep))2487-2551, 2010.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? CoRR, abs/1902. 10811, 2019. URL htttp://arxiv.org/// 1902. 10811
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. arXiv preprint w:XXv:2306.07282, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-2522 2015. doi: 1007/s1 1263-015-0816-y.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEF/CVF international conference on computer vision, pages 8050-8058, 2019.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. arXiv preprint 1rXiv:2105.14144 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2523 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: I 708.00489, 2017.
Peiyang Shi, Michael Welle, Mårten Björkman, and Danica Kragic. Towards understanding the modality gap in clip. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributiinn,, 2023.
14

---

Published as a conference paper at ICLR 2025
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and lae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. URL (RLLttt://axxx
org/abs 0402
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: tikipedia-basee image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 221, page 2443-2449, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 978 450380377. doi: 10.1145730048333446223 [RLhtts:://doo org/ 10. 01145/344448300 3463257
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Ximeng Sun, Pengchuan Zhang, Peizhao Zhang, Hardik Shah, Kate Saenko, and Xide Xia. Dime-fm: Distilling multimodal and efficient foundation models. arXiv preprint arXiv:2303. 18232, 2023.
Zeren Sun, Yazhou Yao, Xiu-Shen Wei, Yongshun Zhang, Fumin Shen, Jianxin Wu, Jian Zhang, and Heng Tao Shen. Webly supervised fine-grained recognition: Benchmark datasets and an approach. In Proceedings of the IEEE/CVF international conference computer vision, pages 10602-10611, 2021.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data multimedia research. Communications of the ACM, 5432)364-73, 2016. URL http::/cacm. acm. wtt..orrr/mmagziis////111////
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training- frree name-only transfer of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3725-2736, 2023.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Haohan Wang, songwei Ge, Eric Xing, and Zachary C. Lipton. Learning robust global representations by penalizing local predictive power. CoRR, 1bs/1995.13549, 2019. URL
http:://axxiv http:///axiv...o/abbssso.. org http://b///19000
Max Welling, Herding dynamical weights to Iearn. In Proceedings of the 26th annual international conference on machine learning, pages 1 1121-1128, 2009.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3492, 2010. doi: 10. 2010.........00 5539970.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7097-7107, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners image-text foundation models, 2022.
15

---

Published as a conference paper at ICLR 2025
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4910-4920 2022.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, 13009))3377-2348 jul 2022. doi: 10.10077/112233-22--01633 FPs11233-022016553-1
URL ltttps://bbl.or// 10078
16

---

only need to Published as a conference paper at ICLR 2025
LIMITATIONS
Even though we do not explicitly assume any relationships between the source and target data, our work may not be applicable to problems where the target visual concepts are either not present in the source dataset or completely misaligned with corresponding language concepts, see Figure IO Possible examples include synthetic aperture radar images, images of tissue samples, or medical scans. Additionally, our method may not improve results on datasets where the zero-shot accuracy is already saturated. For example, the VLCS dataset contains 5 classes: bird, car, chair, dog, and person. Our method does not achieve any meaningful improvement over the 7S baseline on these simple classification tasks.
A PROOFS
A.1 PROOF OF THEOREM 1
Step I. For case of notation, use to denote closest centroid to p, and use q to denote the closest point to p:
q
arg min
||x;
P/2
(8)
(xI.xx......))
First, we need to solve for the CDF of the probability distribution over the cosine similarity between p and q.
Prip, q) 2 s] - Pr[[p, q) < s]
_1 Pr[(p. x,) < s]
TT Pr[ xx) s))
For ease of analysis, assumed that Vor(c) is strict subset of the hemisphere centered at c, we only need to consider s < O. This corresponds to 0 < </2 where 0 denotes the angle between p and q. Since the x;S independently uniformly distributed over Std Pr[(p. xx) 2 in Eq. 9
corresponds to the ratio of the surface area of spherical cap with angle 0 = cos (s) to the entire
surface area of the sphere. This ratio given in Li (2010) Li (010)):
1 2 sin²
( ( = p(cos(0))
1
Pr[(p, x,) cos 0)
(10)
where I [0, 1) is the regularized incomplete beta function. We will use p [o, 0.5) denote the face area of spherical cap as a function of the cosine similarity as a fraction of the surface area of Sd
Given C, Pick s' to be the closest point on the boundary to the Voronoi cell to c, i.c. co- sine of half the angle the closest centroid:
( cos
cos
max C;E(()....4))
(c, (c,c.)
(11)
Note that s' is chosen such that the the spherical cap with 0 (s') is the largest possible spherical cap centered at that is still fully contained within Vor(c).
The probability in Eq. can then be decomposed as:
Pr[q E Vor(c)] = Pr[(p, d) 2 s') Pr|q E Vor(c) (p, q) 2 s/
(p(p)
(12)
Pr((p. q) < < Pr(q E Vor(c) (p.g) < s'
In the above equation, we hope that n is large enough and k is small enough such that the second term small, and the theorem is only meaningful in this regime. Intuitively, a large n leads to a exponentially diminishing probability that q is far away from p. see Eq. 99 and relatively small ensures that Pr[(p. q) 2 s'] is large. Let's denote e := Pr[pp, q) s such that the second term in Eq. 12 can be bounded by 0 and c. This simplifies the analysis, since
17

---

Published as a conference paper at ICLR 2025
worry about what happens inside the spherical cap with angle cos (s') around p. By construction, Pr(q E Vor(c) (c.q) 2 s'] = 1, so ge(c) Pr((p..) q) 2 s' = p(s'). This property of Theorem 1.
Step 2. The proof of property 3 of Theorem 1 (monotonicity of ge) be proven from the convexity of Voronoi cells. from Eq. can be written as an integral over a spherical cap of probability density multiplied by an indicator function of whether that part of the spherical cap is still within the Voronoi cell. We can establish monotonicity of each indicator function by simply noticing that a ray originating from a point strictly within a convex can only cross the boundary of that convex set once.
Let C (p) denote the spherical cap in S centered around with 0 = cos cos- Then,
(p) = (1 (1- S (p)
Pr/v q C² (p)) E Vor(c)]dv
(13)
where Pr(v = q] denotes the probability density function that v the closest sample to (out of the samples). is the indicator function. When p c, the indicator functions in Eq. [3 are equal to I. All indicator functions are non-increasing in all directions from within the Voronoi cell in the following sense:
1 [proisa + v) E Vor(c)] < [proiss (bu + v) E Vor(c)], Va >b 0, Q Rd, E Vor(c)
(14) Eq. 14 follows from the convexity of Vor(c), and property 3 of Theorem 1 follows from the combination of Eq. 13 and 14
Step Finally, we conclude by showing property of Theorem 1. This property states that Pr|q Vor[e(b))] S 0.5 for all points b on the boundary of Vor(c). This is easy to see. We assumed that Vor(c) is a strict subset of a hemisphere. For any point b, obviously possible to construct a hemisphere ea/2 such that all of Vor(c) contained within Ca/2 and b is on the boundary of C²/2, Clearly, the function Pr(q e²/2 is symmetric around the boundary of the hemisphere /²/2, so Pr(q E Vor(c(b))] S Pr(q E Ca/2 0.5, since Vor(e(b) S C=/2, [
(²(p
p
U
s!
Vor(c)
Figure 11: Diagram for proof of Theorem 1.
A.2 PROOF OF THEBREM 2
Firstly, we want to derive the probability that the closest point to p lies in (p)::
Pr(g(p) E (p)) = Pr(q(p) & (,)
Pr[x: (p))" (1 - Pr[x; E (p)))
(15)
18

---

Published as a conference paper at ICLR 2025
Equation 13 can be considered CDF with respect to the radius. Let's derive the PDF w.r.t. the radius by differentiating:
d - Pr(q(p) E (.)p]]
Pr(q(p) e (,))
dr
d
[1 (1 - -Pr(x; E (p))))
dr
(16)
dr (1 Pr(x; E (P()))
n (1 - Pr[x; E (D)))) Pr|xx E B,(p)] dr
Let K (2x)-4/2 Let Pr(x; (P)) denote the probability that single point drawn from (O I2) resides in the RRd ball of radius r centered at p. (p) denotes the IRd sphere of radius
around p (the boundary of (p)).
1-11111 dv
Pr(x; E (p)]
JVEB..D.
K Kexp
_ (VES.(D)
Kexp (-
1 -211v13
dvdr'
(17)
Pr|x; E (p)] = / dr
(-211113) dv
K exp
Substitute Eq. 17 into 16 to get the probability density that the closest sample to p lies in S, (p). Note that the fact that p is the closest point to p does not change the marginal distribution w.r.t r, so Pr(x, x x; E (p) = Priq(p) q(p) E (p))).
K Kexp (- 1
Pr(q(p) x (p)) E S,() (VES.(p) K exp (- | dv
E
(18)
When we substitute, the integrals cancel out: Pr(g(p) x] Pr(g(p) x q(p) E (p())))(((P))) E (p)
n (1 Pr[x; (p()))) (2x) d/2 exp
1x11113 :r: ||x p/1,, Vx Rd
(19)
A.3 PROOF OF COROLLARY 2
Purt 1. This part straightforward.
Pr[[[(()) pll r] =1 11-rr(x (p))))
V.K exp (1))) )))
(20)
r)²
where V,
Id/2+1J is the volume of ball of radius in Rdd K (2x) d/2 The upper bound in Eq. comes from lower bounding the probability density of x; ~ N(O, 1a) within B, (p) with the smallest value.
4/2
Part WLOG assume p lies along the first coordinate axis; let the scalar value of this coordinate be p pl| for simplicity. Let's introduce constant & e (0, 1). Consider the probability that the first coordinate of the n samples is greater than p². This is the probability that the max of n independent samples (.1,..., ~ (0, 1) is bigger than p². This is standard result using Chernoff-derived tail bound and union bound (Vershynin, 2018):
2 n
Pr max(T xx) S
2 1 5, 5 (0, 1)
(21)
19

---

Published as a conference paper at ICLR 2025
Using our value of pt for the bound, we get:
Pr [max(=1,.... ..... zn.) Sp"] 2 ne- ne- -22
(22)
Equation 22 implies that (r(q(p)[0] > p²1 + as p- oc, for (0, 1), where q(p) [0] denotes the
first coordinate of q. Using another union bound, we can easily show that as p oc, the probability that q()) resides within a hypercube with side lengths 2p goes to exponentially. Concretely, let. -| do denote the infinity norm, then:
Pr maxx(1 (1 ||,,,l lxc) S p²] 2 nde p /2
(23)
Now, we only need to consider the probability mass within this hypercube. We consider the approxi- mation of the ball (ixxpl(()) with the half-space Hxjoj : {x' R2 x 2 x[0]]. Considering only points that lie within the hypercube with side lengths 2p the difference between the union and intersection of B xx-p (P) and Hx goes to zero. This is because the discrepancy between the
two sets is a spherical cap with max height h = v(p (p )²)² 2 + (d 1)p2k (p p²). This occurs at the corner the hypercube. As p + oc, - 0, for & 2. This limit can be easily seen by writing h fraction:
h a² 1/2
(d - 1)p2h
a O(p + + vdptt a = v( 1-2 (d - 1)p2* b b p p²
Therefore, Pr[x; E || pl| (p)) + Pr|x, [0] 2 x(0]] The later probability is the tail of
ID Gaussian, (x)))) Substituting this into Eq. of Theorem we recover the limiting distribution in Eq. of the corollary. 0
corner of pypercube
pk]² + 11p22
1)p22
origin
p p-pA
h
Figure 12: Diagram for proof of Corollary 2.
A.4 ALTERNATIVE GEOMETRIC INTUITION TO THEOREM 2
As an alternative intuition to Theorem 2, consider the set difference between the Voronoi cell defined by centroids and the union of all Voronoi cells defined by individual gallery samples within the cluster. Queries that fall into this difference region do not retrieve the correct nearest neighbor. This difference region grows as query moves farther away from the gallery distribution. We illustrate this intuition in Figure 13 For this figure, we generate 10,000 2D Gaussian samples and cluster them into 20 clusters. The Voronoi cells of the 20 clusters plotted in solid black lines. The Voronoi cells formed by the 10,000 samples are also plotted and color-coded by cluster. When query belongs Voronoi cell that is different from the Voronoi cell of the closest sample, nearest neighbor retrieval fails. Clearly, the approximation of the union of Voronoi cells of samples by the Voronoi regions of the centroids becomes worse with increasing distance from the origin. This results in lower retrieval accuracy.
B EXPERIMENTAL DETAILS AND ADDITIONAL. RESULTS
Domain abbreviations Office Home domains: A - art; C clipart; P product; R - real. Terra Incognita domain names are anonymous location identifiers for camera traps. DomainNet domains: C- clipart; I- infograph; P painting; Q quickdraw; R real; S sketch. PACS domains: A art; C- cartoon; P- photo; S sketch. VLCS domain names are dataset names.
20

---

Published as a conference paper at ICLR 2025
Vorunol Diogram
75
30-
-30
-7.5
-10.
-7.5
-5.0
-2.
2.5 5.0
3.5 20.0
Figure 13: Alternative intuition Theorem Section A.4. We generate 10,000 2D Gaussian samples and assign them to 20 clusters. The Voronoi cells of the 20 clusters is plotted in solid black lines. The Voronoi cells formed by the 10,000 samples are also plotted and color-coded by cluster. When query belongs Voronoi cell that is different from the Voronoi cell of the closest sample, retrieval fails. Clearly, the approximation of the union Voronoi cells samples by the Voronoi regions of the centroids becomes worse with increasing distance from the origin.
11 datasets
ImageNet
Diversity preserving loss
Adaptive angrecattiion
Paired
ImageNet
Calteela
Pess
Cars
Flowers
Food
Airenft
SUN
DTD
EumSAT
UCF
Mean
Sketch
A
Mean
Open-AI CLIP
CL.IP VIT-B/16
70.3
94.6
93.5
72.7
75.9
86.6
33.9
69.1
54.7
59.1
71.5
71.1
63.5
50.0
51.9
79.0
61.1
70.0
94.6
93.5
74.6
75.7
86.5
33.3
69.5
53.7
$7.0
69.2
70.7
63.0
49.9
49.1
79.1
60.2
70.4
94.6
93.4
73.4
75.3
86.4
32.9
69.6
54.0
58.9
69.0
70.7
63.5
50.2
51.3
79.4
61.1
70.2
94.3
93.2
75.2
76.4
86.5
33.5
68.5
53.1
59,5
71.9
71.1
62.9
50.1
49.8
79.7
60.6
70.4
94.6
92.9
73.8
76.5
86.7
32.8
68.8
53.3
61.3
71.0
71.1
63.6
50.4
51.5
80.1
61.4
Table 4: Ablations experiments part 1.
The finetuning parameters are displayed in Table The training construction parameters neighhons m. and k are dataset-specific and listed Table 10 Moreover, the number of training iterations N and query/prompt template also varies with the dataset, as listed in Table I0
Hyperparameters
Ablation study An ablation study on the training set construction parameters Reniihhbrr m. A and
Nprob included in Figure [5] We perform these experiments for ImageNet, DomainNet, and Office Home. When varying the values of Reeighhess and m, we scale the value of k by the same amount. Note that changing the values of these hyperparameters changes the size of the training
21

---

Published as a conference paper at ICLR 2025
DomainNet
Ollicellome
Divensily preserving loss
Adaptive ougmenttto
Paired k-means
R
S
Mean
A
C
P
R
Mean
Open-AI CLIP VTT-RV16
74.8
54.3
69.1
15.5
85.5
66.3
60.9
85.5
73.1
92.0
91.4
85.5
74.8
52.6
68.9
15.7
85.2
66.1
60.5
85.2
70.7
91.0
90.5
84.3
74.9
53.7
69.5
16.1
85.4
66.3
61.0
84,8
70.4
90.9
90.9
84.3
75.3
52.6
69.6
16.3
85.4
66.5
60.9
85.8
72.7
91.9
90.9
85.3
75.3
53.8
69.8
16.4
85.6
66.6
61.2
85.9
73.3
92.0
91.4
85.7
Table 5: Ablation experiments part 2.
ImageNet Caltech
Pets
Cars
Flowers
Food
Aireraft
SUN DTD
EaroSAT
UCF
Mean
Open-AI CLIP VTT-B/16
neighhors
27.2
52.4
Soft pacudo labels
69.9
94.8
93.1
70.7
74.7
86.7
31.9
67.6
52.3
51.5
70.8
69.5
Contrastive loss KKhosla et 2020
68.4
93.7
93.4
72.7
77.0
86.3
33.7
68.6
54.3
59.8
71.6
70.9
training
70.3
94.6
72.7
86.6
33,9
54.7
59.1
71.1
No sample selection (skip step 3)
Random sample selection Our sample selection
69.8
94.3
93.3
72.8
76.7
86.6
33.4
68.6
54,3
57.7
73.0
70.9
69.9
94.1
93.4
73.7
76.8
86.6
33.7
68.8
54.1
58.5
72.8
71.1
Table 6: Additional ablation experiments comparing different training losses (top) and different samples selection strategies (bottom). These experiments waffleCLIP augmentation instead of the adaptive augmentation.
dataset. For example, scaling ki by scales the number of training samples by the amount. The main take-away from Figure 13 is that increasing the number of samples in the training data improves the target accuracy, but only up to a point. The target accuracy saturates some point, and is not beneficial to increase neighhonns m, or k further.
Tables and p ablation experiments justifies paired k-means, adaptive label augmentation, and the diversity preserving loss. we place a check mark next to components being used in the corresponding row. The baseline for paired k-means is k-means clustering of image features only. The baseline for adaptive label augmentation is waffleCLIP, The baseline for the diversity preserving loss (X 0.2) vanilla cross entropy.
Table performs ablation experiments that compare our loss function against existing loss functions. In this table, soft pscudo labels refer to using the logits of the teacher prediction as the label. We tuned the teacher's temperature parameter. Contrastive loss refers to finetuning with LCE + Leomrnstive where the first loss is the cross entropy loss with hard labels, and the second loss is the supervised contrastive loss (Khosla al. 2020). Lcontrastive is calculated from the image encoder outputs. Training model using both CE and a contrastive loss in this manner is commonly used in domain generalization, e.g. (Yao al. 2022). Table performs ablation experiments justifying our sample selection method in step of Algorithm 2. Our clustering-band based sample selection achieves better results than random selection or skipping sample selection.
Additional Notes We do not verify the check-sums of the downloaded images, instead we filter out retrieved images where the cosine similarity between the image embedding and query text embedding very low (<0..5)). The size of the retrieved datasets is listed in Table 10 and emphasize again that our framework achieves impressive improvements accuracy with a small number of retrieved image samples (<100K).
22

---

Published as a conference paper at ICLR 2025
Hardware and Computational Cost We experiments hybrid computing cluster with A40, A100 and L40S GPÜs. All experiments require only one GPU at a time. ViT-B/16 experiments require GPU with 40 GB of memory; VIT-B/14 experiments require GPU with 80 GB of memory. The paired k-means algorithm was run once on 20M subset of LAION. This took one hour. Adding day. The 4-bit indices require approximately
all of LAION-2B to the index takes approximately 850 GB of disk space. For Algorithm 2, using ImageNet-1K as an example, we augment each of the 1000 class names 16 times, for total of 16,000 queries. The retrieval step took 30 seconds in total. For each query, retrieve the 64 nearest neighbors, but this is not any slower than retrieving only one nearest neighbor when using FAISS search. Upon retrieval, the 64 nearest neighbors are already ranked by similarity to the query. The implementation step 2 only compares the rank of each image relative This finished in 55 seconds for the ImageNet- K target task. Clustering (step 3) then took 9 minutes and 20 seconds on one CPU, but could be casily sped up using GPU implementation of k-means. Finally, downloading the 96,000 selected images took 158 seconds.
FAISSDouuze Douze al. (2024) for approximate nearest neighbors
queries that retrieved
DomainNet
Terra Incognita
C
P
Q
R
S
Mean
100
38
43
46
Mean
Open-AI CLIP VIT-B/16
CLIP ZS
71.4
47.1
66.2
13.8
83.4
63.4
57.6
51.5
26.1
34.1
29.3
35.2
waffleCLIP
73.0
52.0
68.3
14.0
84.9
65.8
59.7
54,2
29.5
36.4
30.6
37.7
Random Descriptors
73.5
51.0
67.6
14.6
84.7
65.9
59.6
51.3
21.7
36.7
28.8
34.6
Handcrafted Ensemble
73.7
69.3
16.0
66.2
60.2
28.5
33.4
37.1
PromptStyler f
73.1
50.9
68.2
13.3
65.3
59.4
MUDG (ours)
75.3
53.8
69.8
16.4
85.6
66.6
61.2
57.7
34.6
35.7
26.8
38.7
Open-AI CLIP VIT-L/14
CLIP ZS
79.5
52.2
70.9
22.5
86.8
71.5
63.9
46.3
50.9
43.0
32.4
43.1
waffleCLIP
80.4
56.5
72.8
22.0
88.1
73.0
65.4
45.6
45.2
43.7
31.4
41.4
Random Descriptors
80.6
56.0
73.4
23.3
87.9
73.2
65.7
40.9
36.3
38.5
26.3
35.5
Handcrafted Ensemble
81.1
55.8
73.9
24.2
87.9
73.7
66.1
47.5
50.9
41.8
30.5
42.7
PrompiStyler
80.7
73.8
21.7
73.2
MUDG (ours)
81.6
58.3
74.9
24.5
88.5
74.1
67.0
53.4
53.9
46.1
32.7
46.5
Table 7: Terra Incognita and DomainNet results.
PACS
VLCS
A
C
P
S
Mean
Caltech
Labelme
SUN
VOC
Mean
Open-AI CLIP VIT-B/16
CLIP ZS
97.1
99.0
99.9
88.0
96.0
99.9
68.3
75.3
85.5
82.2
waffleCLIP
97.3
99.0
99.9
90.3
96.6
99.9
68.6
74.4
86.3
82.3
Random Descriptors
97.1
99.2
99.9
89.2
96.4
99.9
70.3
77.9
87.0
83.8
Ensemble
97.6
99.2
99.9
89.9
96.7
99.9
69.1
76.4
82.4
PromptStyler
97.6
99.1
99.9
92.3
97.2
99.9
71.5
73.9
86.3
82.9
MUDG (ours)
97.9
99.2
99.9
90.7
96.9
99.9
65.5
78.5
86.3
82.6
Open-AI CLIP VIT-L/14
CLIP ZS
98.8
99.6
99.9
95.6
98.5
99.9
70.7
73.8
85.7
82.5
waffleCIIP
99.1
99.7
100.0
95.7
98.6
99.9
70.8
74.1
87.1
83.0
Random Descriptors
98.9
99.6
100.0
95.6
98.5
99.9
67.6
78.0
86.4
83.0
Ensemble
99.6
100.0
95.7
98.5
65.5
76.1
85.1
81.7
PromptStyler t 99.1
99.7
100.0
95.5
98.6
99.9
71.1
71.8
86.8
82.4
MUDG (ours)
98.8
99.6
100.0
95.8
98.6
99.9
70.0
75.5
86.0
82.9
Table 8: Comparison of our MUDG method with ZS baselines and PromptStyler on PACS and VLCS. Average of three trials. Dataset construction and model training is performed once and evaluated on all domains. t denotes author reported numbers; all other results are our reproductions.
23

---

Published as a conference paper at ICLR 2025
ImageNet
Mean
Caltech101
50
70
UCF101
OxfordPets
60
EuroSAT
StanfordCars
DTD
Flowers002
SUN397
Food101
FGVCAircraft
Handcrafted Ensemble
waffleCLIP
Visual Descriptors
Random Descriptors
MUDG (ours)
Figure 14: Radial plot comparisons of the baselines with the pretrained VIT L/14 weights.
C DETAILED DESCRIPTION AND MOTIVATION OF ALGORITHM
Algorithm 2 consists of a three-stage pipeline presented in Figure 4 (top) used to build a pseudo- labeled subset of the source data.
Assumptions and notation. We given unlabeled source dataset X' (e.g. LAION-2B English, with text labels discarded). X, must be indexed in a joint image-text embedding space by a pair of CLIP encoders findex and findex, imago- Both are frozen. We are also given label tokens for the target classification task, formatted as "a photo of (class name)", ", and denoted as (t,,... te where c is the number of classes. The goal is optimize "student" CLIP model /suueen to classify images from the given classes. Note that fsuudent and findex can be the same or different models, and we experiment with both possibilities.
C. STEP 1: DIVERSIFIE RETRIEVAL
Goal: Retrieve a diverse set of image data for training.
The simplest way build dataset from the list of class names is calculate the text feature for each class and retrieve the nearest neighbors from X=. This is straightforward, but the results are not promising as shown the left of Figure 16 The retrieved images are not identical, but contain very little variation. For instance, images of wallets only contain one possible orientation; images of couches only contain stock photos of a perfect couch. Figure 19 (the line with blue x) shows that when trained on these images, the model severely overfits to the retrieved dataset. To diversify the dataset, we augment the query text tokens using the adaptive label augmentation scheme Section 3.2 of the main paper. From inspecting Figure right, our augmentation seems capture broad range of visual variation within each class. We also demonstrate this diversity using t-SNE plots in Figure 17
24

---

Published as a conference paper at ICLR 2025
70.0
70.0
14.5
70.40
69.0
50.0
60.5
70.30
68.0
68.0
$7.3
$7.3
67.5
67.0
67.0
57.01
Fumber ne ghocrs
number of sugmentations
sumber Custera
number of probes per quert Tpode
60.0
50.0
60.0
61.5
Damerter Keseoge
54.5
59.0
59.0
69.5
61.3
58.0
18.0
38.0
57.5
rumber ne ghaces
number sugmerbatins
sumber custra
number pribee per query
85.0
85.0
85.0
85.
64.0
04.0 0
84.0
15.50
83.0
83.0
83.0
05.35
number meighhoo
number sugmertations.
number of dustersssss ly
number probes query
Figure 15: Ablation experiments for varying values of Rneighhors m, R1, and niprobe- Reference Algorithm 2 in the main paper and Table IÔ in the Appendix for default values. Top row: ImageNet; middle row: DomainNet; bottom row: Office Home. Increasing either Reeighhorss the target accuracy by retrieving a larger training set, but these plots show that the accuracy saturates certain value. Generally, increasing Ppoube also improves the target accuracy.
k; improves
Query:
photo
Simple nearest neighber retrieval
Diversified retrieval using augmented text queries
giant panda
wallet
combine
crate
Chibuahua
rugby
matchstick
Figure 16: Qualitative results for step 1: diversified retrieval. Left: nearest neighbors to text query in LAION-2B. Right: images retrieved using diversified text features. Images retrieved using diverse queries cover a broader spectrum of appearances in the wild.
25

---

Published as a conference paper at ICLR 2025
Paired k-means Parameters
71
number of samples in LAION-2B-en
k
131072
number of iterations
10
Adaptive Label Augmentation Parameters
M
4227
unordered ImageNet descriptors from Menon and Vondrick (2022
k2
16
m
dataset dependent
Finetuning Parameters
ViT-B/16
ViT-L/14
Finetune last 3 layers of text and vision encoders batch size learning rate weight decay
128
64
0.00064
0.00016
le-5
number of iterations (N)
dataset dependent none
learning rate decay
softmax temperature
25
optimizer
SGD momentum=0.9
label smoothing
0
EMA weight averaging |3
0.995
text prompt length
3
text prompt initialization
"a photo of"" 10 x
text prompt learning rate multiplier
X
0.2
Parameters for Baselines
WaffleCLIP ensemble size
8
Table Training hyperparameters.
giam: Darda
malle!
cor combine haromber
LONE
cni roane: teghters nereed ntriwaa
Figure 17: (-SNE plots of image features in the indexing model's embedding space, showing the benefits of steps 1 and 3. Simple nearest neighbor retrieval (blue circle) covers only a small portion of the image distribution for each label. Diversified retrieval (orange dot) covers a broader portion of the image distribution, but contains semantically-redudaan samples. After the clustering step (green diamond), the selected image samples are evenly spaced across the entire distribution, and thus the best representation for each label.
C.2 STEP 2: RANK PSEUDO-LLABLLING
Goal: Mitigate hubness effect.
If each image sample is only retrieved by queries from one label, then pseudo-labeling is trivial. However, there is a large amount overlap between retrievals from different labels, especially for datasets with large number of classes fine-grained concepts. For each image that is retrieved
26

---

Published as a conference paper at ICLR 2025
Actual training dataset size
Dataset
rocighhors
m7
k1
N
Query template
ImageNet Caltech
64
16
96
300
96K
a photo
{
64
64
384
100
38K
a photo of {}.
Pets
64
64
384
200
12K
a photo
{}
a type of pet.
Cars
64
96
1000
18K
a photo of a (}.
Flowers
64
64 384
200
31K
photo of {}
a type of flower. photo of (}
Food
64
64
384 100
34K
a type of food.
Aircraft
64
64 384 1000
26K
photo of ()
a type of aircraft.
SUN
64
16 96 300
38K
photo of ().
DTD
64
64
384
200
18K
photo of {} texture. {}
EuroSAT
64
64
384
200
3K
a photo from a satellite.
UCF
64
64
384
200
37K
a photo doing ().
person
ImageNet-V V2
64
96
200
96K
a photo a (}.
ImagcNet-Sketch
64
16 96
200
96K
photo of {}.
ImageNet-A
64
96
200
19K
a photo of a {}.
ImageNet-R
64
16 96
200
19K
photo of (}.
DomainNet
64
16 96
200
33K
a photo of a (}.
Office Home
64
64 384 200
25K
photo of ().
PACS
64
64 384 100
3K
a photo of ().
VLCS
64
64 384 50
2K
photo of ().
Terra Incognita
64
64 384 100
3K
a photo of {}
from camera trap.
Table 10: Dataset-specific hyperparameters, reference Algorithm 2 in the main paper. Rneighhors is number of nearest neighbors to be retrieved; m number of augmentations; k is number of k-means clusters; N is number of training iterations.
Augmentation
Loss (Eq.
photo C. which may have multiple setcings (LoM, nedium, high).
photo of C, which often has a design or logo.
photo of Cs which has people often in close preximity.
photo of
which is gradually incressing decreasing diameter.
photo Cr which usually rectangulas square shape.
photo of C, whicch ch 1a a piece of clothing.
photo of Us whicch 1a piece o* armor.
16
photo of
which is
dish.
16
cord. photo of C. which is a pen with deccrative band or ring.
photo C. which phone
16
Table 11: Qualitative results for our adaptive text augmentation on ImageNet. Losses are calculated based on Equation 6 ko 16. The loss value is an integer in range [0, kz).
by multiple queries, we can assign it either (1) the label of the closest text feature measured by their cosine similarity, or (2) the label of the text feature to which it is ranked the highest. We choose the latter option (detailed concretely in Algorithm 2) to address the well-known hubness effect (Radovanovic et al. 2010). In simple terms, hubs are samples in the dataset which tend to closer to other samples in high- dimennioon embedding space, regardless of relevance. Specific to application, "hub" text feature is one that close to a disproportionately large number of image samples, resulting in large number of image samples being assigned the hub label. In other words,
27

---

Published as a conference paper at ICLR 2025
Dase Classification Problem
Good Augmented Clssification Problem
Bad Augmented Gasiiicction Problem
Figure 18: t-SNE visualization corresponding to Figure the introduction. We randomly select 100 ImageNet class names and compute the text features corresponding to the "a photo a { prompt (referred to the base classification problem). Additionally, from Table 11 with a loss of 16, and good augmentation with a loss O. The features are aggregated and visualized in this figure. The augmentation with higher loss does not preserve the variance of the classifier prototypes, likely reducing the ZS accuracy and diminishing its effectiveness.
use bad augmentation selected
ImagoNet MT
75 havelre
230
Figure 19: Target accuracy training iterations for the datasets corresponding to Figure 17 (colors match). This confirms our intuition that both the diversified retrieval and clustering steps are necessary. here refers to neighbbrrs
Terra Incognits, m = n-64
Preceton imgrovement
Recall Improvement
100
0.15
6.10
powst abord Uoms sitt enyy pbddt
0.05
100
0.00
46.0
rabat
reccom
deg
reccis
rouree
cal
coyett
dag
reset
coutrer
Figure 20: Hubness effect and the value of aseudo-labeling based on rank (step 2). The x-axis labels are the label names for Terra Incognita. The label "empty" is a hub because about 50 more images were labeled as empty when cosine similarity is used instead of rank (left bar plot). The right two bar plots show the precision and recall improvement of rank labeling over cosine similarity labeling, after clustering and training. Rank labeling improves precision for images labeled empty while improving the recall for most animal images. This is desireable: The cost of mislabeling an animal image empty is much greater than the cost of mislabeling empty image. here refers to neiighoors
the pseudo-label is biased towards any hubs in the label space when cosine similarity is used directly. However, when we use rank to assign labels, the hub label cannot be overused because closeness to the hub is determined by rank relative to other image samples.
28

---

Published as a conference paper at ICLR 2025
Figure 21: selection of images from the 50 that were labeled as "empty" by cosine similarity but as one of the animals by rank.
Terra Incognita
Moan Target accuracy over D.maann
38.0
rank poeudo labeled
cosine similarit pseudo-labeled
37.5
baseline
37.0
36.0
35.5
20
40
60
100
Training kerations
Target accuracy training iterations for the two datasets constructed for Figure 20 left. Colors match. This confirms our intuition that rank labeling improves overall target accuracy in addition to the good precision-recall properties in Figure 20 right. here refers Tneighhers-
Figure 22:
Not all datasets have hubs, but found that the Terra Incognita dataset illustrates the effect perfectly. This dataset contains camera trap images of different animals, and the labels are the animal names along with "empty" for empty images. As case study, we retrieve images from LAION-2B using step and the query: "a photo of (class name). from camera trap.". We then compare pscudo- labeling using cosine similarity versus using rank. The left bar plot in Figure 20 shows that cosine similarity pseudo-labeling assigns some images the "empty" label, which are labeled as one of the animals when using rank. Figure displays examples of these images. For this dataset, "empty" likely functions as hub, since many camera-tray images are mostly empty, especially if the animal small. We verify in the two right bar plots of Figure 200that using rank pseudo-labeling improves the recall of most animal images the expense of decreasing the recall of empty images. This is a favorable trade-off for this application. We further verify in Figure 22 that rank pscudo-labeling improves the overall accuracy as well, compared to cosine similarity labeling.
Number of samples per label constructed dataset
200
400
600
800
1006
ImageNet-1K 1K labels sorted increasing samples simple neighbars retrieval
diversified retrieval ly axis scalle by 16) diversified retrieval
Figure 23: Label distribution of datasets constructed before and after clustering (step 3). For this figure, m neeighhors = 1 16.
C.3
STEP 3: CLUSTERING
Goal: Select representative samples and balance the label distribution.
29

---

Published as a conference paper at ICLR 2025
note that the dataset resulting from diversified retrieval (in orange) actually lowers target accuracy on ImageNet when used to train the student CLIP model, despite containing large number of samples (O (e(Cnnnnggh This stems from two problems: (1) Some images are semantic-duplicater as evident by the small clusters of orange dots in Figure 17 e.g. pictures of the same object in different orientations. (2) The dataset is imbalanced as shown by the orange distribution over labels in Figure 23 This is simply caused by asymmetries in the retrieval and download process (e.g. dead links, linked image changed since dataset creation, etc.), As result, the training process overfits to dominant semantic-duplicat images and the pseudo-label distribution; both are artifacts of the dataset construction process.
Referring back Figure 19
To address both of the above issues, we first use k-means clustering to cluster the image features in the embedding space of the indexing model into k <<, McTrneighhon clusters, then randomly select an image from each cluster. If k is chosen conservatively, semantic duplicates fall into single cluster, and only one can be selected for the final training set. Additionally, each label should have 1 training samples. Figure 23 illustrates the final balanced label distribution in green, and Figure shows the corresponding target accuracy improvements in matching colors. For reference, ImageNet- IK has 1000 labels, and we found
16, Reiighbrrs = 64 and l
48 to yield good results.
30


Reviews:
[{"review_id": "2vDfnbBZLq", "reviewer": "Reviewer_qfP3", "summary": "This paper introduces Multimodal Unsupervised Domain Generalization (MUDG), a novel framework that leverages large-scale unlabeled datasets (like LAION-2B) to improve model generalization on target domains. The paper makes three main contributions: (1) proposing paired k-means algorithm to improve cross-modal retrieval accuracy by updating centroids in query space rather than image space; (2) designing an unsupervised text augmentation scheme that adaptively selects appropriate descriptors for target labels; and (3) introducing a clustering-based sample selection strategy and diversity-preserving loss function to construct more representative training datasets. The method demonstrates significant improvements over existing zero-shot learning and source-free domain generalization baselines across 20 diverse datasets.", "questions": "For detailed questions and suggestions, please refer to the Weaknesses section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel MUDG framework that effectively leverages large-scale unlabeled datasets for domain generalization through cross-modal retrieval and adaptive text augmentation.\n2. The paired k-means algorithm effectively addresses the modality gap in cross-modal retrieval, supported by rigorous theoretical analysis and empirical validation.\n3. Extensive experiments across 20 diverse datasets demonstrate significant improvements over state-of-the-art baselines in zero-shot learning and domain generalization, showing strong practical value.", "weaknesses": "1. While the paper introduces MUDG as a new paradigm beyond UDG, it would benefit from a visual comparison/illustration showing the differences between DG, UDG, and MUDG. Moreover, the necessity and practical benefits of this new task definition are not sufficiently justified - authors should more explicitly demonstrate why existing frameworks like UDG are inadequate and how MUDG addresses real-world challenges.\n2. In this paper, authors explore a way to retrieval task-agonistic source dataset to assist in model learning in the target domain. What is the intuition behind this operation? Is the assumption reasonable?\n3. The writing of the Introduction Section is confusing: authors present a group of operations and six contributions in the section of introduction. However, what is the core challenge and the corresponding solution method proposed in this paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsvvLM4DEI", "reviewer": "Reviewer_TM1x", "summary": "This paper presents a novel approach to multimodal unsupervised domain generalization (MUDA), where the source data is both unlabeled and task-agnostic. First, it theoretically examines the issues with existing cross-modal approximate nearest neighbor search methods. Next, it proposes a paired k-means algorithm to enhance neighbor search efficiency. Finally, we introduce a text augmentation scheme aimed at improving zero-shot accuracy.", "questions": "In Table 3, there are no MUDG methods included. Could you combine the PromptStyler method with the proposed methods for a fair comparison?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The paper introduces the MUDG setting and presents a novel paired k-means algorithm to address the cross-modal nearest neighbor problem.\n\n2.The paper provides a thorough theoretical analysis demonstrating the limitations of existing cross-modal nearest neighbor methods, supported by extensive experimental validation\n\n3.Figures are effectively used to illustrate empirical observations, the improvements brought by paired k-means, and the visualization of Voronoi cells, enhancing the understanding of the proposed method.", "weaknesses": "1.The paper compares MUDG setting to zero-shot (ZS) and source-free domain generalization (SFDG) setting in Tables 1. ZS and SFDG impose more stringent conditions, while MUDG relies on task-agnostic source data, making it less practical in some contexts. Therefore, comparing MUDG methods with SFDG or ZS methods may not be entirely fair.\n\n2.The paper lacks a comparison of the computational resources and time required by its proposed method in relation to other SFDG and ZS methods. Since this setting involves large-scale datasets during training, it may significantly increase the computational resources and time needed\n\n3.In Table 2, the proposed method shows only a slight increase of 0.6% compared to the second-best method.\n\n4.In the ablation experiments (Table 5), when adaptive augmentation is used alone, there is a drop of about 1%, with some datasets not even reaching that. This indicates that the paired k-means method, which is the main contribution of this work, is not effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CaWstQAXIl", "reviewer": "Reviewer_gdYC", "summary": "This paper presents a novel approach to multimodal unsupervised domain generalization that leverages a large unlabeled source dataset for finetuning, without requiring a direct relationship with the target task. Key contributions include (1) a theoretically derived paired k-means algorithm that enhances nearest neighbor recall, (2) an adaptive text augmentation scheme to improve zero-shot accuracy, and (3) two additional components that boost target accuracy. Experimental results demonstrate consistent accuracy improvements across 20 diverse datasets.", "questions": "See 'weakness' for details. A more thorough introduction to the motivation and clarification of the methods are needed. If the author can improve the presentation of the paper in the next version, I would be happy to raise my score!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper is well-motivated and appears to be reproducible based on the author's code. The author discusses the challenges of cross-modal retrieval from a theoretical perspective and proposes methods to address the issue of low recall. Additionally, the author proposes an adaptive text augmentation scheme to tackle unsupervised domain generalization in multimodal contexts. Comprehensive analyses, comparative experiments, and insightful visualizations highlight the appropriateness of the method's genuine effectiveness.", "weaknesses": "1. While the methodology appears to be fundamentally solid, my primary concern regarding this paper is the clarity of expression and how it affects overall readability. Several figures in the paper, such as Figure 1, present ambiguous meanings, and others, like Figures 4 and 5, lack sufficient quality. Additionally, there are issues with the details in the descriptions, including an overuse of abbreviations and numerous typos in the introduction. These factors contribute to a challenging reading experience for the audience.\n2. I believe a more detailed introduction to the task setting and motivation is necessary. Although the author mentions that MUDG is more practical, it seems to be primarily a combination of text-image representation and UDG. Could the author elaborate on specific application scenarios?\n3. In terms of retrieval, I’m curious why only the first-level k-means is utilized instead of employing a fine quantization scheme at the second level. What are the implications of this choice, and might it impact the recall rate?\n4. In Table 2, the proposed method does not seem to achieve superior performance across many datasets. Given that MUDG is being addressed so specifically, what could explain the lack of a more significant effect? What might be the underlying reasons for this?\n5. Lastly, regarding the key hyper-parameters listed in Table 9, how are these parameters determined? Are they sensitive to variations, and how do they influence the performance of the method? This aspect warrants further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JaAM7i0TUk", "reviewer": "Reviewer_6mKA", "summary": "This paper studies multimodal unsupervised domain generalization (MUDG), where only task-agnostic source data and target label names are required. Accurate yet diverse retrieving image samples that match the specific text description for a certain class name becomes the crux. Towards the former, the authors propose paired k-means, maximizing the probability of a text query and its closest image sample belonging to the same Voronoi cell by updating the centroids to be in the query distribution. As for the latter, the authors develop a heuristic augmentation strategy. Experiments under various evaluation protocols demonstrate the effectiveness of the proposed method.", "questions": "I have no further questions. Please refer to the \"Weaknesses\" section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper is well-written overall, and the motivation is quite reasonable.\n- Each proposed component is supported by sufficient theoretical and empirical analysis.\n- Experiments are sufficient.", "weaknesses": "- Table 1 should be placed before Figure 1 as Table 1 was mentioned before Figure 1.\n- The bottom right subfigure of Figure 1 was an illustration. Could you provide a quantitative comparison between good augmentation and bad ones (such as using t-SNE visualization)?\n- Ablations in Tables 4 and 5 can not reflect the effectiveness of each component, as the improvements are marginal. Could you report the results of statistical testing to see if the improvements are significant or not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)