# LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts

Anh-Quan Cao, Maximilian Jaritz, Matthieu Guillaumin, Raoul de Charette, Loris Bazzani

► To cite this version:

Anh-Quan Cao, Maximilian Jaritz, Matthieu Guillaumin, Raoul de Charette, Loris Bazzani. Latte-CLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts. WACV 2025 - Winter Conference on Applications of Computer Vision, Feb 2025, Tucson (Arizona), United States. hal-04986162

HAL Id: hal-04986162

https://inria.hal.science/hal-04986162v1

Submitted on 11 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# LATTECLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts

Anh-Quan Cao²*

Maximilian Jaritz¹

Matthieu Guillaumin¹

Raoul de Charette²

Loris Bazzani¹

¹Amazon

²Inria

## Abstract

Large-scale vision-language pre-trained (VLP) models (e.g., CLIP [46]) are renowned for their versatility, as they can be applied to diverse applications in a zero-shot setup. However, when these models are used in specific domains, their performance often falls short due to domain gaps or the under-representation of these domains in the training data. While fine-tuning VLP models on custom datasets with human-annotated labels can address this issue, annotating even a small-scale dataset (e.g., 100k samples) can be an expensive endeavor, often requiring expert annotators if the task is complex. To address these challenges, we propose LATTECLIP, an unsupervised method for fine-tuning CLIP models on classification with known class names in custom domains, without relying on human annotations. Our method leverages Large Multimodal Models (LMMs) to generate expressive textual descriptions for both individual images and groups of images. These provide additional contextual information to guide the fine-tuning process in the custom domains. Since LMM-generated descriptions are prone to hallucination or missing details, we introduce a novel strategy to distill only the useful information and stabilise the training. Specifically, we learn rich per-class prototype representations from noisy generated texts and dual pseudo-labels. Our experiments on 10 domain-specific datasets show that LATTECLIP outperforms pre-trained zero-shot methods by an average improvement of +4.74 points in top-1 accuracy and other state-of-the-art unsupervised methods by +3.45 points.

## 1. Introduction

Large-scale vision-language pre-training [46] has emerged recently and demonstrated impressive generalization performance on various downstream tasks [7, 8, 23, 30, 44], especially in zero-shot classification [46, 47]. This success is attributed to its robust visio-linguistic representation, learned from a vast amount of large-scale web-scraped datasets [48]. However, these models often face challenges

<figure><img src="image_2.png" /><figcaption>Figure 1. Overview of LATTECLIP. Our prototype-based method leverages different types of pseudo-labels and LMM-synthetic texts for improved unsupervised CLIP fine-tuning on domain-specific datasets (e.g., texture). During inference, image features are compared with prototypes to generate predictions. Here, $f(·)$ and $g(·)$ are the CLIP image and text encoders, respectively.</figcaption></figure>

in specialized domains due to domain discrepancies and insufficient representation in the training data. Prior studies have demonstrated improvements on custom datasets through supervised fine-tuning [15, 60] or few-shot learning [50, 68]. Nevertheless, acquiring human-annotated labels is costly, even for relatively small datasets (e.g., 100k samples), and often requires expert annotators for complex tasks. To address this, we propose LATTECLIP, which fine-tunes CLIP for classification on unlabeled training data to maximize performance on a test set from the same domain. Here, a *domain* refers to a set of shared characteristics within a dataset (e.g., cars, flowers, textures). Like in Unsupervised Domain Adaptation (UDA) [12, 19, 58], we consider the list of class names to be known a priori. An overview of LATTECLIP is shown in Fig. 1.

Recent progress of Large Language Models (LLMs) [3, 24, 41, 55, 56] and Large Multimodal Models (LMMs) [4, 34] have led to a fundamental shift in training and fine-tuning methodologies. The research community is transitioning from a class-focused paradigm towards a more descriptive approach, where data is annotated with detailed textual de-

arXiv:2410.08211v1 [cs.CV] 10 Oct 2024

<footnote>*The main work was done while interning at Amazon.</footnote>
scriptions for training, and rich textual answers are provided at inference time. Consequently, an increasing number of methods [10, 16, 28] now leverage synthetically-generated text from LMMs as an additional source of supervision or contextual information to improve performance. Similar to these approaches, we harness the power of LMMs to generate descriptions for training, but with a strong emphasis on producing more expressive descriptions. Instead of only generating per-image descriptions, we also generate descriptions for groups of images, capturing their common characteristics, as well as class-level descriptions for all images within a category. These descriptions provide better contextual information, offering richer supervision for training, leading to improved classification accuracy in specific domains compared to the limited information from pseudo-labels and label propagation [20, 29].

However, directly fine-tuning CLIP with LMM-generated texts leads to poor performance due to CLIP overfitting to hallucinations and noise present in these texts. To address this, we propose a fine-tuning framework based on prototype learning [1, 39], where classes are represented as a set of prototypes, typically as feature vectors. Prototypes provide better control and interpretability of class representations through direct manipulation in the embedding space, helping regulate the influence of each synthetic description during training. To further improve the per-class prototype representations, we combine the synthetic texts with two types of pseudo-labels derived from both zero-shot and fine-tuning models. The zero-shot model offers better generalization thanks to pre-training knowledge, while the fine-tuning model provides stronger in-domain performance. During inference, these prototypes are compared with image features for classification. As LMMs are only employed during during fine-tuning, the inference time remains consistent with standard CLIP methods. We validate the effectiveness of our method across 10 domain-specific datasets. Compared to pre-trained CLIP models, LATTECLIP achieves an average improvement of +4.74 points in top-1 accuracy, surpassing other unsupervised fine-tuning baselines by +3.45 points.

Our contributions can be summarized as follows:

* We propose LATTECLIP, a novel method that synthesizes multiple types of image descriptions to enhance the unsupervised fine-tuning of CLIP models on domain-specific datasets, leveraging the language expressiveness of LMMs.
* To make training robust to noisy texts and pseudo-labels, we employ a prototype framework with a momentum update, enabling us to control the influence of synthetic text features. To further refine the useful image descriptions, we introduce a Dynamic Feature Mixer module that assigns higher weight to important text, resulting in better-combined text embeddings.
* We show that mixing pseudo-labels from zero-shot

model and fine-tuning model significantly improves performance; the former preserves pre-trained knowledge, while the latter improves the accuracy on the target distribution. Experiments show that LATTECLIP significantly outperforms all baselines on average across 10 domain-specific datasets.

## 2. Related works

**Adapting CLIP for Classification.** CLIP-based methods [46, 62, 63, 65] exhibit competitive zero-shot classification performance. For further improvement on downstream classification datasets, CLIP can be adapted to close the gap between pre-trained representations and specific domains.

In few-shot learning, one has access to a small number of labels, typically between 1-16 samples per class, and many works have adapted it to CLIP [9, 13, 14, 38, 45, 50, 64, 66, 68, 69]. Prototypical learning [51] is a seminal work in few-shot learning and builds an average embedding (prototype) for each class. During inference, one then matches the test sample to the nearest prototype. This concept recently re-emerged for adapting CLIP, by building a cache model holding the knowledge from the few-shot training set [66,69]. Different from that, we continuously update the prototypes with momentum during the training process with multimodal features from *unsupervised* texts and images. Other works leverage prompt learning [68] or efficient fine-tuning [9, 64].

Supervised fine-tuning methods require a significant amount of labeled examples for training [15, 26, 46, 59, 60]. Linear probing [46, 60] is a simple technique that trains a classifier on top of frozen image features, but can lead to worse results due to overfitting. This problem has been tackled by using two-step training schedules of linear probing and full fine-tuning [26], masked image modeling [59] and by fine-tuning with contrastive loss by aligning the image with a template text including the class label (FLYP [15]). In contrast to FLYP, we add LMM-generated descriptions to the contrastive loss, and stabilize unsupervised training by learning prototypes with momentum.

Different from few-shot and fine-tuning, we focus on the challenging scenario of unsupervised fine-tuning, where no labels are available, because they are too costly to annotate.

**Unsupervised Model Adaptation** Our Unsupervised fine-tuning task is related to Unsupervised Domain Adaptation, where one typically reduces the discrepancy between the source and target data. However, lately, the task of source-free domain adaptation (SFDA) has emerged, where target adaptation is performed without access to the source data, see survey paper [31]. Many methods exploit that the source model can partially generalize to the target domain, and fine-tune with pseudo-labels [33], adversarial learning [32], historical contrastive learning [21] or mixup [27]. While [21] perform momentum contrastive learning [17] on different
image augmentations, we contrast image-text pairs. The
above SFDA works train on a narrow source distribution.
Instead, ReCLIP [20] leverages CLIP, which is pre-trained
on wide-distribution large-scale data. ReCLIP leverages
pseudo labels, cross entropy between separate modalities
and focus on transductive setup (train/test on test set). Test-
time adaptation methods [35, 49, 53, 57, 67] update the model
to align with the target distribution at test time using a single
image in self-supervised manner, requiring optimization at
inference. Unlike these approaches, we leverage LMM-
generated texts to maximize test performance by fine-tuning
model parameters on unlabeled training data, thus keeping
the model parameters fixed during testing.

**LMMs for Synthetic Labels.** Using synthetically-generated labels and textual descriptions is becoming a standard in the field, because of the general availability of LLMs and LMMs that can be prompted and guided with task-specific examples [2, 10, 16, 28, 34, 42, 54]. This provides an opportunity for VLP, that typically uses large-scale image-text pair data scraped from the internet, e.g., LAION-5B [48]. Instead of using noisy and inconsistent captions or annotating a large set, we synthetically-generate descriptions. While LaCLIP [10] rewrites existing captions with LLMs (text-only input), VeCLIP [28] prompts an LMM to caption the image, followed by LLM text processing. Synth-CLIP [16] synthesizes first the text, and then the images with text-to-image generative models. In our work, we leverage LMMs to caption images, focusing on fine-tuning rather than pre-training, combining the pre-trained model’s knowledge with new synthetic captions in a balanced approach.

3. Method

Fine-tuning CLIP with combination of predefined tem-
plates, such as "a photo of a [class].", was shown
to yield effective results when using ground-truth class
labels [15, 26, 60]. However, in the absence of ground-
truth class labels, fine-tuning CLIP models with pseudo-
labels [29], using FLYP [15], leads to limited improvements<fnref n="1" />
This can be caused by two factors. First, the text employed as
supervision, resulting from the combination of the template
and pseudo-label, lacks expressivity and discriminativity.
This is typically the case for classes that are not visually
descriptive, such as types of land use (e.g., annual crop, in-
dustrial, etc.) or names of textures (e.g., paisley, sprinkled,
etc.). Second, pseudo-labels are inherently noisy, which neg-
atively affects the downstream classification performance
due to domain shifts relative to the original training data.

Our method, LATTECLIP, addresses these limitations

<figure><img src="image_3.png" /><figcaption>Figure 2. **Text Generation with LMM.** In addition to the usual *class-description* (middle), combining template text and pseudo-label, we leverage LMM [34] to generate *image-description* (top) which provide more expressive visual description of the image. Further, by considering random group of images with the same pseudo-labels, we prompt [34] to capture shared characteristics as *group-description* (bottom).</figcaption></figure>

by proposing an expressive unsupervised text genera-
tion (Sec. 3.1) and a prototype-based learning mecha-
nism (Sec. 3.2) to mitigate noisy pseudo labels. To improve
expressivity beyond pseudo-labeling, we build upon a re-
cent LMM [34], generating descriptions at multiple levels
of contextual granularity, describing the individual image,
group of similar images, and entire class. Individual im-
age descriptions offer detailed though possibly extraneous
information, which is addressed by group descriptions that
capture shared characteristics of similar images, albeit with
some noise. This noise is mitigated by class descriptions,
which provide stable representations to address inconsisten-
cies. Equipped with such textual description, we addition-
ally introduce a prototype-based learning framework that
learns a set of class prototypes from the generated text fea-
tures. These prototypes are updated in a momentum setting
to produce a smooth optimization over the whole training
set, reducing the effect of noise from outlier samples and
incorrect synthesized texts.

**3.1. Expressive Text Generation with LMMs**

Without access to ground-truth labels for training CLIP
models, we must rely on noisy pseudo-labels. Furthermore,
class names alone often lack visual descriptiveness. Conse-
quently, using only cross-entropy loss or solely relying on
class names leads to suboptimal performance in our setting.
To address this challenge, we introduce a novel approach that
leverages generated text to provide additional contextual in-
formation. In addition to the more standard *class-description*
mentioned above, we propose two additional ways using a
recent LMM [34] to generate textual descriptions of im-

<footnote n="1">¹Tab. 1 reports the performance of “FLYP + pseudo labels” where we show limited improvements with respect to CLIP across 10 datasets on average. We also observe performance drops on some datasets (e.g., Food101, Flower102).</footnote>
ages: *image-description* and *group-description*, as depicted in Fig. 2. These generated texts hold complementary information with increasing semantic abstraction, from class<fnref n="2" />, to single image, to group of images, all of which help the model to learn more precise classification boundaries. The *image-description* texts provide detailed descriptions of individual images, capturing their unique characteristics and subtle features. The *group-description* texts offer a comprehensive description representing the entire class, covering shared features and common attributes.

Importantly, we found that the above mentioned descriptions are complementary to the use of template text with pseudo-label class, which we refer as *class-description*. In fact, we later show that preserving this *class-description* in the training process is crucial as our generated texts can be noisy due to missing details or hallucination. The combination of class-/image-/group-description provides a stable and reliable representation corresponding to the classes.

More formally, for each image $x$ we generate three texts, illustrated in Fig. 2, and defined as follows:

**Class-description ($T^{class}$)** provides a consistent class representation using template "a photo of a [class]." where `[class]` is substituted with the image pseudo-label $c$ obtained from a CLIP zero-shot.

**Image-description ($T^{image}$)** captures unique features of image $x$. We generate $T^{image}$ by prompting LLAVA [34] with: "Describe the [domain] in the photo concisely, using less than 20 words." where `[domain]` is replaced with the dataset domain (e.g., flower, product, pet, car, etc.). We show *image-description* examples in Fig. 5.

**Group-description ($T^{group}$)** captures shared visual characteristics between similar images, to combat known limitation of LMM which may miss or hallucinate visual characteristics [36]. To generate $T^{group}$ from image $x$, we randomly sample multiple images with the same pseudo-label as $x$. These are collaged into a single image $x^{group}$ fed to LLAVA which is prompted with: "Describe the common visual attributes of the [domain] in all the photos concisely, in fewer than 20 words.". Examples of such group-descriptions are illustrated in Fig. 5.

To generate these descriptions, we use LLAVA 1.6 [34] with a 4-bit quantized Mistral 7B model. This model requires approximately 5GB of GPU memory and takes around 1.2 seconds to generate a single description per image on a Tesla V100 GPU. This makes description generation relatively cost-effective, as we can run five instances of this model in parallel on a Tesla V100 32GB GPU, taking approximately 3.4 hours to generate descriptions for 50k images.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** **Training.** For image $x$, we predict pseudo-label $c \in \{c_{zs}, c_{ft}\}$ and create three type of descriptions per pseudo-label as described in Sec. 3.1. Our Dynamic Feature Mixer combines these descriptions with the corresponding prototype $p_c$ to produce a prototype-text embedding $\bar{t}$, which updates the prototype $p_c$. Lastly, the contrastive loss Eq. (3) is computed between $\bar{t}$ and the image embedding $f(x)$.</figcaption></figure>

## 3.2. Prototype-based CLIP fine-tuning

Adopting directly the generated texts from Sec. 3.1 is ineffective, because the text encoder overfits to the distribution of generated texts, which are noisy by construction due to hallucinations of the LMM and missing details. We confirm this experimentally in Tab. 2, rows 4, 5, 6. Therefore, we propose a prototype learning approach that is capable of determining the important synthetic texts and learning better class representations from them. Our approach mixes three key ingredients as shown in Fig. 3: (1) a simple strategy to preserve robustness by leveraging pseudo-labels from both frozen and fine-tuning CLIP models; (2) a feature mixer that dynamically balances the importance of each text $T^{class}$, $T^{image}$ and $T^{group}$; (3) a module that updates the prototypes during training, stabilizing the learning process.

**Dual Pseudo-labels.** As in WISE-FT [60], we observe that training only with pseudo-labels from the fine-tuning model improves accuracy but at the cost of overfitting to the training distribution. Hence, to preserve robustness, for each image we employ two pseudo-labels $\{c_{zs}, c_{ft}\}$ originating from both the zero-shot model ($c_{zs}$) and the fine-tuning model ($c_{ft}$). We later show that this simple strategy offers greater generalization and accuracy.

**Prototype Learning.** From the generated texts and pseudo-labels, we aim to learn a set of prototypes corresponding to all classes, denoted as $\{p_c\}_{c=1}^C$. These prototypes are designed to capture class-specific details of the syn-

<footnote n="2">²Here, "class" refers to the class-description.</footnote>
thesized texts and pseudo-labels within the CLIP embedding space. First, the prototypes are initialized with features derived from the $T^{class}$, generated based on its associated class name. Then, for an image $x$, we use our dual pseudo-labels from zero-shot and fine-tuning {$c_{zs}, c_{ft}$} to generate two class-description texts {$T_{zs}^{class}, T_{ft}^{class}$} and select the corresponding prototypes $p_{zs}$ and $p_{ft}$. Our feature mixer strategy, detailed below, then combines the two class-descriptions with the image-description and the group-description, therefore obtaining two prototype-text embeddings $\bar{t}_{zs}$ and $\bar{t}_{ft}$, see Fig. 3. We then apply a momentum update to the corresponding prototypes. Finally, we apply two contrastive losses [15] between the image embedding $f(x)$ and each of the prototype-text embeddings $\bar{t}_{zs}$ and $\bar{t}_{ft}$.

**Dynamic Feature Mixer.** To compensate for noisy text descriptions, we propose a mechanism that dynamically re-weights the three descriptions as a function of the cosine similarity between each description embedding and corresponding prototype, see Fig. 4. Intuitively, our goal is to assign higher weights to descriptions uniquely describing a class and lower weights to generic descriptions. In the general case, for a text $T$ we first compute the cosine similarities between its CLIP embedding $g(T)$ and each of the prototypes, and obtain its weight $w$ from the difference between the two closest similarities. This writes:

$$w=\text{top}_1\left(\frac{g(T) \cdot p_c}{\|g(T)\|\|p_c\|}\right)_{c=1}^{C}-\text{top}_2\left(\frac{g(T) \cdot p_c}{\|g(T)\|\|p_c\|}\right)_{c=1}^{C} \quad (1)$$

where $\text{top}_1(\cdot)$ and $\text{top}_2(\cdot)$ return the largest and second largest values of the input set, respectively. A high weight indicates a large gap between $\text{top}_1(\cdot)$ and $\text{top}_2(\cdot)$ similarities, ensuring the text feature is uniquely similar to a single prototype while dissimilar from the rest, as $\text{top}_2(\cdot)$ value serving as an upper bound for the similarity of the remaining prototypes. Alternatively, we could use the mean or median, but this might result in a text being very similar to a few prototypes while remaining dissimilar to others. Subsequently, given the set of texts {$T^{image}, T^{group}, T^{class}$} and the weights {$w^{image}, w^{group}, w^{class}$} computed using Eq. (1). The resulting prototype embedding $\bar{t}$ is defined as

$$\bar{t} = (1 - \alpha) \frac{\sum_{i \in I} w^i \cdot g(T^i)}{\sum_{i \in I} w^i} + \alpha p_c \quad (2)$$

where $I$={image, group, class} and $\alpha$ is the prototype weight. We empirically set $\alpha$ to 0.99 in all experiments to stabilize training, as the prototypes are more reliable than the synthetic text embeddings. Thus, this act a strong regularization mechanism against the noise induced by the synthetic texts. Yet, $\bar{t}$ remains tailored for each image as $T^{image}$ and $T^{group}$ differ. With two pseudo-labels per image, this results in two prototype-text embeddings {$\bar{t}_{zs}, \bar{t}_{ft}$}.

<figure><img src="image_5.png" /><figcaption>Figure 4. **Dynamic Feature Mixer.** We compute cosine similarities between each text feature and all prototypes. Weights are determined by the difference between the top two similarity scores. We calculate a weighted average of the features and combine it with the prototype (Sec. 3.2), creating a representation relevant to the input prototype yet distinct from others.</figcaption></figure>

**Training.** Given an image $x$, we train both the image encoder $f(\cdot)$ and text encoder $g(\cdot)$ using contrastive loss to align the image embedding $f(x)$ with both the prototype-text embeddings $\bar{t}_{zs}$ and $\bar{t}_{ft}$, resulting in two losses $\mathcal{L}_{zs} = \mathcal{L}_{\text{con}}(x, \bar{t}_{zs})$ and $\mathcal{L}_{ft} = \mathcal{L}_{\text{con}}(x, \bar{t}_{ft})$ respectively with $\mathcal{L}_{\text{con}}(\cdot, \cdot)$ defined as:

$$\mathcal{L}_{\text{con}}(x, \bar{t}) = - \frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(f(x) \cdot \bar{t}_i / \tau)}{\sum_{j=1}^{N} \exp(f(x) \cdot \bar{t}_j / \tau)} - \frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\bar{t}_i \cdot f(x) / \tau)}{\sum_{j=1}^{N} \exp(\bar{t}_j \cdot f(x_j) / \tau)}, \quad (3)$$

where $N$ is the batch size and $\tau$ is the temperature parameter as in [46]. The first term of Eq. (3) normalizes over text embeddings to match the correct text to an image, while the second normalizes over image embeddings to match the correct image to a text. The final loss is $\mathcal{L}_{zs} + \mathcal{L}_{ft}$.

**Momentum update prototypes.** For a pseudo-label $c$, we derive the corresponding prototype-text embedding $\bar{t}$ for each image. During training, the average prototype-text embedding $\bar{t}_{\text{batch}}$ is computed over the images in the batch. Using the pseudo-label $c$, we update the respective prototype $p_c$ with a momentum $\mu$, obtaining the updated embedding $\bar{p}_c = (1 - \mu)\bar{t}_{\text{batch}} + \mu p_c$, which is then stored back in the prototype bank as the prototype for class $c$. Momentum update works effectively when $\mu \in \{0.99, 0.999, 0.9999\}$ [17]. As we fine-tune on smaller dataset with fewer iterations, we set $\mu$ to 0.99 for faster updates of the prototypes. Intuitively, the prototype can be viewed as the running average
<table><caption>Table 1. Top-1 accuracy on 10 classification datasets. We report the results for five baselines and our method. The 'Average' column shows the average results across all datasets. **Best**/2<sup>nd</sup> best.</caption><thead><tr><th>Method</th><th>Average</th><th>EuroSAT</th><th>Sun397</th><th>Food101</th><th>Flower102</th><th>DTD</th><th>FGVC</th><th>Oxford Pets</th><th>Cars</th><th>UCF101</th><th>Caltech101</th></tr></thead><tbody><tr><td>Oracle</td><td>81.76</td><td>94.46</td><td>77.45</td><td>85.01</td><td>87.90</td><td>76.65</td><td>37.95</td><td>92.42</td><td>90.21</td><td>79.49</td><td>96.02</td></tr><tr><td>LLAVA zero-shot</td><td>27.23</td><td>44.78</td><td>15.74</td><td>29.81</td><td>6.58</td><td>20.27</td><td>3.18</td><td>28.92</td><td>3.38</td><td>44.25</td><td>75.38</td></tr><tr><td>Pre-trained CLIP</td><td>67.49</td><td>42.95</td><td>68.20</td><td><u>78.65</u></td><td><u>71.30</u></td><td>55.32</td><td><u>23.79</u></td><td>87.30</td><td><b>88.25</b></td><td>64.37</td><td><u>94.73</u></td></tr><tr><td>ReCLIP [20]</td><td>68.78</td><td>49.25</td><td>69.07</td><td>77.91</td><td>71.13</td><td><u>56.91</u></td><td><b>25.92</b></td><td>88.50</td><td><b>87.84</b></td><td>68.86</td><td>92.37</td></tr><tr><td>FLYP [15] + pseudo-label [29]</td><td><u>70.01</u></td><td><u>67.12</u></td><td><u>70.19</u></td><td>76.83</td><td>68.78</td><td><b>61.82</b></td><td>17.40</td><td><b>88.96</b></td><td>84.19</td><td><u>69.44</u></td><td>94.69</td></tr><tr><td>LATTECLIP (ours)</td><td><b>72.23</b></td><td><b>80.27</b></td><td><b>70.68</b></td><td><b>79.63</b></td><td><b>71.94</b></td><td>56.26</td><td>22.02</td><td><b>89.21</b></td><td>87.40</td><td><b>70.08</b></td><td><b>94.77</b></td></tr></tbody></table>

of the text-prototype embeddings assigned to the class. This process is repeat for each pseudo-label in {$c_{zs}, c_{ft}$}.

**Inference.** The predictions for an image $x$ are made by comparing the image embedding $f(x)$, where $f(\cdot)$ is the fine-tuned CLIP image encoder, with {$p_c$}$_{c=1}^C$ and taking the prototype with the highest cosine similarity as output.

# 4. Experiments

We evaluate LATTECLIP on the task of fine-tuning on 10 specialized classification datasets, without using any ground truth labels. We use the training set for unsupervised training and use the test set to compute the top-1 accuracy.

**Datasets.** We employ a mixture of datasets covering various specialized domains, including satellite imagery, food dishes, airplane models, and others: EuroSAT [18], SUN397 [61], Food101 [5], DTD [6], FGVC [37], Oxford Pets [43], Cars [25], UCF101 [52], Caltech101 [11], Flower102 [40]. These datasets feature specific classes, such as the car model, making the unsupervised fine-tuning setup challenging. We follow the standard train/val/test splits in [68]. We train LATTECLIP using the combined train and val sets and report its performance on the test set.

**Baselines.** We compare our method to four unsupervised baselines and one fully supervised baseline, which serves as an oracle. First, we perform zero-shot classification with a pre-trained CLIP model. As in CLIP [46], we compute text embeddings for all classes with template "a photo of a [class].". For classification, we compute the cosine similarity between each image and all class text embeddings. Our second baseline, ReCLIP [20], also performs fine-tuning without labels but utilizes improved pseudo labels and self-training. However, ReCLIP primarily focuses on experiments conducted in a transductive manner, which involves training and evaluating on the test split of each dataset. To ensure a fair comparison, we retrained ReCLIP using the same CLIP-based model and identical dataset splits as our method. Third, we combined FLYP [15] with pseudo-labeling [29] for unsupervised fine-tuning, as the original method relies on supervised fine-tuning. Note that we use FLYP without weight ensembling to

maintain a fair comparison with ReCLIP, which also does not employ weight ensembling. Finally, we add "LLAVA zero-shot" baseline which prompts LLAVA to classify the image from a given list of classes, using the following prompt "Select the most appropriate category for the image from the following options: [options]. Write only the category name." , where options is replaced with the list of class names. For the supervised baseline, we train FLYP using ground-truth labels, serving as an oracle. The evaluation is performed in a zero-shot fashion, like zero-shot CLIP. Since our method uses prototypes, no class template embeddings have to be computed, and we directly use the prototype vectors. For all baselines and ours, we use OpenCLIP [22], the open-source implementation of CLIP [46], with a ViT/B-32 architecture, pre-trained on the LAION-2B dataset. Performance is reported based on the last epoch since we have no supervision signal. Additional implementation details are in Appendix B.

## 4.1. Results

The main results with top-1 accuracy on the 10 datasets are shown in Tab. 1. Across all datasets, LATTECLIP improves the average top-1 accuracy of CLIP by 4.74 points. Furthermore, it outperforms all unsupervised baselines, including the recently published ReCLIP [20] and our proposed baseline that integrates FLYP [15] with pseudo-labeling [29], by 3.45 and 2.22 points, respectively. Interestingly, FLYP + pseudo-label outperforms ReCLIP, likely due to the robustness and effectiveness of fine-tuning both image and text encoders with contrastive loss, instead of just the image encoder with cross-entropy loss, as demonstrated in FLYP [15]. Notably, LLAVA zero-shot has low overall performance, which could be attributed to LLAVA being trained in generative autogressive manner, thus not optimal for discriminative tasks. Lastly, the oracle is shown on the first line by training FLYP with ground-truth labels. The 9.53-point average performance gap between the fully supervised oracle and unsupervised LATTECLIP highlights room for improvement. Still, LATTECLIP performs competitively, narrowing the gap across multiple datasets, particularly on Oxford Pets, Cars, and Caltech101, to less than 3%.
<table><caption>Table 2. **Impact of generated texts.** Best performance is achieved when using all types of descriptions. **Best**/2nd best.</caption><thead><tr><th></th><th>T<sup>class</sup></th><th>T<sup>image</sup></th><th>T<sup>group</sup></th><th>Average</th><th>EuroSAT</th><th>Sun397</th><th>Food101</th><th>Flower102</th><th>DTD</th><th>FGVC</th><th>Oxford Pets</th><th>Cars</th><th>UCF101</th><th>Caltech101</th></tr></thead><tbody><tr><td>1</td><td>✓</td><td>✓</td><td>✓</td><td><strong>72.23</strong></td><td><strong>80.27</strong></td><td><strong>70.68</strong></td><td>79.63</td><td>71.94</td><td>56.26</td><td><strong>22.02</strong></td><td>89.21</td><td><strong>87.40</strong></td><td><strong>70.08</strong></td><td>94.77</td></tr><tr><td>2</td><td>✓</td><td>✓</td><td></td><td>70.74</td><td><u>79.98</u></td><td>64.85</td><td>75.52</td><td><strong>72.31</strong></td><td><u>57.03</u></td><td>16.44</td><td>89.45</td><td>87.09</td><td>69.50</td><td><strong>95.21</strong></td></tr><tr><td>3</td><td>✓</td><td></td><td></td><td>70.67</td><td>78.22</td><td>59.79</td><td><strong>81.52</strong></td><td>71.21</td><td>56.74</td><td>16.50</td><td><strong>90.00</strong></td><td><strong>87.89</strong></td><td>69.84</td><td>94.97</td></tr><tr><td>4</td><td></td><td>✓</td><td>✓</td><td>55.97</td><td>64.75</td><td>63.28</td><td>76.73</td><td>50.95</td><td>48.76</td><td>9.00</td><td>64.51</td><td>32.98</td><td>60.45</td><td>88.24</td></tr><tr><td>5</td><td></td><td>✓</td><td></td><td>52.37</td><td>44.31</td><td>62.54</td><td>77.00</td><td>48.44</td><td>43.09</td><td>7.89</td><td>56.75</td><td>33.86</td><td>58.58</td><td>91.24</td></tr><tr><td>6</td><td></td><td></td><td>✓</td><td>53.52</td><td>59.35</td><td>65.00</td><td>77.06</td><td>31.67</td><td>49.05</td><td>9.57</td><td>64.49</td><td>25.99</td><td>66.90</td><td>86.09</td></tr><tr><td>7</td><td>✓</td><td></td><td>✓</td><td><u>71.63</u></td><td>79.68</td><td><u>70.07</u></td><td>80.16</td><td><u>71.99</u></td><td><strong>57.47</strong></td><td><u>18.00</u></td><td><strong>89.81</strong></td><td>86.15</td><td>68.84</td><td>94.12</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Figure 5. **Examples of generated captions.** We either generate a caption from the group of 4 images, by inputting them as tiled single image into LLaVA ($T^{group}$), or we input a single image to LLaVA ($T^{image}$). For simplicity, in this figure, we only show a single image caption (highlighted by red bounding box).</figcaption></figure>

In Fig. 5, we show examples of generated image-description $T^{image}$ and group-description $T^{group}$. Overall, $T^{group}$ offers more comprehensive and contextual information. For instance, in the top-right example, $T^{image}$ is simply "striped pattern," whereas $T^{group}$ provides richer details, including "zebra, animal print, geometric shape, lines". This trend is evident in other examples as well. For example, in the bottom-right example, $T^{image}$ is "Girl applying makeup," while $T^{group}$ elaborates with "Makeup application, close-up, hands holding tools." Additionally, image-description fails to capture "forest" in the top-left example, describing it merely as "The image shows a large body of water with no visible land use." In contrast, $T^{group}$ includes relevant details such as "vegetation, land, green and brown colors."

## 4.2. Ablations

**Different types of synthetic descriptions.** Tab. 2 illustrates the impact of different generated texts on overall performance. We observe that all texts are essential for achieving the best performance. Specifically, excluding the *image-description* reduces the average performance across all datasets by 0.6 (row 1 vs. row 7). The impact of removing the *group-description* is even more significant with a 1.49 points reduction (row 1 vs. row 2). Additionally, omitting both the *image-description* and *group-description* results in an even larger loss of 1.56 points (comparing row 1 to row 3). Rows 4, 5, and 6 show that relying solely on synthetic texts causes a drop in performance due to the noise and inaccuracies introduced by the generated descriptions.

**Dynamic Feature Mixer.** We ablate our Dynamic Feature Mixer in Tab. 3 (row "w/o Dynamic Feature Mixer") by setting all the text weights to 1.0, so that all texts contribute equally. The average performance drops by 2 points, with significant decreases on multiple datasets, such as -14.23 on EuroSAT, -1.41 on DTD, and -2.03 on Cars. This demonstrates that our Dynamic Feature Mixer module effectively assigns relevant weights to the meaningful descriptions.

**Dual Pseudo-Labels.** Best performance is achieved using both zero-shot and fine-tuning pseudo-labels {$c_{zs}, c_{ft}$}. This is assessed in Tab. 3 by removing the corresponding losses. Removing the zero-shot pseudo-label (row "w/o $L_{zs}$") leads
<table><caption>Table 3. **Method ablation.** All components contribute to the best performance. **Best**/2<sup>nd</sup> best.</caption><thead><tr><th>Method</th><th>Average</th><th>EuroSAT</th><th>Sun397</th><th>Food101</th><th>Flower102</th><th>DTD</th><th>FGVC</th><th>Oxford Pets</th><th>Cars</th><th>UCF101</th><th>Caltech101</th></tr></thead><tbody><tr><td>LATTECLIP (ours)</td><td><b>72.23</b></td><td><b>80.27</b></td><td><b>70.68</b></td><td><b>79.63</b></td><td><b>71.94</b></td><td>56.26</td><td>22.02</td><td><b>89.21</b></td><td><b>87.40</b></td><td><b>70.08</b></td><td><b>94.77</b></td></tr><tr><td>w/o Dynamic Feature Mixer</td><td>70.23</td><td>66.04</td><td>69.41</td><td><b>80.18</b></td><td><b>72.51</b></td><td>54.85</td><td><u>23.43</u></td><td>87.14</td><td>85.37</td><td>69.02</td><td>94.32</td></tr><tr><td>w/o L<sub>ft</sub></td><td>68.26</td><td>47.06</td><td>69.80</td><td>79.23</td><td>70.48</td><td><u>56.97</u></td><td><b>23.82</b></td><td><b>87.65</b></td><td><b>87.19</b></td><td>65.77</td><td><b>94.60</b></td></tr><tr><td>w/o L<sub>zs</sub></td><td><u>70.58</u></td><td><u>76.96</u></td><td>68.18</td><td>70.29</td><td>71.01</td><td><b>61.05</b></td><td>19.89</td><td>87.49</td><td>86.13</td><td><b>70.39</b></td><td>94.36</td></tr><tr><td>w/o Momentum Update</td><td>45.72</td><td>31.19</td><td>56.17</td><td>68.08</td><td>57.41</td><td>31.32</td><td>13.41</td><td>13.03</td><td>43.31</td><td>54.69</td><td>88.56</td></tr></tbody></table>

to a significant drop across multiple datasets: -3.31 on EuroSAT, -2.5 on SUN397, -9.34 on Food101, and an average decline of -1.65 across all datasets. Furthermore, removing the fine-tuned pseudo-labels (row "w/o L<sub>ft</sub>") results in an even more substantial average performance drop of -3.97, with particularly notable decreases of -33.21 on EuroSAT and -4.31 on UCF101. We conjecture that this is because the zero-shot pseudo-label is more robust, while the fine-tuned pseudo-label has higher accuracy on the training dataset.

**Momentum Update.** We ablate the impact of the momentum update by setting $\mu=0$, as shown in row "w/o Momentum Update" in Tab. 3, therefore directly replacing the prototype by the new weighted text features. Without momentum update, performance declines dramatically, with an average decrease of -26.51 across all datasets. Significant declines are observed in many datasets, such as -44.09 on Cars, -14.51 on SUN397, and -14.53 on Flower102. This substantial drop is attributed to the high variance in the prototypes due to the noisy generated texts.

**Incorrect images in generating T<sup>group</sup>.** We analyze how incorrect images within a group affect the generated group description T<sup>group</sup>. We test groups of 4 images with different number of correct images, selected using ground-truth labels. For 1, 2, 3 and 4 correct images, this results in top-1 accuracy of 72.48, 72.61, 72.72, and 72.64, respectively, averaged across all datasets. Performance improves slightly with more correct images. Notably, our method using pseudo-labels achieves a performance of 72.23, which is competitive with the ground-truth label selection. This demonstrates tolerance to noise and pseudo-label inaccuracies within the image group. Detailed performance is provided in Tab. 4.

**Number of images per group.** We analyze the impact of increasing the number of images per group on performance by testing groups with 2, 4, 8, and 16 images, resulting in average performance scores of 71.55, 72.23, 72.31, and 72.49, respectively, across all datasets. The performance improves with more images per group, likely because the probability of including the correct images increases. The most notable improvement occurs when increasing the number of images per group from 2 to 4, with score rising from 71.55 to 72.23. This is because achieving a majority with only 2 images requires 100% accuracy, whereas larger groups can tolerate some errors while still maintaining a correct majority. Performance plateaus after 4 images, possibly due to the fixed input resolution of LLAVA [34], leading to lower per-image resolution as the number of images increases. We provide the performance for all datasets in Tab. 5.

<figure><img src="image_7.png" /><figcaption>Figure 6. **Impact of amount of training data.** Average top-1 accuracy on 10 datasets while varying the amount of training data.</figcaption></figure>

**Impact of amount of training data.** Sec. 4.2 illustrates the effect of training data size on the average top-1 accuracy of LATTECLIP and oracle across 10 datasets. Despite being unsupervised, LATTECLIP exhibits strong robustness to varying amounts of training data, comparable to an oracle. Specifically, LATTECLIP's performance drops only 0.77/6.36 on 20%/1% data, respectively, compared to 7.28/1.30 of the oracle. Overall, more data improves performance but diminishes notably for both after 20%.

## 5. Conclusion

LATTECLIP is a novel method for unsupervised CLIP fine-tuning on specialized datasets where human annotations are costly or require expert knowledge. Leveraging LMMs, LATTECLIP generates rich and expressive synthetic textual descriptions at various levels of contextual granularity, including *image-description*, *group-description*, and *class-description*. To effectively learn from these potentially noisy descriptions, we propose a prototype learning framework with three key elements: (1) dual pseudo-labels from frozen and fine-tuning CLIP models; (2) a Dynamic Feature Mixer for optimal text feature weighting; and (3) momentum update to enhance training stability. LATTECLIP surpasses comparable baselines on average across all datasets.
<table><caption>Table 4. Impact of varying the number of correctly chosen images based on ground-truth labels when using 4 images for group-description generation. Our approach yields comparable performance despite relying solely on pseudo-labels for image selection.</caption><thead><tr><th>label type</th><th>#correct</th><th>Avg.</th><th>EuroSAT</th><th>Sun397</th><th>Food101</th><th>Flower102</th><th>DTD</th><th>FGVC</th><th>Oxford Pets</th><th>Cars</th><th>UCF101</th><th>Caltech101</th></tr></thead><tbody><tr><td>Pseudo (ours)</td><td>N/A</td><td>72.23</td><td>80.27</td><td><strong>70.68</strong></td><td><u>79.63</u></td><td>71.94</td><td>56.26</td><td><strong>22.02</strong></td><td>89.21</td><td>87.40</td><td><u>70.08</u></td><td>94.77</td></tr><tr><td rowspan="4">Ground-truth</td><td>1</td><td>72.48</td><td>80.02</td><td>69.19</td><td>79.04</td><td><u>72.88</u></td><td><strong>61.11</strong></td><td>20.55</td><td><strong>89.62</strong></td><td>87.35</td><td><strong>70.16</strong></td><td>94.89</td></tr><tr><td>2</td><td>72.61</td><td><strong>81.28</strong></td><td>69.73</td><td>79.13</td><td>72.55</td><td><u>60.82</u></td><td>20.76</td><td><u>89.51</u></td><td><strong>87.53</strong></td><td>69.65</td><td><u>95.13</u></td></tr><tr><td>3</td><td><strong>72.72</strong></td><td><u>80.81</u></td><td>70.28</td><td><strong>79.80</strong></td><td>72.72</td><td>60.28</td><td><u>21.87</u></td><td>89.48</td><td>87.29</td><td>69.52</td><td><strong>95.17</strong></td></tr><tr><td>4</td><td><u>72.64</u></td><td>80.40</td><td><strong>70.54</strong></td><td>78.79</td><td><strong>72.96</strong></td><td>60.17</td><td>21.42</td><td><strong>89.62</strong></td><td><strong>87.96</strong></td><td>70.00</td><td>94.56</td></tr></tbody></table>

<table><caption>Table 5. Number of images per group impact on generating group-descriptions. Overall, more images improve performance due to richer information and increased robustness against the inclusion of incorrect images. However, the performance plateaus on some datasets, e.g., UCF101 or SUN397, could be attributed to LLAVA's fixed resolution, resulting in lower resolution per image when using more images.</caption><thead><tr><th>#Images</th><th>Average</th><th>EuroSAT</th><th>Sun397</th><th>Food101</th><th>Flower102</th><th>DTD</th><th>FGVC</th><th>Oxford Pets</th><th>Cars</th><th>UCF101</th><th>Caltech101</th></tr></thead><tbody><tr><td>2</td><td>71.55</td><td><strong>80.74</strong></td><td>69.36</td><td>76.03</td><td>71.24</td><td>56.03</td><td>21.12</td><td><u>89.29</u></td><td>87.32</td><td>69.88</td><td><u>94.52</u></td></tr><tr><td>4</td><td>72.23</td><td>80.27</td><td><strong>70.68</strong></td><td><u>79.63</u></td><td>71.94</td><td>56.26</td><td><u>22.02</u></td><td>89.21</td><td>87.40</td><td><strong>70.08</strong></td><td><strong>94.77</strong></td></tr><tr><td>8</td><td><u>72.31</u></td><td>79.90</td><td>69.90</td><td><u>79.55</u></td><td><u>73.04</u></td><td><u>57.69</u></td><td>22.00</td><td>89.18</td><td><strong>87.65</strong></td><td>69.71</td><td>94.44</td></tr><tr><td>16</td><td><strong>72.49</strong></td><td><u>80.67</u></td><td><u>70.18</u></td><td>78.24</td><td><strong>73.20</strong></td><td><strong>58.64</strong></td><td><strong>22.28</strong></td><td><strong>89.53</strong></td><td><strong>87.85</strong></td><td><u>70.05</u></td><td>94.28</td></tr></tbody></table>

**Acknowledgment.** The research was conducted during Quan's internship at Amazon. The research was also supported by the ANR project SIGHT (ANR-20-CE23-0016) and SAMBA collaborative project co-funded by BpiFrance in the Investissement d'Avenir Program. Computation was performed using HPC resources from GENCI-IDRIS (AD011012808R2, AD011014102R1). We thank Ajanthan Thalaiyasingam and Mohammad Fahes for their insightful suggestions. We also extend our gratitude to Mohammad Fahes and Ivan Lopes for their thorough proofreading.

# Appendices

## A. Limitations

Despite promising results, LATTECLIP considers a limited number of description types. Expanding description generation to include more contextual levels, such as scenes, objects, and attributes, would provide richer contextual information. Additionally, our performance is constrained by the underlying LMM model, and improvements could be made with better models in the future. Lastly, it is unclear why the method improves on some datasets but not others. Understanding this discrepancy could lead to better methods.

## B. Implementation details

We implement LATTECLIP based on the standard fine-tuning pipeline of OpenCLIP [22] using the VIT-B/32 model. The hyperparameters used are the default ones provided in OpenCLIP [22], except for batch size and learning rate.

We use a batch size of 512 and a learning rate of 1e-7 for the datasets Caltech101 [11], DTD [6], Eurosat [18], FGVC [37], Oxford Pets [43], Cars [25], Flower102 [40], and UCF101 [52]. For the datasets Food101 [5] and SUN397 [61], we use a learning rate of 1e-6. LATTECLIP is trained for min{2000 iterations, 50 epochs}.

For FLYP [15], we reimplement it based on its official implementation<fnref n="3" /> and OpenCLIP [22], as its idea is intuitive and simple: fine-tuning using contrastive loss with class templates instead of cross-entropy loss. We use the same OpenCLIP-based model and training hyperparameters as LATTECLIP. The pseudo-labels are recalculated after every weight update, following [29].

For ReCLIP [20], we use the official implementation<fnref n="4" />, but substitute OpenCLIP as the base CLIP model to ensure a fair comparison across all methods. While ReCLIP is designed for transductive learning (train/test on test set), as shown in the paper and by its official implementation, we adapt it to our experimental setup. Specifically, we retrain and evaluate ReCLIP using identical dataset splits as LATTECLIP.

## C. Additional ablations

**Incorrect images in generating *T*<sup>group</sup>.** Tab. 4 presents the results across all datasets when varying the number of correct images, which are selected using ground-truth labels, in groups of 4 images used for generating *group-descriptions*. Using more correct images generally leads to improvements in most datasets. However, the average performance gap remains small, demonstrating the robustness of our method

<footnote n="3">³https://github.com/locuslab/FLYP</footnote>

<footnote n="4">⁴https://github.com/michiganleon/ReCLIP_WACV</footnote>
<table><caption>Figure 7. Examples of image-description $T^{image}$ generated from image $x$ and group-description $T^{group}$ generated from image group $x^{group}$, and two types of pseudo-labels: zero-shot $c_{zs}$ and fine-tuning $c_{ft}$. The class-description is generated by substituting the pseudo-label $c \in \{c_{zs}, c_{ft}\}$ into a predefined template: "a photo of a [c].".</caption><thead><tr><th>x</th><th>T<sup>image</sup></th><th>x<sup>group</sup></th><th>T<sup>group</sup></th><th>pseudo-labels</th><th>GT</th></tr></thead><tbody><tr><td></td><td>Buildings and green spaces.</td><td></td><td>Green, brown, and blue colors, indicating vegetation, soil, and water.</td><td>c<sub>zs</sub>: permanent crop land, c<sub>ft</sub>: river</td><td>river (Eurosat)</td></tr><tr><td></td><td>The texture in the photo is a wooden floor with a herringbone pattern.</td><td></td><td>Zigzag patterns, geometric shapes, and vibrant colors.</td><td>c<sub>zs</sub>: zigzagged, c<sub>ft</sub>: grooved</td><td>zigzagged (DTD)</td></tr><tr><td></td><td>The pink primrose flower in the photo is a beautiful and vibrant display of nature's beauty.</td><td></td><td>Purple and yellow petals, green stems, multiple layers of petals.</td><td>c<sub>zs</sub>: pink primrose, c<sub>ft</sub>: silverbush</td><td>garden phlox (Flower102)</td></tr><tr><td></td><td>Woman in white shirt holding blue shoe.</td><td></td><td>Shoes, women, shopping, retail, store, display, merchandise, fashion, sales, shopping center, mall, department store, commercial, consumer.</td><td>c<sub>zs</sub>: shoe shop, c<sub>ft</sub>: shoe shop</td><td>shoe shop (SUN397)</td></tr><tr><td></td><td>Person on trampoline.</td><td></td><td>Gymnastics, acrobatics, high jumps, flips, and aerial stunts.</td><td>c<sub>zs</sub>: uneven bars, c<sub>ft</sub>: parallel bars</td><td>parallel bars (UCF101)</td></tr></tbody></table>

to the presence of incorrect images in the group. This robustness is further evidenced by the performance of LATTECLIP, which remains competitive even when relying on pseudo-labels for image selection instead of ground-truth labels.

**Number of images per group.** Tab. 5 analyzes the performance as the number of images per group used for generating *group-description* increases. Generally, more images per group lead to higher performance on most datasets. This is intuitive, as more images provide richer information and a higher chance of including correct images. Using only two images results in the worst performance because selecting a wrong image would significantly impact the outcome, making 50% or 100% of the selected images incorrect. Consequently, larger groups are more robust to the inclusion of wrong images. As LLAVA [34] has a fixed resolution,

adding more images results in lower resolution per image. This could explain the performance plateau on datasets with more image details, such as UCF101 or SUN397.

## D. Additional results

**Examples of LMM-synthetic texts and pseudo-labels.** Fig. 7 illustrates examples of image-description $T^{image}$ and group-description $T^{group}$ generated from individual images $x$ and image groups $x^{group}$, respectively. The figure also presents ground-truth labels (GT) along with pseudo-labels derived from the frozen CLIP model ($c_{zs}$) and the fine-tuning model ($c_{ft}$). Note that the class-description is generated by substituting the pseudo-label $c \in \{c_{zs}, c_{ft}\}$ into a predefined template: "a photo of a [c].". Combining both types of pseudo-labels increases the chance of capturing
the ground-truth label, as each type of pseudo-label is correct for different examples. For instance, $c_{zs}$ is correct for rows 2, 3, and 4, while $c_{ft}$ is correct for rows 1 and 4. Regarding the synthetic description, $T^{group}$ provides richer contextual information, particularly in rows 1, 2, 4, and 5, and contains less hallucinated information compared to $T^{image}$, as seen in rows 2 and 3, with greater accuracy in rows 1, 4, and 5.

## References

[1] Agnar Aamodt and Enric Plaza. Case-based reasoning: Foundational issues, methodological variations, and system approaches. *AI communications*, 1994. 2

[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. *arXiv*, 2023. 3

[3] AI Anthropic. The clade 3 model family: Opus, sonnet, haiku. *Claude-3 Model Card*, 2024. 1

[4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. 1

[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In *ECCV*, 2014. 6, 9

[6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In *CVPR*, 2014. 6, 9

[7] Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Poda: Prompt-driven zero-shot domain adaptation. In *ICCV*, 2023. 1

[8] Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. A simple recipe for language-guided domain generalized segmentation. In *CVPR*, 2024. 1

[9] Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Fine-tuning clip's last visual projector: A few-shot cornucopia. *arXiv*, 2024. 2

[10] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In *NeurIPS*, 2023. 2, 3

[11] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. *CVPRW*, 2004. 6, 9

[12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. *JMLR*, 2016. 1

[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. *IJCV*, 2024. 2

[14] Muhammad Waleed Gondal, Jochen Gast, Inigo Alonso Ruiz, Richard Droste, Tommaso Macri, Suren Kumar, and Luitpold Staudigl. Domain aligned clip for few-shot classification. In *WACV*, 2024. 2

[15] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In *CVPR*, 2023. 1, 2, 3, 5, 6, 9

[16] Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? *arXiv*, 2024. 2, 3

[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In *CVPR*, 2020. 2, 5

[18] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. *JSTAR*, 2019. 6, 9

[19] Lukas Hoyer, Dengxin Dai, Haoran Wang, and Luc Van Gool. Mic: Masked image consistency for context-enhanced domain adaptation. In *CVPR*, 2023. 1

[20] Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, et al. Reclip: Refine contrastive language image pre-training with source free domain adaptation. In *WACV*, 2024. 2, 3, 6, 9

[21] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. In *NeurIPS*, 2021. 2

[22] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, jul 2021. 6, 9

[23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In *ICML*, 2021. 1

[24] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. 1

[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In *ICCVW*, 2013. 6, 9

[26] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In *ICML*, 2022. 2, 3
[27] Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. In *ICML*, 2022. 2
[28] Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Alekse Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions. In *ECCV*, 2024. 2, 3
[29] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In *ICMLW*, 2013. 2, 3, 6, 9
[30] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In *NeurIPS*, 2021. 1
[31] Jingjing Li, Zhiqi Yu, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source-free domain adaptation. *TPAMI*, 2024. 2
[32] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In *CVPR*, 2020. 2
[33] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In *ICML*, 2020. 2
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In *NeurIPS*, 2023. 1, 3, 4, 8, 10
[35] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? In *NeurIPS*, 2021. 3
[36] Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms-augmented visual-language representation learning. *arXiv*, 2023. 4
[37] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. *arXiv*, 2013. 6, 9
[38] Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot clip. In *CVPR*, 2024. 2
[39] Allen Newell, Herbert Alexander Simon, et al. *Human problem solving*, volume 104. Prentice-hall Englewood Cliffs, NJ, 1972. 2
[40] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In *Indian conference on computer vision, graphics & image processing*, 2008. 6, 9
[41] OpenAI. Gpt-4 technical report, 2024. 1
[42] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In *NeurIPS*, 2022. 3
[43] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In *CVPR*, 2012. 6, 9

[44] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In *ICCV*, 2021. 1
[45] Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with clip. In *NeurIPS*, 2023. 2
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *ICML*, 2021. 1, 2, 5, 6
[47] Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In *ICCV*, 2023. 1
[48] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. 1, 3
[49] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In *NeurIPS*, 2022. 3
[50] Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In *CVPR*, 2024. 1, 2
[51] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In *NeurIPS*, 2017. 2
[52] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. *arXiv*, 2012. 6, 9
[53] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In *ICML*, 2020. 3
[54] Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jun-dong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. *arXiv*, 2024. 3
[55] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. 1
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 1
[57] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In *ICLR*, 2021. 3
[58] Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In *CVPR*, 2021. 1
[59] Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine-tuning performance. In *ICCV*, 2023. 2
[60] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gongtijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In *CVPR*, 2022. 1, 2, 3, 4
[61] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In *CVPR*, 2010. 6, 9
[62] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In *ICLR*, 2022. 2
[63] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. *TMLR*, 2022. 2
[64] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In *CVPR*, 2024. 2
[65] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In *ICCV*, 2023. 2
[66] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In *ECCV*, 2022. 2
[67] Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with clip reward for zero-shot generalization in vision-language models. In *ICLR*, 2024. 3
[68] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. *IJCV*, 2022. 1, 2, 6
[69] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In *ICCV*, 2023. 2