Language-Aware Information Maximization for Transductive Few-Shot CLIP
Authors: Ghassen Baklouti
Maxime Zanella
Ismail Ben Ayed
Venue: Accepted by TMLR ()

Abstract: Transductive few-shot learning has triggered an abundant literature focusing on vision-only models, but is still at a nascent stage within the recent context of foundational vision-language models (VLMs). Only a few recent methods addressed the problem, pointing to the potential of tranduction in VLMs and to the need for VLM-tailored methods. Building on this momentum, we leverage information-theoretic concepts and recent progress in parameter-efficient fine-tuning (PEFT), developing a highly competitive transductive few-shot CLIP method. Specifically, we introduce a novel Language-aware Information MaximizatiOn (LIMO) loss integrating three complementary terms: (i) the mutual information between the vision inputs and the textual class descriptions; (ii) a Kullback-Leibler (KL) divergence penalizing deviation of the network's probabilistic outputs from the text-driven zero-shot predictions; and (iii) a standard cross-entropy loss based on the labeled shots. Furthermore, we challenge the commonly followed fine-tuning practices in the context of transductive few-shot learning, and explore PEFT strategies, completely overlooked in this context. Surprisingly, we observe substantial boosts in performances, which points to the potential of adapting a subset of the model's parameters in the transductive few-shot setting. We report comprehensive evaluations, which show that LIMO outperforms the very recent transductive few-shot CLIP methods by a large margin and yields significant gains over the best-performing inductive methods. We will publicly release our code.

Language-Aware Information Maximization for Transductive Few-Shot CLIP
Ghassen Baklouti
ghassen. boklouti. etseess..tsmtt.cc
École de Technologie Supérieure (ETS), Montréal, Canada
Maxime Zanella
maciime.zameella@@uclleoeeee
Université Catholique de Louvain (UCLouvain) Lowwain-La-Neure, Belgium Université de Mons (UMons), Mons, Belgium
Ismail Ben Ayed
ismail. ismail bet.yeedeteeedettt.lcoom
École de Technologie Supéricure (ETS), Montréal, Canada
Abstract
Transductive few-shot learning has triggered an abundant literature focusing on vision-only models, but is still at nascent stage within the recent context of foundational vision- language models (VLMs). Only few recent methods addressed the problem, pointing to the potential of tranduction in VLMs and to the need for VLM-tailored methods. Build- ing on this momentum, we leverage parametor-efficien fine-tuning (PEFT), developing a highly competitive transductive few- shot CLIP method. Specifically, we introduce novel Language-aware Information Max- imizatiOn (LIMO) loss integrating three complementary terms: (i) the mutual informa- tion between the vision inputs and the textual class descriptions; (ii) a Kullback-Leii Leibler (KL) divergence penalizing deviation of the network's probabilistic outputs from the text- driven zero-shot predictions; and (iii) standard cross-entropy loss based the labeled shots. Furthermore, we challenge the commonly followed fine-tuming practices in the con- text of transductive fow-shot lcarning, and explore PEFT strategics, completely overlooked in this context. Surprisingly, we observe substantial boosts in performances, which points to the potential of adapting a subset of the model's parameters the transductive few- shot setting. We report comprehensive evaluations, which show that LIMO outperforms the very recent transductive few-shot CLIP methods by large margin and yields signifi- cant gains over the best-performing inductive methods. Our code is publicly available at:
cntemaation-hheoree
concepts and recent progress in
1 Introduction
The recent rapid vdnaceeentss in hardware capabilities have driven substantial interest in multi-modal learn- ing within the field of computer vision. In particular, vision-language models (VLMs) have received notable attention, evidenced by a sharp rise in related publications in recent years (Zhang et al. 2024). Leading this wave, CLIP (Radford et al. 2021) is a pioneering VL. that leverages natural language supervision, learning visual representations from a vast, task-agnostic dataset 400 million imago-text pairs. The pro-training process of CLIP involves jointly optimizing both image and text encoders in contrastive fashion. This approach aligns paired images and text prompts in shared embedding space, forcing the matched textual and visual representations to be close, while pushing non-matched pairs apart. Consequently, CLIP can per- form robust zero-shot classification at the inference stage, where embeddings of test images are matched to the text-based prompts derived from the class names of the downstream task (e.g., "a photo of a [class name].")
VLMs have demonstrated strong zero-shot performances across a wide range of tasks, yet they could be se- riously challenged when dealing with fine-grained tasks that significantly different. from the pre-training

---

84.9
85
CLIP-LORA TransCLIP
(ours)
80.2
80
Top-1 Accuracy (%)
77.8
77.4
77.8
78.4
75.5
75.9
75
74.2
71.3
70
64.9
65
60 2-shot
4-shot
16-shot
Figure The reported performance is the average accuracy over the 11 datasets studied in this paper. LIMO outperforms a recent transductive VL. method (TransCLIP (Zanella et al. 2024g) and related vision-only method (TIM (Boudiaf al. 2020b)) by large margins. It also demonstrates significant gain when compared to related inductive method (CLIP-LoRA (Zanella & Ben Ayed 2024a)).
data (Radford et al. 2021 To address this, ongoing rescarch currently being focusod on adaptation methods, aiming enhancing the generalization of VLMs and bridging the gap toward reliable deployments in real-world downstream applications {Zhou et al. 2022b Zhang et al 2022 Gao et al. 2024 Predom- inantly, these methods operate within an inductive framework, conducting inference independently for each test sample and omitting the relationships between the testing samples and their statistics. While effective, this approach may limit the model to isolated predictions, potentially overlooking valuable patterns within the overall target data distribution. The transductive setting a powerful alternative, in which inference is performed jointly across the entire test datasct. This framework enables the model leverage the statistical properties of the unlabeled target data as whole, facilitating the discovery of shared structures as well latent patterns that missed when samples demonst.rated significant success in the context of vision-only models, with large body of works that ad-
processed independently. This paradigm has already
dressed transductive few-shot inference, e.g. [Liu et al.) 2018 Dhillon et al. 2019 Ziko et al.) 2020 Boudiaf
et al. 2020b Liu et al. 2020 Hu et al. 2021b), to cite a few. Currently, the transductive setting is gaining
traction in VLMs, with several very recent works emerging on the subject (Martin et al. 2024 Kalantidis
et al. 2024 Zanella et al. 2024 El Khoury et al. 2025).
These recent transductive few-shot methods for VLMs leverage the text-encoder knowledge, as form of supervision, in conjunction with an unsupervised learning term, taking the form of generative mixture model-based clustering (Martin et al. 2024 Zanella et al.
2024 One of the main driving motivations
behind these recent VLM-tailored developments is that the standard transductive few-shot inference methods perform poorly with VLMs (Martin et al. As for the question "what model parameters to fine-tune", all the very recent transductive few-shot CLIP methods to our knowledge, i.c., (Martin et al. 2024 Zanella et al. 2024). focused on operating on the output embedding space, without exploring the possibility of fine-tuning the internualrepreesntaaioon parameters. This is, indeed, in line with the de facto choice in the abundant vision-only few-shot literature (Chen et al, 2019 Boudiaf et al.] 2020b), in which fine-tuning focuses on the output layer, while freezing the remaining network-cncoder parameters. In fact, full fine-tuning, i.c., fine-tuning the whole network is widely avoided in low-shot regimes as it is prone to over-fitting, substantially degrading the performances (Chen et. al. 2019 Boudiaf et al. 2020b). However, in parallel, inductive few-shot learning methods for VLMs have shown impressive performance gains, building on the recent advances Parameter- Eetetttrr--fiiiintt--iiiiii TinoTTuingg (PEFT) strategies, which originated from and were popularized in NLP. This includes the seminal work of CoOp.
2024 Zanella et al. 2024), hence the need of dedicated techniques.
which pioneered prompt learning in VLMs (Zhou al. 2022b), well as adapters (Zhang et. al. 2022
2

---

and, more recently, Low-Rank Adaptation (LoRA) Zanella & Ben Ayed 2024a These recent inductive few-shot CLIP developments challenged this status quo in fine-tuning with low supervision, showing that updating the inner representations, or the imput text prompts, could be beneficial even with a few labeled concepts, together with these
samples. Building on this momentum, we leverage information-theoren advances in PEFT strategies to push further the potential of the transductive paradigm in the context of VLMs. Specifically, our contributions could be summarized as follows:
Main contributions.
We propose a novel transductive language-aware pntrmaatio-mmxiiizzatee loss. Our objective func- tion aims at maximizing the mutual information between the vision inputs and the textual class descriptions, while penalizing deviation of the network's 's sofumax oulputs from the text-driven sero- shot predictions. It could be viewed as a generalization of information coommtton-maxinization maximization losses widely
deployed in vision-only tasks.
We challenge the commonly followed fino-tuning practices in the context of transductive few-shot learning, and explore recent advances in PEFT strategies, completely overlooked in this context, Surprisingly, we observe substantial boosts in performance, which points to the great potential of adapting subset of the model parameters in the transductive few-shot setting.
In Figure [ our method, named LIMO (Language-aware Information MaximizatiOn) outperforms a recent transductive method for VLMs by a large margin, especially in the higher-shot regimes, and also demonstrates significant gains over one of the best-performing inductive few-shot CLIIP methods in the literature.
2 Related Work
Transductive Inference. In recent years, transductive inference has emerged as a highly effective ap- proach in few-shot learning, where the objective is to generalize to new tasks or classes with very limited data. In contrast to traditional inductive methods, which solely rely on labeled training data (support set), transductive approaches utilize both labeled support samples and unlabeled test data (query samples). This enables to adjust predictions in response to the test set distribution, enhancing adaptability to new tasks. A seminal work that introduced transduction in vision-based few-shot learning proposed to use a meta-learned graph to propagate labels from the labeled support set to the unlabeled query set (Liu et al. 2018). Building on this, a range of other methods have since been developed. For instance, LaplacianShot Ziko et al. 2020 exploits data structure via graph clustering, encouraging consistent class predictions among samples with similar features, while aligning each query sample with the closct support prototype; Transductive Fine- Tuning (TF) (Dhillon et al., 2019) incorporates entropy minimization on query samples as a regularization mechanism; TIM (Boudiaf al. 2020b) maximizes the mutual information between the query images and the labels in conjunction with pross-cntropy the support samples; BD-CSPN (Liu et al., 2020) refines class prototypes by addressing feature biases between the support and query sets, selectively updating pro- totypes based on the most confident query samples; and PT-MAP (Hu et al. 2021b) formulates optimal transport problem to align the distributions of the support and query samples.
Despite their large potential, previously discussed transductive methods have been demonstrated to experi- ence considerable performance declines when applied to VLMs (Zanella al., 2024 Martin et al., 2024 as they rely exclusively on the visual features. This has prompted the development of several recent transduc- tive approaches, designed to explicitly leverage the textual modality alongside image embeddings, feature that made possible by the advent of VLMs (Martin et al., 2024 Kalantidis et al. 2024 Zanella et al, 2024h. The Dirichlet-based method in (Martin et al.) (202)) used maximum likelihood estimation objective function, while penalizing the total number of predieted classes in a given batch. TransCLIP (Zanella el al., 2024) also pursued a maximum likelihood formulation, but with different statistical models, and added a Kulback-LLibler divergence term penalizing deviation of the network's probability outputs from the zero- shot predictions. The ZLaP method in (Kalantidis et al., 2024) propagates the zero-shot labels based on the similarities between pairs of instances, following classical label-propagation principles.
3

---

Parameter-Efficien Fine-Tuning Tuning. Full fine-tuning (FFT) is one of the go-to-choice methods for adapt- ing machine learning models to downstream tasks due to its excellent performance. This approach involves updating all the trainable parameters of a model during the adaptation process. However, for large-scale pre- trained models (often consisting of millions or even billions of paramcters), FFT becomes less desirable as it requires substantial ial computational resources, leading to significant costs in lerms of memory, time, and data requirements. To address this challenge, an alternative adaptation paradigm known as parameter-cfficien line-tuning (PEFT) has emerged. PEFT techniques aim to reduce the high expense associated with FFT by adjusting only small subset of the model's parameters. Based the choice of learnable parameters, PEFT methods can be grouped into four main categories: Addition-based methods, which introduce new trainable layers or adapters within the original frozen architecture to absorb task-specific knowledge (Zhang 2022 Gao et al.) 2024); Selective-based methods, which focus on updating a carefully-chosee subset of the model's existing parameters such as adjusting only the bias terms within certain blocks (Zaken et al. 2021) or fine tuning particular layers, like the last visual projector layer in CLIP (Fahes et al., 2024]; Prompt-tuning methods, which involve adding tunable tokens to the imput space or within intermediate sequences, enabling the model's outputs to be tailored for new tasks without altering its core parameters (Zhou et al. 2022b Jia et al.] 2022 Shu et al. 2022; Khattak et al, 2023 Ma et al., 2024p; Low-rank adaptation methods, which use additional low rank matrices that are adaptable to approximate the change of weights during adaptation while keeping the original matrices frozen. 2021a) pioneered this approach in the field of Natural Language Processing (NLP), and ever since, it has been adopted in vision tasks (Zanella Ben Ayed (0244a). Following LoRA, numerous
et al. 2022: Chen el al.) 2022; Lian et. al.)
LoRA (Hu et al.)
variations aiming to make the rank adaptive [Valipour et al.) 2022 Zhang al.
2023). improve performance 2024 Dettmers et al. 2024
(Chavan et al. 2023 Kim et al. 2024]. 1, or enhance efficiency (Rajabzadeh et al.
have emerged.
Zero- and Few-Shot CLIP Adaptation. With its extensive training on large-scale image-text data, CLIP has shown impressive performance across a variety of downstream tasks in zero-shot scenarios. How- ever, for more specific tasks, it may not achieve the desired results (Radford et al.) 2021). To tackle this issue, recent efforts have been directed towards exploring its full potential and further adapting it. Among those, prompt-tuning methods under limited supervision have become particularly prominent techniques
(Zhou et al. 2022b a Zhu et al., 2023 Yao et al. 2023; Shu et al. 2022: Ma et al. 2024).. CoOp (Zhou et al. 2022b introduces a context optimization approach in which imput text- pext-prompt tokens are represented as continuous learnable vectors that are end-to-end optimized from the data. Building upon this, CoCoOp (Zhou et al. 2022a integrates a meta-learning network to learn image-conditioned tokens, thereby improving model generalization to unseen classes. Other approaches, such as ProGrad (Zhu et al., 2023) and KgCoOp (Yao et al. 2023), focus aligning learned prompts with predefined handcrafted ones. ProGrad (Zhu et al, 2023), for instance, uses gradient projection to guide prompt-tuning, aiming to preserve the model's original knowledge during the adaptation process. Additionally, prompt-tune comppttuuning has been applied in test-time adap- tation settings (Shu et al., 2022 Ma et al. 2024), wherein imago-specific prompts are optimized in real time to maximize consistency across predictions generated from multiple augmentations of the same image.
Despite their effectiveness, prompt tuning methods remain computationally intensive, as they require multi- ple backpropagation passes through the entire text encoder. To mitigate these demands, several alternative approaches have been developed. For instance, MTA (Zanella & Ben Ayed 2024b) introduces novel test- time adaptation technique that optimizes visual features, providing an alternative to traditional test-time prompt tuning. Moreover, CLIP-adapter (Gao et al. (024) and Tip-adapter (Zhang et al. 2022) propose visual classifiers at the output of the vision encoder to combinc adapted and original features. LP++ (Huang el al., 2024) exlends the Linear-Probe baseline by incorporating knowledge from the text encoder. ProLIP (Fahes et al. 2024) limits updates to CLIP's last visual projector layer while CLIP-LoRA (Zanella) & Ben Ayed] (0244a) leverages low-rank matrices (LoRA) to adapt the attention matrices of both vision and text encoders at the same time.
Information Maximization. Information Maximization has been widely used in machine learning and computer vision tasks including representation learning (Tschannen et al. 2019 Hjelm et al. 2018 Bachman et al. 2019 Kemertas et al. 2020), deep clustering (Hu et al. 2017 Jabi et al.] 2019 Krause et al. 2010).
4

---

metric learning Boudiaf et al. 2020a), domain adaptation Pan al. 2020) semi-supervised learning CChia roni et al. 2023) and few-shot learning (Boudial et al. 2020b). This approach aims to maximize the mutual information between two sets of variables, leading to more informative and discriminative representations. While extensively studied for vision-only classifiers, this strategy, to the best of our knowledge, has not yet been applied to VLMs.
3 Method
In this section, we first introduce some basic notation and provide detailed presentation of the proposed transductive Language-aware Information MaximizatiOn (LIMO) loss and the fine-tuning strategy to opti- mize it.
3.1 CLIP Few-Shot Adaptation
Assume that we are given K candidate classes for a zero-shot classification task using a VLM, such as CLIP [Radford et al.) 2021). In such a setting, textual descriptions, often referred to as 'prompts', are created for each class, e.g. C₁
photo a [kth class name]. ". k = I,.., K. Fach prompt is then embedded into normalized representation on the unit hypersphere, ty f(c,, 0z), where is the language encoder and 0, its vector of learnable parameters. For given image classification dataset (or task), we have N images, X, i == L,..,,,, NN including both labeled and unlabeled ones. Let S [I....,N] denotes the set of sample indices within the set of labeled images in the data, often called the support set in the few-shot literature. Also, Q C (I,..., N contains the indices of the unlabeled samples (for which we need to make predictions), called the query set.
Each image x; in the data, i 1..,.,,, N, is mapped to normalized embodding space having the same dimensionality as the text-embeddingg space, using vision encoder g parameterized by O.. This yields, for each image X;, visual embedding f, = g(x,, 0z). For zero-shot prediction, a straightforward approach for applying VLMs to downstream classification tasks, each text embedding tx is paired with the visual embedding f of a test image x,, and their cosine similarity is computed to produce a prediction logit:
= f.Tt.
This logit is then converted into prediction in the form of posterior softmax probability of the class" textual description, ck. given test input x,:
exp(lin/T)
PLA Σx exp(lig/7)
(2)
where 7 is a temperature scaling parameter. Then, the zero-shot classification of an image x, performed by selecting the class with the highest posterior probability: à: == arg max; Pi,k-
In the standard few-shot setting, we assume that we have access to | labeled samples for each target class, with |-| denoting the cardinality of a set. Typically, [S|/K, the number of shots per class, is small (less than 16). Let zi denote the one-hot encoded label for labeled support image X;, with ziz- 1 if k is the class of image x; and 0 otherwise. In the inductive setting (Zanella & Ben Ayed 2024ah, model adaptation is done by minimizing the cross-entropy (CF. loss with respect to the parameters of both the text and vision encoders:
K
C(O., 02)
2u In
(3)
LES 1=1
3.2 Language-aware information maximization
Inspired by previous works on vision-only transductive few-shot learning, such as TIM (Boudial et al. we propose to minimize with respect to Og and 0₂ a generalization of the mutual-information loss to VLMs::
2020b
L(O,,, 0.) C(O,, 0c)
1(0,, 0c) + +T(,,,
Cross Entropy
Mutual Info
Text Reg.
5

---

In the following, describe the notations occurring in model in Eq. 44 as well as the effect of each of its terms:
The mutual Information. I( 0,) is the mutual information between the visual inputs and the textual class representations, considering both the input images and the text prompts random variables, denoted X' and C, respectively. We compute the posterior distribution over the textual class descriptions, given the input query images, using the zero-shot predictions:
Pr(C cx x;: 0,: 0.) Pi,k
(5)
and the marginal distribution of the textual descriptions over the query set as follows:
Pr(C C25 Ou. 0r)
1
Σ
Pi.k Pk
(6)
This part of the objective integrates two terms, conditional entropy HCC]X 0.) 0.) and marginal entropy H(C; 0,, 0z), as detailed in the following:
10,, 0.) (C; (C)na 0,, 0.)
where dent and Acond are two nonnegative weighting factors that control the influence each term, H(C)X; 0,, 0z) is the empirical estimate of the conditional entropy expressed as a function of the language- driven predictions as follows:
K
0.) = K
01, 0.) - Pr(ca|X'= x;; 0, 0.) In =x;;
Pi In
(8)
1E0
k= ieQ
and HCC; 0,) denotes the empirical estimate of the marginal entropy given by:
H(C: P., 0.)
Σ Pr cRi 0.) Pr(C cx; 0z)
Σ Pk In Pk
(9)
Each entropy term plays a distinct and complementary role during adaptation. On the one hand, O,, 0.) aims to minimize the uncertainty associated with the text-based prediction of each query sample. On the other hand, HC;; 0., 0.) promotes balanced distribution between classes, encouraging diversity in class predictions, and preventing the model from favoring limited number of classes. Thus, integrating the marginal entropy serves as regularization term preventing degenerate solutions that may arise when solely minimizing conditional entropy. Together, these entropy terms enable the model to produce confident and well-distributed predictions on the unlabeled samples.
The text-based regularization. This term provides text-based supervision, as it minimizes Kullback- Leibler (KL) divergence between the network's probability outputs, P; = (Prx) and the zero-shot predictions (i.e., the initial network's 's outpuls), which we denote 8. = (vi,))
T(O., 0.) = -eeent Σ KL(p:l1S+)
(10)
150
with
KL(P:ll5.) = P log Pi i
(11)
y:
and Atert is a nonnegative weighting factor enabling control the contribution of this term our overall loss. This text-based regularizer helps preserve the zero-shot generalization capabilities of CLIP, penalizing large deviations of predictions p, from initial zero-shot predictions ý:-

---

3.3 Umbrella method
Thanks its structure, which simultaneously incorporates both visual and textual feature embeddings, and both labeled and unlabeled data, our loss could be viewed generalization of state-of-the-arr methods.
Generalization of TIM (Boudiaf et al., 2020b). Since TIM a transductive information maximization approach initially designed for vision-only classifiers, it relics only on vision features and omits textual elements. By setting Atert in our loss function I(O,, 0r) in equation 4 and fixing the text encoder to act just linear classifier, whose weights given by the text embedding, can perceive TIM specific case of LIMO.
Generalization of CLIP-LoRA (Zanella & Ben Ayed 2024a). CLIP-LoRA leverages low-rank ma- trices to adapt the vision and test encoders of CLIP. This method operates in a purely inductive manner that relies exclusively on labeled data without incorporating unlabeled data. By setting Atezt 0, Xent 0 and Acond = 0 in our loss function, we recover CLIP-LoRA as specific instance of LIMO.
3.4 Low-Rank Adaptation
In the following, we present our fine-tuning strategy based on LoRA (Hu et al. 2021a] to adapt vision and text encoders. This choice is motivated by the fact that full fine-tuning is well known to produce degenerate solutions (Chen et al.) 2019 Boudiaf et al.) (020b), hence a more efficient way to apply LIMO is required. LoRA approximates incremental updates of the pre-trained weights as the product of two small matrices, and B, based on the concept of "intrinsic rank" of the downstream task where A R B E Rmxr are two low-rank matricos and typically << m,n. This low-rank update is then incorporated into the forward pass of specific network layer I. The output hidden state h() is computed using the hidden state from the preceding layer, h(t-1), and the layer's original, frozen weight matrix, W(), as follows:
h = w( + AW (-1) (0-1) BB A (1-1)
(12)
where y is scaling factor that controls the influence of the low-rank updates. Values in matrix A are randomly initialized via Kaiming initialization and (() is filled with zeros so that there is no incremental update before training.
4 Experiments
4.1 Datasets.
We evaluate our method on eleven publicly available image classification datasets, which have already been
used in prior studics (Zhou et al. 2022b Zanella et al.]
2024 Martin et al. 2024). Those datasots include
ImageNet {Deng et al. [2009), SUN (Xiao et al. 2010), FGVC (Maji et al. 2013) EuroSAT (Helber et al.
2019), Cars et al., 2013). (Bosaadd et al. 2014), Pets(Parkhi
al. 2012). Flowers (Nilsback
& Zisserman 2008), Caltech Feii-Fei et al.) 2004), DTD (Cimpoi et al.] 2014) and UCF (Soomro 2012).
Unless otherwise specified, all experiments were conducted with the pre-trained ViT-B/16 vision encoder from CLIP, and the numerical results are presented as top-1 accuracy, averaged over three distinct random seeds.
4.2 Implementation details
Our fine-tuning strategy. We follow the design choices of (Zanella & Ben Ayed 2024a) by applying LoRA to all the query, key, and value matrices of the vision and text encoders of CLIP, using a rank of
= 2. A dropout layer with p= 0.25 is used to regularize the input to the LoRA module. In the few-shot setting, the number of iterations is set 500 times [S/K (the number of labeled samples per class). Unless otherwise specified, these hyperparameters remain fixed across all experiments.
7

---

The core component of our transductive formulation is the language-aware information maxinization loss function, which comprises three terms, see Eq. A We set the conditional entropy weight Acond to 1, the marginal entropy weight Xent to 10, and the text regularization weight Ateaz to 0.1. To ensure simplicity and gencralizability throughout the experiments, these hyperparameters are kept conslant across all experiments.
LIMO hyperparameters.
4.3 Results
LIMO overall performances. Table [ highlights our results in comparison with state-of-the-art few- shot learning methods. We can observe that LIMO consistently achieves leading performances across all datasets in the 4-shot and 16-shot settings and across 10 datasets in the 2-shot setting, yielding significant improvements over both transductive and inductive methods. For instance, LIMO outperforms TransCL.IP ZZanela et al. 2024 by average of 5% in the 4-shot setting.
Table 1: Comparison of on 11 datascts. Bolded values indicate highest accuracy. denotes transductive methods, denotes non- transductive methods.
methods in few-shot learning: Top-1 classification accuracy
ImegeNiee.
Sun
FGVC
FuncSAT
Cars
Food
Petx
Floames
Calborl
DTTD
UCP
Zeeo Sbot
66.7
62.6
24.7
47.5
65.8
56.1
89.1
71.4
92.9
43.0
56.7
65.1
TP
Dillim et al.
2019
00.5
51.0
25.3
63.1
15.1
58.8
31.8
53.3
67.0
47.3
56.1
56.0
Vidloe-only
BD-OSPN
Lau
2020
46.1
36.1
25.7
64.7
50.7
67.6
64.0
89.6
89.0
48.9
64.0
00.8
LastocinSSot
Ziko ct
2020
45.8
$5.0
27.1
05.2
61.1
68.2
66.0
59.7
80.0
48.9
65.1
61.4
PU-MAP
50.7
6311
71.7
77.3
75.7
73.9
36.1
53.8
08.7
2-thot
11M
Moudlat et
47.9
6U.7
70.8
TH.7
706
91.4
86.6
50.3
06.4
67.0
67.0
65.1
80.0
98.1
54.2
Vielon-1 Langragg
2023
68.8
73.2
93.9
57.5
Hrang
CLIP-LORA
Zundia Bon Avad
20244
70.8
71.1
85.2
82.7
73.2
68.2
91.8
50.5
96.6
50.0
TLPP
Zanella et
2024
70.9
70.0
30.0
77.1
71.7
87.0
91.7
50.6
98.5
56.1
78.5
74.2
LMO (Oun)
71.1
72.3
33.8
91.7
74.5
56.3
93.8
92.4
94.0
63.3
81.2
77.8
TF
51.1
61.0
90.5
61.9
56.8
21.0
65.9
50.9
91.5
53.7
67.9
61.1
Vielce-only
BD-OSPN Lapiociushoot
53.8
62.5
30.5
64.8
18.5
75.4
7200
92.5
92.0
02.3
70.9
65.9
53.5
62.5
23.6
74.3
58.5
75.7
73.4
52.6
92.0
52.7
71.7
07.0
PT-MAP
57.6
68.1
31.2
74.9
61.1
61.1
70.5
76.2
60.2
58.4
73.9
65.8
4-ttoc
'IM
57.4
67.0
32.8
79.3
655.5
63.5
82.3
93.4
88.5
08.1
76.5
Vieloon-1 Langringe
Cotip
68.8
60.7
30.9
09.7
92.5
9812
90.5
06.5
Tip-Allattee-P
70.8
76.8
60.5
96.8
70.79
73.21
CLIP-LoRA
72.8
37.9
84.9
77.4
52.7
91.0
90.7
96.2
63.5
61.1
ThanCLIP
71.0
BL.
70.4
74.0
50.4
91.0
90.6
94.0
61.1
79.1
75.9
LIMO (Omm)
71.9
76.1
40.3
90.1
78.6
86.6
94.5
96
85.8
67.6
88
80.2
TF
61.8
70.1
35.8
24.8
71.2
50.7
70.5
56.4
98.8
62.0
76.0
73.1
Vieten-mnly -nnl
BD-CSPN LeplociaSShot
61.7
60.1
37.7
73.4
70.7
90.2
81.2
$4.6
95.3
61.5
76.0
72.7
00.0
68.1
3EL1
75.1
103.2
$1.2
81.7
94.5
98.1
56.6
76.3
72.6
PT-MAP
64.0
7200
37.4
73.6
72.0
52.7
8G.1
78.5
68.7
64..7
76.3
70.2
10-shot
TIM
67.8
754
40.6
83.6
79.5
61.9
88.7
95.1
92.1
67.5
89.1
77.8
Vielon-I Langingg
ChOp
71.0
71.8
45.2
63.0
67.9
61.3
92.00
90.8
95.8
09.7
85.
Tip-Adastie-F
73.4
76.0
65.9
$6.6
92.0
56.2
96.7
20.6
83.9
LP++
72:95
78.07
41.46
64.05
80.64
02 28
71.50
CLIP-LORA
76.1
M2.1
55.3
54.2
92.4
98.0
96.4
72.0
767
83.0
79.5
56.9
92.4
94.4
96.0
65.1
62.1
LIMO (Oum)
74
77.7
37.5
94.1
87.4
87.1
95.5
98.7
97.1
71.7
80.3
Comparison with vision-only classifiers. Transductive few-shot learning methods, designed specifically for vision-only-cassifiees, rely exclusively on visual information, omitting textual elements. By comparing LIMO to these approaches, we can assess the impact of incorporating the textual modality to enhance demonstrates that integrating language within our transductive framework leads to substantial gains over all vision-only methods. Notably, our loss achieves
our mutual information regularization term. Table D
8

---

average improvement exceeding 12% in the 2-shot setting compared to the TIM method [Boudiaf et al, 2020b) based only on vision features.
Transduction improvements. We compare LIMO with stale-of-the-art inductive few-shot methods, pri- marily developed for VLMs. This comparison enables us to evaluate the impact of our loss in effectively leveraging both textual and visual information in transductive fashion enhance model performance. As shown in Table [ our transductive approach significantly outperforms inductive VLMs across all tasks, par- ticularly achieving
average improvement of 2.8% over CL.P-LoRA (Zanella & Ben Ayed 2024a] in the 4-shot scenario. Interestingly, while other works have reported diminished transductive gains the number of shots increases (Zanella et al. 2024 aatribbuting this to inductive methods better capturing the underly- ing data structure with more samples approaches, even in the 16-shot scenario. This highlights that transduction can be highly valuable across varying levels of supervision.
LIMO consistently delivers significant improvements over inductive
LIMO's behavior. We further analyze how dataset properties influence LIMO's gains by comparing our rosults to CLIP's zero-shot performance. We observe that datascts exhibiting substantial distributional shift from CLIP's pretraining distribution, such as Eurosat, DTD and FGVC, show the largest improvement compared to other datasets. This trend aligns with LIMO's objective, which maximizes the mutual informa- tion between visual inputs and textual class representations, particularly effective when the default alignment between CLIP's encoders is weak. In contrast, datasets on which the model already achieves high zero-shot accuracy (e.g.; Food), tend to exhibit smaller gains in low-shot settings, and may even under-perform, as the limited guidance provided by the support can cause LIMO to slightly over-regularize. This behavior is reflected in the 2-shot results for Food reported in Table
Statistical analysis. Figure 2 plots additional results showing error bars (conventional 95%confidence in- tervals) averaged across four representative datasets including Ane-grained (FGVC), texture (DTD), satellite (Furosat)) and generic (Caltech) over 10 random seeds. This comparison includes the top competing meth- ods: TransCLIP (Zanella et al., 2024), CLIP-LoRA (Zanella & Ben Ayed 2024a), and our proposed LIMO under the 2-, 4- and 16-shot scttings. One may observe that there are no overlapping crror bars between LIMO and other methods, which suggests stalistically significant improvements relative to the state of the art.
TransCLIP
CLIP-LORA
F
LIMO
2-shot
4-shot
16-shot
74
Top-1 Accuracy (%)
70
80.0
68
70
F
77.5
68
75.0
64
66
72.5
62
60
70.0
TransCLIP
CLIP-LORA
UMO
TransCLIP
CLIP-LORA
UMO
TransCLIP
CLIP-LORA
LIMO
Figure 2: Statistical comparison of LIMO, TransCLIP (anella et al. 2024 and CLIP-LORA Zanella &
Ben Ayed| 2024a under the 2-, 4- and 16-shot settings. Error bars represent the standard deviation averaged across four datasets (FGVC, EuroSAT, Caltech, and DTD) using 10 seeds.
4.4 Ablation studies
LIMO performances are consistent across various vision encoders. As presented in Table 2 LIMO outperforms leading transductive approaches for vision-language models (TransCI.IP (Zanella et al. 2024
9

---

and vision-only classifiers (TIM (Boudiaf al. 2020b ) with both the VIT-B/32 architecture and the larger VIT-T./14. This performance advantage also holds when compared to one of the best-performing inductive few-shot methods in the literature, CLIP-LoRA (Zanella & Ben Ayed, 2024a), thereby highlighting the versatility of LIMO. Detailed results for both backbones are provided in the Appendix (Table and 7
Table 2: Average Top-1 accuracy over 11 datascts for the ViT-B/32 and ViT-L/14 visual backbones. Bolded values indicate highest accuracy. v denotes transductive methods, X denotes non- transductive methods.
2shot
4-shot
16-hoo
TIM BBodiial
20/2bb
58.7
65.5
73.3
CoOp Zhou es al.
20226
X
65.5
80.5
76.7
VIT-B/32
CLIP-LoRA Zanella Ben Aven
72.9
78.9
TransCI. IP
Zanelin ri
2724
70
71.5
74.3
LIKO [ours]
/
73.2
75.7
81.1
83.4
CoOp
X
78.9
79.4
84.6
WT-L/14
CLIP-LoRA
X 80.9
82.7
86.9
ThrasCLIP
LIMO (ours)
83.2
85.1
88.6
Components of LIMO. We assess the contribution of each term in our loss function the overall perfor- mance of LIMO. The results, shown in Table 3 demonstrate that the full integration of all four components achieves better results than any other configuration. Interestingly, removing the mutual information term, I(0,,0)) reduces significantly the performance of LIMO, particularly when only the last visual projector is updated. This outcome can be attributed to the mutual information term's role in improving the model's confidence over individual decisions while promoting a balanced distribution of predictions across classes. Additionally, integrating the text-based regularization term T(0,, 0.) with the mutual information increases the general performance of LIMO for both LoRA and LVP fine-tuning strategy. Aligning the model's pre- dictions with the zero-shot predictions may help the model to preserve relevant zero-shot generalization properties and prevent overfitting. These results highlight the importance of each term in Eq. (4.
Effects of different fine-tuning strategies. While LIMO is optimized based on LoRA, alternative fine- tuming strategies exist such adapting the textual input (e.g., CoOp (Zhou al. 2022b) or selectively adjust some specific components of the model (e.g., ProLIP (Fahes et al., 2024)). To assess different ap- proaches, introduce two LIMO variants in Table LIMO-CoOp, in which the textual input is finc-tuned set of trainable vectors, and LIMO-LVP, where only the last visual projector layer is updated. Across nearly all configurations, LIMO (using LoRA) outperforms both variants by an average margin of 4%, sug- gesting that the parameters requiring adaptation for optimal performance are likely distributed throughout the model rather than concentrated solely in the final visual projection layer or the textual input. In one specific configuration (when only text-based regularization term T(Q., 0.) integrated), LIMO-CoOp out- performs the loRA-based variant by approximately 1%.
Orthogonality of LIMO. Table 3 also allows us to assess the compatibility of LIMO's ' objective function with other methods, such as CoOp Zhou et al. 2022b) and ProLIP (Fahes et al., 2024 As reported, integrating our objective function leads to consistent performance gains improving CoOp by approximately 1% and ProLIP by around 5%. These results highlight the orthogonality of LIMO and its effectiveness in enhancing the performance of other adaptation methods.
Computational Requirements. LIMO's cost varies based several factors such as batch size and the scope of parameter adaptation (e.g., whether one or both encoders are fine-tuned), However, it remains minimal due to its orthogonality. For instance, integrating LIMO with CL.P-LoRA, as in the paper, increases training time by only ~ 5 minutes when adapting CLIP VrT-B/16 to Image on a single NVIDIA RTX A6000 GPU.
Sensitivity analysis of LIMO'S hyperparameters. Table presents a comprehensive analysis of the impact of the regularization weights Aent; Acond, and Atext model performance. We observe that increasing
10

---

Table 3: Ablation study on the contribution of each term of our proposed loss function LIMO under differ- ent fine-tuning strategies: LoRA Low-Rank Adaptation), LVP (fine-tuning only the Last Visual Projector of CLIP), and CoOp (learning only the textual prompts). We report 4-shot results as top-1 accuracy and the average top-1 accuracy across 10 datasets (excluding ImageNet).
Method
Tow
Sun
PGVC
EArSAAT
Cars
Food
Pets
Floneers
Caltech
DTD
CCF
Average
C(9., 8.)
72.78
37.60
81.76
77:41
$2.96
90.39
94.19
95.30
64.19
80.77
78.05
LaRA
Fiu
((9.. 6.)
73.68
39.72
78.11
85.89
96.78
95.73
68.3
50.79
C(A,, 6r + 718..%)
73.00
36.51
06.63
77.82
84.77
90.14
91.77
95.11
54.87
79.08
74.04
C(o., Ic)..00) T(.....)
73.94
39.78
92.69
78.75
86.67
94.3
06.08
95.91
67.95
84.01
81.01
(( ..)
71:22
33.67
76.2
72.03
84,09
89.91
92.11
110,322
09.1
77.46
75.15
CuOp
Zhou
C(O,, (.) I10.. 6.)
71.11
$1.09
76.98
70.87
86.05
02.61
00.12
04.07
62,96
78.87
75.02
C1O., 0v) + 70 90
71.32
33.76
75.28
72.06
84.61
80.0
92.08
04.86
57.73
77.39
74.9
20220
CWO,, I(G., 6.) T(9...)
71,18
31.77
76.52
70.82
86.04
92.6
95.29
95.02
63.2
78.77
75.92
C10.,00 0.
60:29
34.42
75.06
72.M
77.68
82.38
08. 86
03:47
02:18
75.65
73.73
LVP Palocs
C(9., 0.) 6.)
72.85
33.01
84.27
74.80
84.91
92.04
97.05
05.28
06.47
81.85
78:28
MS m
C(A,, 9:) + T(0.,%)
60.02
34.22
63.29
73:45
83.34
88.63
87:23
03.66
49.45
71.64
71.39
20124
C(O..0,) ()).. 0(),.0)
73.25
35.1
86.52
76.24
86.19
93.12
96.7
95.58
65.74
82.41
78.09
Table 4: Ablations on hyper-parameters: Average top-1 accuracy across 7 datasets {DTD, EuroSAT, Food, UCF, FGVC, Caltech and Flowers} using CLIP ViT-B/16 under a 2-shot setting. Each hyperparam- eter studiod individually, while keeping other parameters fixed as in the main configuration.
0.1
1
10
Aent
77.27
77.29
77.49
Acond
73.5
77.49
71.81
Ateart
77.49
70.22
64.12
the marginal entropy weight dent consistently improves classification accuracy, reaching maximum of 77 17.49% at dent = 10. This trend suggests that stronger marginal entropy regularization- -whihh encourages balanced class predictions across the dataset- the case of the conditional entropy weight Acond, performance is optimal at the intermediate value of 1, while both lower (Acond = 0.1) and higher (Acond = 10) values result in notable degradation. This finding indicates that a moderate degree of conditional entropy minimization is necessary to encourage confident, predictions without leading to overfitting or instability. Lastly, the KL divergence weight Atext, which controls alignment with the zero-shot prior, exhibits the highest sensitivity among the three. While small values (Areat 0.1) yield the best performance (77.49%) increasing this weight substantially reduces accuracy cecuracy-flllingg to 70.22% at Atezt = 1 and to 64.12% at Atext = 10. These results underscore the importance of carcfully constraining the influence of the zero-shot prior to preserve the model's adaptability in low-data regimes.
beneficial guidance in the transductive setting. In
Table 5: Comparing LIMO with a TransCLIP variant that fine-tunes CLIP's text and vision encoders using the TransCLIP objective and LoRA: Top-1 classification accuracy is reported over 11 datasets. The highest accuracy values are highlighted in bold, and the second highest are underlined.
ImageNet
Sun
FCVC
EurSAAT
Car
Food
Pets
Flowers
Cahoda
DTD
UCP
Avcrage
2-hhot
TransCLIP
Zazella
a124
2010
71.7
87.0
91.7
70.2
71.18
72.44
27.53
73.93
96.95
92.
94.52
75.53
LIMO (Oun)
21.08
22.81
$5.8
91.88
74.47
96.25
98.78
92.45
94.99
65.52
81.21
77.75
4-alok
TranCCIPP
70.3
71.9
36.0
70.1
71.0
50.4
91.6
93.0
91.0
61.1
20.1
75.9
TronCCLPP LoRA
71.99
73.5
33.3
17.3
53.02
27.20
LIMO (0rm)
40.3
16-ahot
Tran
76.7
38.6
85.0
70.5
50.9
02.4
81.1
01.0
65.1
80.1
79.1
TrancC &PA
25.2
27.4
47.08
91.01
90.75
87.6
111 189
95 95
98. 46
68.33
S3.11
$1.91
LIMO (Ouas)
73.99
77.69
57.54
94.42
87.42
57. .20
95.28
98.73
07.05
74.73
$9.25
84.86
11

---

Fairly comparing LIMO with TransCLIP. To enable fair comparison between transductive method, LIMO, and TransCIIIP (Zanella et al. 2024), we introduce new variant, termed TransCTIP--TRRA.
In this variant, we replace the objective function of LIMO, presented in Eq. ( with the objective function proposed in the TransCLIP paper (Zanella et al. 2024) while maintaining the same fine-tuning strategy, LoRA, as used our approach. The TransCLIP objective function is composed of four components: (i) a cross-entropy loss applied to labeled samples, (ii) a GMM clustering loss structuring the unlabcled data, (iii) KL divergence loss aligning the model predictions with the zero-shot predictions, and (iv) Laplacian regularization term promoting smooth predictions across samples with similar feature representations. To balance their contributions, we assign specific weighting factors to each term; 2 to the cross-entropy loss, 2 to the GMM clustering loss, 1 to the KL divergence term, and 0.05 to the Laplacian regularization term. Experimental results for this variant are reported in Table 5
As shown Table 5 substituting our proposed objective function with the TransCLIP objective leads to a performance dogradation of approximately 2.5% across all evaluated settings. This consistent performance drop clearly demonstrates the effectiveness of our objective function in achieving superior performances in transductive few-shot learning. Moreover, the results show that integrating the TransCLIP objective with LoRA fine-tuning strategy yiclds an approximate 2% improvement compared to the standard TransCLIP setting, further confirming the significant benefits of our fine-tuning strategy based on LoRA, which focuses on adapting the inner representations underlying the proposed method LIMO specifically, the combination our information maximization loss function in Eq. 4 and the LoRA fine-tuning strategy- for achieving optimal performance.
the model. Overall, these findings validate the design choices
5 Discussion
A natural question concerns our focus on the all-to-all setting, where training and testing share the same set of classes, and the absence of results under the base-to-novel setting popularized by CoOp (Zhou et al., 2022b). This choice is motivated by the fact that our proposed loss function is conceptually designed under the assumption that the support and query sets operate within the same category space. The information- maximization objective we employ promotes a balanced inter-class distribution, encouraging diversity in predictions, and preventing the model from favoring a limited subset of classes. However, as discussed in Chiaroni et al. (223)) , such a behavior is particularly effective when the class space is fixed (all-to-all selting) but becomes harmfull on long-tailed data, where the model required to generalize to unseen categories, as in the base-to-1 tow setting. A promising direction for future work would be to extend our loss this famous setting by redefining the mutual information term to account for unseen classes, following approaches such in generalized category discovery CCharoni et al. 2023 and by adjusting the text regularization term accordingly.
6 Conclusion
In this work, we presented novel approach to handle transductive learning for vision-language models. We first introduced Language-aware Information MaximizatiOn (LIMO) loss function which is based on strong text-based regularization terms. Second, we demonstrated the potential of fine-tuning subset of the model parameters within the transductive paradigm, notably with low-rank matrices (LoRA), a perspective overlooked by current transductive methods working solely in the embedding space. We evaluated our proposed LIMO on 11 datasets and showed that outperforms state-of-the-art transductive and inductive sew-shot learning methods, highlighting new possibilities for combining PEFT and transductive learning in future research.
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), Also, thank Calcul Québec and Compute Canada for providing compute ressources.
12

---

References
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In Computer vision-ECCo 2014: 18th European conference, zurich, Switzerland, Seplember 6-12, 2014, proceedings, port VT 13, pp. 446-461. Springer, 2014.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Tsmail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European conference on computer vision, pp. 548-564. Springer, 2020a.
Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 33:2445-2457, 2020b.
Arnay Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficien fine-tuning. arXiv preprint or orx:22306. 07967, 2023.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang. and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678, 2022.
Wei-Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv: 1111999440...... 2019.
Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Parametric infor- mation maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1729-1739) 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3606 3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1ms. Advances Neural Information Processing Systems, 36, 2024.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. XrXie preprint 1909:0999007220 2019.
Karim El Khoury, Maxime Zanella, Benoit Gérin, Tiffanie Godelaine, Benoît Macq, Said Mahmoudi, Christophe De Vleeschouwer, and Ismail Ben Ayed. Enhancing remote sensing vision-language models for zero-shot scene classification. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEFF, 2025.
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Fine- Fine-tuning clip's last visual projector: A few-shot cornucopia. arXiv preprint arXim:2410.0522 2024.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pottern recognition workshop, pp. 178 178. IEEE, 2004.
Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongshong Li, and Yu Qiao. Clip-adapter: Better langunee models with feature adapters. International Journal of Computer Vision, 132(():581-588 -595, 2024.
13

---

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. TEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7))2217 2226, 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint Xix: 066800 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arX::2206.. 09685, 2021a.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eichh Matsumoto, and Masashi Sugiyama. Learning discrete rep- resentations via information maximizing self-augmented training. In International conference on machine learning, pp. 1558 1567. PMLR, 2017.
Yuqing Hu, Vincent Gripon, and Stéphanc Pateux. Leveraging the feature distribution in transfer-based few- shot learning. In International Conference on Artificial Newral Nelworks, pp. 197-499. Springer, 2021b.
Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudial, Houda Bahig, and Ismail Ben Ayed. Lp+++ A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition, pp. 23773 23782, 2024.
Mohammed Jabi, Marco Pedersoli, Amar Mitiche, and Ismail Ben Ayed. Deep clustering: On the link between discriminative models and k-means. JEEE transactions pattern analysis and machine intelligence, 43 (6):1887-188)) 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Haribaran, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Yannis Kalantidis, Giorgos Tolias, et al. Label propagation for zero-shot classification with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp.
2024.
Mete Kemertas, Leila Pishdad, Konstantinos G Derpanis, and Afsanch Fazly. Rankmi: A mutual information maximizing ranking loss. In Proceedings of the IEEE/ /VF Conference on Computer Vision and Pattern Recognition, pp. 14362-140 14371, 2020.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113 19122, 2023.
Sanghyeon Kim, Hyummo Yang, Yunghyun Kim, Youngjoon Hong, and Eunbyung Park. Hydra: Multi-head low-rank adaptation for parameter efficient fine-tuning. Neural Networks, pp. 106414, 2024.
Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. Advances in neural information processing systems, 23, 2010.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catego- rization. In Proccedings of the IEEE international conference computer vision workshops, pp. 554-561, 2013.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new bascline for efficient model tuning. Advances in Neural Information Processing Systems, 15:1091123, 2022.
Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In Computer Vision-ECoo 2020: 16th Europeon Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part I 16, pp. 741-756. Springer, 2020.
14

---

Yanbin Liu, Juho Lee, Minscop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. orXiv preprinl arXiv:1805.10000 2018.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems, 36, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fino-grained visual classification of aircraft. Xiv preprint 1rXis11306.515,, 2013.
Ségolène Martin, Yunshi Huang, Feroshtch Shakeri, Jean-Christophe Pesquet, and Ismail Ben Ayed. Trans- ductive zero-shot and few-shot clip. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 28816 28826, 2024.
Maria-Elena Nilsback and Andrew Zisserman. Aulomated flower classification over a large number of classes. In 2008 Siath Indian conference on computer vision, graphics & image processing, pp.
IEEE, 2008,
Yingwei Pan, Ting Yao, Yehao Li, Chong-War Wah Ngo, and Tao Mei. Exploring calegory-agnoslic clusters for open-set domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13867-13875, 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMI.R 2021.
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzich Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model luming, arXie preprinl ar Xiv:v220 10462, 2024.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14222 2022.
K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual infor- mation maximization for representation learning. arXio preprint 1rXiv:1907.13663 2019.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint wxXiv:2210007588 2022.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEFE, 2010.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with showledge-guideed context optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6757 6767, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient finc-tuning for transformer- masked language-models. Xix preprinl ar.iv:2106. 10199, 2021.
Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of rision-languan models. In Proceed- ings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593 1603, 2024a.
15

---

Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 23783 23793, 2024b.
Maxime Zanella, Benoît Gérin, and Tsmail Ben Ayed. Boosting vision-language models with transduction. Newral Information Processing Systems (NewrIPS), 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budgel allocation for parameter-efficiett fine-tuning, arXiv preprint orXic:2303.1011: 2023.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493 510. Springer, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IBEE/CVP conference compuler vision and pallern recognilion, pp. 18816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision langualanguage models. International Journal of Computer Vision, 130(9):2337-29)) 2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659 15669, 2023.
Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In International conference on machine learning, pp. -6660-16670. PMLR, 2020.
16

---

A Ablations on Model Architecture
Table 6: Detailed results of state-of-the-art methods the few-shot setting for the 11 datasets with the ViT-B/32 as visual backbone: Top-1 classification accuracy averaged over random seeds is reported. Bolded values indicate highest. accuracy. / denotes transductive methods, Xdenotes non- transductive methods.
Imragehes
Sin
FUVC
EunSAT
Cars
Food
Piss
Flowers
Caltedt
LTTD
UUF
Auerage
Dhillon
2010
34.7
57.4
48.7
47.4
Vistorontl
3012
51.1
90.7
57.2
21.1
55.5
00.3
82.4
86
45.6
01.6
54.5
Leplacirathor
21ho et al. 20121
39.1
51.9
20.4
58.3
02.4
57.7
57.3
82.5
86.7
45.9
02.6
56.2
PT-MAP
Ho et al
21215
42.8
00.1
22.5
63.7
08.0
65.9
61.0
69.5
85.0
50.4
66.8
55.0
TiM
Bundial et al
41.1
59.0
21.1
68.9
51.1
68.3
00.1
66.5
61.5
15.6
68.1
58.7
Vidoos-onll Visions Langungg
CoOp
Rion
20226
61.3
01.5
16.0
02.4
02.3
73.1
85.5
62.0
91.2
40.6
70.4
56.5
Tip- TTp-Adaptus- 7 Zhang 2
2002
64.9
06.5
24.5
67.4
61.7
80.8
88.2
86.5
92.8
54.9
71.8
50.2
CLIMLORA
Zanolla Hon
05.7
84.1
61.8
76-3
ThonChlll
Zanello
2024
64.8
61.8
LIM
23.2
68.6
60.8
57.0
65.2
BD-CSPN
40.1
60.3
64.8
67.1
SoplactorShet
46.8
23.8
08.4
60.2
65.0
00.6
8933
51.4
67.5
PT-MAP
36.1
63.5
94.1
08
57.3
70.3
00.00
73.3
37.3
56
70.1
59.7
TIM
50.4
05.0
24.7
70.0
66.1
73.0
74.4
90.5
68.7
56.0
21.8
66.5
Visioon Langnnngg
CoOp
65.2
07.1
24.0
68.7
06.2
75.8
88.6
87.9
95.0
56.3
76.0
50.5
Tip-blapteeFF
60.8
65.3
26.8
71.5
07.6
50.5
H5.6
88.9
94.6
38.0
70.1
71.6
CLIP-LORA
60.5
70,3
27.7
85.5
63.3
76.6
86.3
90.1
94.3
50.3
76.5
72.9
Tra
64.7
70.1
25.1
78.0
00.5
80-3
87.2
58.0
74.3
71.5
LIM (Dund)
66.9
71.6
29
90.8
70.8
N0.8
91.44
63.5
70.3
75.7
35.8
Vidoomrnnly
399.8
02.3
72.
71.2
LoplactasiShot
155.4
00.9
7312
70.3
91.3
PT-MAP
36.9
71.5
63.1
76.1
78.7
60.7
61.9
10bahort
TTM
60.5
71.8
$3.0
73.2
85.0
86.1
66
78
73.3
Visioon Langunnge
CoOp
66.8
72.2
92.9
60.5
78.0
78.6
88.7
96.4
94.9
66.3
78.6
76.7
Te--Mlppte-F
68.4
74.1
$4.8
60.4
77.0
81.7
90.4
94.3
96.1
68.0
50.5
77.1
CLI--LRA
08.4
71.0
64.9
01.5
71.7
78/2
88.88
96.2
90.2
08.2
53.6
78.9
TrassULaP
60.6
72.6
20.1
76.9
71.2
51.1
89.5
90.9
94.4
62.7
77.2
74.3
LIMO (Ouns)
0B
76.3
46.8
83.6
81.9
$1.0
02.5
97.4
96.3
72.5
86.8
81.1
17

---

Table 7: Detailed results of state-of-the-art methods in the few-shot setting for the 11 datasets with the ViT-L/14 as visual backbone: Top-1 classification accuracy averaged over random seeds is reported. Bolded values indicate highest accuracy. denotes transductive mothods, Xdenotes non- transductive methods.
Imangedees
FGVC
EwoSAT
Clus
Food
Cultedi
TP Dhillen ef al
2019
56.1
55.6
$3.5
71.7
58.5
71.8
05.7
95.0
90.5
49.8
69.4
61.6
Vislom only
BD-CSPN
Lin et
37.0
01.2
33.6
77.6
03.1
70.9
77.2
95.7
92.8
52.3
74.7
08.5
Loplactenttor
2lko al
21211
36.5
61.3
35.0
70.5
05.4
5033
77.4
96.2
933
32.4
74.6
70.0
PT-MAP
Bu
20215
61.3
03.0
37.0
78.4
68.4
87.3
86.7
77.9
61.1
50.5
76.2
68.9
2-hhot
TIM
Bundiaf et al
39.7
07.6
$5.4
60.2
00.5
87.4
85.5
96.1
91.4
582
28.6
73.2
Viskon Langunge
0o0p
2022b
79.1
38.8
Tip- Tip-blaptceF
Zhang
2102
76.6
90.4
CHIP-EORA
Zanalia
(Ous))
77.8
83.6
91.6
1V
61.6
66.5
40.8
71.4
03.0
61.5
73.0
96.2
94.4
38.5
77.5
71.5
Vaion-onby
UD-CSSPN
64.3
67.8
40.6
71.4
71.2
84.7
H2.8
96.7
96.2
56.9
79.6
71.8
LaplarionShut
65.8
67.6
60.0
18.9
72.0
65.4
85.7
97.3
95.2
56.7
79.6
71.7
PT-MAP
68.0
72.7
41.7
77.4
73.8
66.9
89.9
76.3
69.9
60.1
79.3
72.1
TDM
68.0
72.7
42.0
78.4
77.8
90.0
92.3
97.4
91.1
69.5
59.7
78.0
Viitoee Langinnge
CoOp
76.9
73.1
43.6
76.9
61.5
88.7
91.6
96.9
96.5
63.9
52.8
79.4
Tip-Adaytee-
77.1
M2.3
CLIP-LORA
77.9
15.2
TrassCLIP
76.9
1JM (Ound)
71.1
81.5
Visbowoonl
BD-08PP
21.1
74.4
09.4
78-1
81.2
88.0
89.88
98.2
95.6
06.5
69.5
LaplacterShot
69.8
78.7
47.0
61.7
10.2
140.1
96
95.7
63331
80
78.0
PT-MAP
72.9
75.9
68.1
70.1
70.0
80.4
52.0
50.5
66.0
65.8
50.5
76.4
TTM
26.1
18.7
$2.5
80.4
80.5
91.0
52.0
96.2
94.5
75.2
54.6
63.4
Viskon Langungg
CoOp
78.2
77.5
05.2
89.5
50.0
20.8
11.00
90.1
07.2
74.4
$7.3
$4.6
Tip-blapte-FF
70.3
79.6
55.8
56.1
59.1
91.6
94.6
98.3
97.5
74.0
57.4
$4.5
CLIPARRAA
7166
79.1
05.2
50.1
10.9
01.5
Tra
77.8
53.9
15..3
LIMO (Cuns)
18


Reviews:
[{"review_id": "rPtPk2CIjb", "reviewer": "Reviewer_nHa9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\n- This paper proposes LIMO, a transductive few-shot learning method for CLIP, which introduces a novel loss function that calculates mutual information between visual inputs and textual class representations. Comprehensive evaluations across 11 datasets demonstrate that LIMO outperforms state-of-the-art methods by significant margins.\n\nStrengths:\n\n- It integrates mutual information with text-based regularization, creating a principled objective function for transductive settings.\n- Comprehensive experiments across multiple datasets and shot settings demonstrate the superiority of the proposed method.\n- Ablation studies highlight the importance of each loss component.\n\nWeaknesses:\n- The fundamental concept of the mutual information objective is not new. This loss term appears to be a direct adaptation of a prior vision-only method [1], primarily by substituting the predictive logits with vision-language ones from CLIP. \n- The evaluation is conducted exclusively in an *all-to-all* setting, where training and testing occur on the same set of classes. This fails to assess the method's performance in the widely used *base-to-novel* setting (as established in CoOp).\n\n[1] Transductive Information Maximization For Few-Shot Learning, NeurIPS 2020", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the submission are well-supported by comprehensive evidence from extensive experiments across 11 datasets and multiple few-shot settings.", "audience_yn": "Yes", "audience_explanation": "Yes, the integration of mutual information with text-based regularization presents a practical and valuable approach for adapting vision-language models in transductive learning settings.", "requested_changes": "- It is encouraged to include some recent methods such as [2, 3].\n- Experiments on the widely used *base-to-novel* setting [3] would strengthen the evaluation.\n- There are some writing issues:\n   - Add brackets for citations;\n   - Replace `'prompts'' with \"prompts\"  on Page 5;\n   - Resolve the inconsistent notation symbols between Eq. (4) and Tab. 3;\n   - Verify and correct references (e.g., \"Information maximization for few-shot learning\" should be \"Transductive information xx\" on Page 12).\n\n[2] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR 2024\n\n[3] Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages, CVPR 2025", "broader_impact_concerns": "This work presents a fine-tuning method for vision-language models with few immediate ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mbj9OxHkkM", "reviewer": "Reviewer_Naxi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a fine-tuning loss function for CLIP methods, given a labeled dataset of image-prompt pairs, and an unlabeled dataset of images.The work tackles the transductive setting, under which the structure of the whole unlabel dataset is leveraged when fine-tuning, in opposition to the inductive setting, where samples are considered separately. LIMO, the loss function that they propose has three terms: the cross-entropy term, which is computed on the labeled dataset; the mutual information term, which is computed on the unlabeled dataset; and the text-based regularization term, which is proportional to the sum of KL divergences between the distribution of the model and the base distribution.\n\nTransductive methods are common in vision-only settings, but not as much in vision-language settings (with a single prior work that the authors compare to). In the experiments of the paper, LIMO outperforms all the baselines in most settings.\n\n**Strengths**:\nThe method introduced improves over the baselines on a suite of 12 datasets. The experiments of the paper are very complete: the authors ablate with respect to having only some of the three terms in the loss, with respect to three fine-tuning strategies, and with respect to different values of the multipliers of each loss, and with respect to the number of samples per class.\n\n**Weaknesses**: \nThere are no apparent weaknesses. While not highly novel technically, this is a very strong submission that shows state-of-the-art fine-tuning results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The writing is clear, and the experiments are very complete.", "audience_yn": "Yes", "audience_explanation": "Anyone that wants to fine-tune a CLIP model with labeled and unlabeled data may be interested in the paper.", "requested_changes": "- Use \\citep when needed.\n\n- In Table 1 and throughout the text, the authors compare to vision-only fine-tuning methods. Could you describe the vision-only fine-tuning methods more precisely? In particular, in Table 1, does the term “vision-only” refer to the fact that only the visual embedding is trained?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h6rDd6xgFc", "reviewer": "Reviewer_a4bx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contribution summary:\n\n- Novel loss function: The paper proposes a language-aware information maximization objective with three terms:\n\nMutual information between vision inputs and textual class descriptions (conditional + marginal entropy)\nKL divergence regularization that penalizes deviation from zero-shot predictions\nStandard cross-entropy on labeled support samples\n\n- PEFT exploration: The work challenges conventional practices by applying Low-Rank Adaptation (LoRA) to fine-tune internal model parameters in the transductive setting. Prior transductive VLM methods only operated on the output embedding space.\n\n- Strong empirical results: LIMO achieves consistent improvements across 11 datasets, outperforming TransCLIP by ~5% on average in 4-shot and showing gains over inductive methods like CLIP-LoRA.\n\nKey strengths:\n- The method provides a principled generalization of information maximization to VLMs by incorporating textual modality.\n\n- Comprehensive ablations demonstrate each component's contribution.\n\n-Results hold across different vision encoders (ViT-B/16, ViT-B/32, ViT-L/14).\n\n-The approach is orthogonal and can enhance other methods (CoOp, ProLIP).\n\n- Fixed hyperparameters across all experiments suggest good generalizability.\n\nKey weaknesses:\n- Improvements are sometimes modest (2-3% in certain settings), raising questions about practical significance.\n\n- Limited analysis of why PEFT helps in transductive settings beyond empirical demonstration.\n\n- No investigation of which dataset characteristics or task properties lead to larger gains.\n\n- Sensitivity analysis (Table 4) shows high sensitivity to λ_text but is only conducted on one dataset.\n\n- Computational costs are mentioned briefly (5 minutes added) but not thoroughly analyzed across different scales.\n\n- The comparison with TransCLIP-LoRA (Table 5) somewhat conflates the contributions of the loss function versus the fine-tuning strategy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are **generally supported but with some limitations**. The core performance claims are backed by comprehensive experiments across 11 datasets with three shot settings (2/4/16-shot) and multiple backbones (Tables 1, 2, 6, 7), showing consistent improvements over baselines. The ablation studies (Table 3) provide reasonable evidence that each loss component contributes to performance. However, several gaps weaken the evidence: (1) **no statistical significance testing** despite results averaged over only 3 seeds and some improvements being modest (1-2%); (2) **limited hyperparameter analysis**, with sensitivity tests (Table 4) conducted only on one dataset (DTD) despite claiming fixed hyperparameters generalize across all tasks; (3) **unclear attribution** in the TransCLIP-LoRA comparison (Table 5), which conflates the contributions of the loss function versus the fine-tuning strategy; and (4) **lack of analysis** explaining *why* PEFT helps in transductive settings or *which* dataset characteristics lead to larger gains. The evidence is convincing enough to support that LIMO improves performance in most settings, but the depth of understanding about when, why, and by how much (with statistical confidence) remains somewhat superficial.", "audience_yn": "Yes", "audience_explanation": "Yes, the TMLR audience would likely be interested in these findings, as the paper addresses the timely intersection of vision-language models, few-shot learning, and parameter-efficient fine-tuning—all active research areas—and demonstrates that applying PEFT strategies like LoRA to transductive settings yields consistent empirical improvements, challenging the conventional practice of only fine-tuning output embeddings in transductive few-shot learning.", "requested_changes": "- Add error bars to all tables and figures, and provide statistical tests (e.g., paired t-tests) to demonstrate that improvements over baselines are statistically significant, especially given only 3 random seeds and some modest gains (1-3%).\n\n- The claim that λ_ent=10, λ_cond=1, λ_text=0.1 generalize across all datasets is only validated on DTD (Table 4). Test sensitivity on at least 3-5 diverse datasets to substantiate this claim, or acknowledge dataset-specific tuning may be needed.\n\n- In Table 5, the TransCLIP-LoRA comparison conflates two factors (loss function vs. fine-tuning strategy). Add an ablation showing: (a) TransCLIP loss + output layer only, (b) TransCLIP loss + LoRA, (c) LIMO loss + output layer only, (d) LIMO loss + LoRA, to clearly attribute gains.\n\n- Investigate which dataset characteristics (domain shift, fine-grainedness, number of classes) correlate with larger LIMO gains to provide practical guidance.\n\n- Discuss when LIMO underperforms (e.g., StanfordCars in 2-shot Table 1) and provide hypotheses about why.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)