Text Descriptions are Compressive and Invariant Representations for Visual Learning
Authors: Zhili Feng
Anna Bair
J Zico Kolter
Venue: Accepted by TMLR ()

Abstract: Modern image classification is based on directly predicting classes via large discriminative networks, which do not directly contain information about the intuitive visual features that may constitute a classification decision. Recently, work in vision-language models (VLM) such as CLIP has provided ways to specify natural language descriptions of image classes, but typically focuses on providing single descriptions for each class. In this work, we demonstrate that an alternative approach, in line with humans' understanding of multiple visual features per class, can also provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, \textit{SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors)}. This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify each image. Core to our approach is the fact that, information-theoretically, these descriptive features are more invariant to domain shift than traditional image embeddings, even though the VLM training process is not explicitly designed for invariant representation learning. These invariant descriptive features also compose a better input compression scheme. When combined with finetuning, we show that SLR-AVD is able to outperform existing state-of-the-art finetuning approaches in both in-distribution and out-of-distribution tasks.

Published in Transactions on Machine Learning Research (05/2024)
Text Descriptions are Compressive and Invariant Represen- tations for Visual Learning
Zhili Feng
shilif@ondrew.cmm..dl
Machine Learning Department
Carnegie Mellon University
Anna Bair
abair@emma. edu
Machine Learning Department
Carnegie Mellon University
J. Zico Kolter
akoller@cs.cmme www.ccssscmmccc..
Computer Science Department Carnegie Mellon University Bosch Center for AI
Reviewed on OpenReview: https: // openreview. net/fetftee forum? id- id= 705Fyvv0
Abstract
Modern image classification is based on directly predicting classes large discriminative networks, which do not directly contain information about the intuitive visual features that may constitute classification decision. Recently, work in vision-language models (VLM) such as CLIP has provided ways to specify natural language descriptions of image classes, but typically focuses providing single descriptions for each class. In this work, we demonstrate that an alternative approach, in line with humans' understanding of multiple visual features per class, can also provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via large language model (LLM), then uses VLM to translate these descriptions to set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify each image. Core to these descriptive features are more
our approach is the fact that, invariant to domain shift than traditional image embeddings, even though the VLM training process is not explicitly designed for invariant representation learning. These invariant descriptive features also compose better input compression scheme. When combined with finetuning, we show that SLR-AVD able outperform existing saate-of-the-art finetuning approaches in both in-distribution and out-of-distribution tasks.
1 Introduction
Natural language supervised vision-language models (VLMs) like CLIP (Radford al., 2021) create aligned image and text encoders via contrastive training. Unlike auditionally-traieee classification networks, such alignment enables zero-shot image classification by prompting the text encoder with hand-crafted inputs like "a photo of O" then predicting the target via the maximal inner product with the input image embed- ding. However, choosing effective prompts for zero-shot learning remains largely an ad-hoc process: Radford et al. (2021) has added several prompts like "the cartoon O or "art of the O0" aiming to improve ImageNet-R (Hendrycks et al., 2021a) performance, which (somewhat surprisingly) improved standard Im- ageNet accuracy as well. This has led works that attempt to automatically extract relevant prompts from language models (Pratt et al., 2022), including work that uses these models to extract mulliple visual

---

Published in Transactions on Machine Learning Research (05/2024)
descriptors (Menon & Vondrick, 2022) then use the average prediction of these visual descriptions to classify the image.
In the few-shot setting, however, where small amount of training data is available, a number of techniques can further improve classifier performance beyond zero-shot prompting alone. For example, it has become commonplace to finetune zero-shot classifiers via linear probing or other approaches (Kumar et al., 2022), including methods that interpolate between the zero-shot and finetuned classifiers (Wortsman et al., 2022) to achieve better cittof-distributionn robustness. Alternatively, one can also adapt the prompts themselves using this few-shot data, using e.g. techniques from soft prompt tuning (Zhou et al., 2022b), though these learned prompts are not readable, nor are their nearest dictionary projections (Khashabi et al., 2021). Finally, recent work has also looked at ways to combine automaticall iillmattcally--xxtrateeddd prompts using few-shot learning (Yang et al., 2022), though this approach used a very specific learned weighting over such descriptions for interpretability purposes.
In this work, we investigate the visual learning problem with text descriptive features from an information- theoretic perspective. In particular, our motivation comes from two desiderata: compression and invariance (to domain shifts). The information bottleneck perspective encourages representations to compress the input as much as possible while maintaining high mutual information with the labels. On the other hand, the invariance principle favors representations that are less informative about the domains, in particular, the mutual information between the representations and the domain index should be small (Zhao et al., 2022; Li et al., 2021; 2022; Zhao et al., 2019; Arjovsky et al., 2019; Ahuja et al., 2021). Rooted in these information-theoree
principles, we propose a simple and effective method to generate classifiers based upon multiple automatically-extrer Logistic Regression using Augmented Visual Descriptors), uses language model to extract multiple potential visual features of each class, then uses f l-rreullaized logistic regression to fit a sparse linear classifier on top of these visual descriptions. The key observation that supports our method is that these descriptive features retain substantial information about the true labels, yet are not informative about the domain index, making them good invariant representations of the images. Additionally, these descriplive features are better input compressors and thus can generalize better.
visual descriptors of each class. Our new method, SLRR-AVD (Sparse
Once the important visual descriptors are selected, we can also finetune the image encoder with the selected sparse pattern to further improve classification accuracies. Using this procedure, SLR-AVD outperforms baselines in both in-distribution (ID) and out-of-distributie image datasets. Specifically, SLR-AVD on ImageNet and its variations (including ImageNet-R, ImageNet V2, etc.) outperform linear probing with image features by 6.2% to 10.48% varying k-shot from = 1 to 32. When combining SLR-AVD with WISF-FT (Wortsman et al., 2022), in the in-distribution task, our method outperforms standard finetuning by 1.43% with 1-shot, 1.62% with 2-shot, and 1.61% with 4-shot training data. When average over five ImageNet variations, outperform standard finetuning by 0.88% with 1-shot, 0.73% with 2-shot, and 0.64% with 4-shot training data.
(OOD) image classification across a range of
Notation Throughout the paper, we use g(-) to denote the text encoder and s(-) to denote the image encoder. We use t for text tokens and for images. For vector e, subscripted represents the ith entry. We sometimes overload the notation t to represent a vector belonging to a class c, this should be clear from the context. We use C to denote the set of classes. We use I(X;Y) to denote the mutual information between a pair of random variables Y).
2 Related works and motivation
2.1 Prompt tuning in VLMs
Contrastive VLMs aim to minimize the contrastive loss between matching image-text pairs. Let the image embedding be f(p) E M()+M)xd the text embedding be g(t) E R(I+P)xd, where M, P E R denotes the number of tokens created by the transformer encoder, and d E E is the dimension of each token's embedding. Without loss of generality, let the first entry of the embeddings be the (CT.S token, denote as g(t)o- The
2

---

Published in Transactions on Machine Learning Research (05/2024)
Image Encoder
plane
Fine-tune
1n
Visual Descriptors
T2
T
In
has four wheels
"Give me long
Text
T2 I
T3 I2
T2 IN
Sparse Logistic
Freezee sparsity
can fly
descriptors (class)"
Encoder
Regression
pattern
has feathers wings
Tu I
Tur
Th In
Figure An overview of our proposed method. We prompt GPT-3 for a list of visual descriptors for each class and encode these texts. The image embeddings are instantiated these descriptors by taking inner products. For image embedding in Rd, this operation projects it onto RM dimensional space, but it may live in submanifold. We apply sparse logistic regression over all RaxM training data for feature selection. Finally, we freeze the sparsity pattern and finetune both the lincar layer and the image encoder to align the image features with the visual descriptors.
probability of the prediction is then represented as: ply c/p.t) exp((1illooottboffff where te is exp (1(p))oaltttr))///)) (f the zero-shot text prompt for class The class whose prompt has the largest inner product with the image embedding will be the zero-shot prediction. Zhou et al. (2022b) optimizes over the continuous text embedding space for the best prompts. Several follow-up works (Zhou et al., 2022a; Zhu et al., 2022) propose various prompt tuning methods for different task settings. The methods that eventually use g(t.)o are in essence regularized lincar probing where the search space is constrained by the co-domain of g(-)o- Chen et al. (2022) uses local information of the image embedding f1,...,/M+) for optimizing an optimal transport distance between local image information and prompts. Lu al. (2022) learns distributions over prompts for efficient adaptation to downstream recognition tasks. Wen et al. (2023) discusses discrete prompt scarch in the context of text-to-image settings.
Pratt et al. (2022) prompts LLMs for descriptions of each class and shows that these prompts can achieve better zero-shot image classification accuracy. Menon & Vondrick (2022) prompts LLMs to generate visual descriptors for image classification. For each class c, they query GPT-3 using the prompt "What are useful features for distinguishing a {e} a photo?". A score estimated for e given an image pt s(c,p) [[j] ΣtED(E) p(t,p), where D(c) is the set of descriptors for c, and <(t,p) (f()). g(t)o) the inner product between the image and text embeddings. They show this average ensemble can outperform zero-shot classifiers while maintaining interpretability.
Similar to what propose, LaBo (Yang et al., 2022) also considers per-class level descriptions in the few- shot setting, A key difference is that they perform per-class level description filtering through submodular optimization, and they apply softmax to a lincar weight (WW) to ensemble the selected features. On the other hand, we directly select feat ures using sparse logistic regression. Our approach immediately gives both the important features and the coefficients and is statistically optimal under certain sparsity assumptions. One of the potential drawbacks of LaBo is their visual descriptions are filtered per-class level, which can hinder feature sharing between classes. LaBo uses ((W) order to gain probabilistic interpretations of the features, while emphasis robustness only requires W to be sparse.
3

---

Published in Transactions on Machine Learning Research (05/2024)
2.2 Robust fine-tuning of zero-shot models
There are numerous works that study robust finetuning of zero-shot models (Goyal et al., 2022; Kumar et al., 2022; Wortsman al,, 2022). In this work, we adopt the weight interpolation method WISE-FT to improve the OOD test accuracy (Wortsman et al., 2022). In general, let & refer to any set of weights in the network (just the linear layer, linear layer + image encoder, etc). Let the finetuned weight be + Pearnedd and let the zero-shot predictor be Lzs. Wortsman et al. (2022) observes that while Plearned performs better than 2x on ID tasks, it is worse at OOD tasks. Hence they propose to interpolate the two sets of weights as a learned + (1 - 222 This surprisingly simple weight ensemble helps both in-distribution and out-of- distribution tasks. This method also naturally applies to linear probing by simply freezing the CLIP encoder throughout, and only training and interpolating the linear head.
2.3 Compression and Invariant Representation
The term "compression" has been given various meanings under different contexts. Arora et al. (2018) derived a PAC bound where generalization depends on the compression of the model parameters; Moran & Yehudayoff (2016) developed a sample compression scheme where both the features and labels are compressed; information bottleneck (Tishby & Zaslavsky, 2015) proposed to learn representations Z that "compresses" the inputs X by minimizing (X; Z) subject to some constraints. Blier & Ollivier (2018); Blum & Langford (2003) discussed label compression in terms of model description length. In this work, use the term to represent input compression (as in the information bottlencck), such that the features contain little information about the inputs features. From a PAC-learning perspective, a better input compression will lead to smaller generalization error (Shwartz-Ziv Ziv et al., 2018; Galloway et al., 2022), motivating our use of text descriptive features. A complementary idea from information theory is the invariance principle. The idea is that we want to learn representations that are very informative about the labels, but not so about the domain information. Mathematically, the principle encourages max T(Y; Z) AT(Z; A) where A the domain information (Zhao et al., 2022). While it is understood that invariance by itself is insufficient for OOD generalization (Ahuja et al., 2021; Rosenfeld et al., 2020), algorithms based on the invariance principle still achieve competilive results on several 0OD benchmarks (Koh el al., 2021).
3 Proposed method
In this section, we present our proposed method, SLR-AVD, summarized in fig. 1. We will discuss how to generate fcatures, select a sparse set of useful descriptions, and finally, how to align the encoder in detail. We will also state how the proposed method aligns with information--hhoorliee principles.
3.1 Generating visual descriptors
To generate the visual descriptors for ImageNet and its variations, we first use the following prompt to query GPT-3: "Give me long list of descriptions for 0
GPT-3 is quite sensitive to format instruction. Using the prompt "Give me a list" always leads to list format, making it straightforward to select the useful text with regular expressions. Following the method in Menon & Vondrick (2022), we condition these descriptors on the class name, using texts of the form "[c] which has { for cach class and the ith descriptor. For cach class c, we gather Mc descriptors from GPT-3.
Furthermore, for each class, there exists set of hand-crafted prompt templates like "a photo of 0 "an art of O" If there are T total number of such templates, using the class name c, we can generate T total prompt embeddings for each class. We take the average of these prompt embeddings in addition to the aforementioned visual descriptors, leading to Mc + 1 number of prompts for each class. For simplicity, we will refer to the GPT-3 generated text features as visual descriptors (VD), the templates with class names as class prompts (CP), and the union as augmented visual descriptors (AVD). We will also refer to their embeddings using the same names, which should be clear from the context.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Denote M Ecec Mc where C is the set of all classes. The visual descriptors, class prompts, and augmented visual descriptors can be encoded into three matrices Uvd E R Uep E R Uavd E IR (R(++() Given an image embedding = f()) Rd, these three matrices respectively created three sets of new features hyd vvaz, Rep Uep2, and havd a Notice that all three U matrices are fixed and never trained. We call the action of inner product (U,-) as "Instantiating". We will also refer to the instantiated features h as the (text/ /languagg) descriptive features. Given h, we can learn three matrices Wod E RR Wap E CIiii Wavd E R(([x(M(+((1)
Setting Wvd bikdiag ((
)oEc), then Wvd leads to the average ensemble in Menon & [M
Me
|Mal copies
Vondrick (2022). Here blkdiag( .A...., An) creates the block diagonal matrix:
A1 0
0
0
A2
...
blkdiag(A) An))
=
0
0
An
Setting Wep = [ci we get back the zcro-shot classifier W WepUpp- One can naturally merge Wid and Wep into Wavd = [Wra, Wepl. which we use in our proposed method. We note that these three W matrices can all serve zero-shot classifiers. During inference, the prediction is made by picking arg mex (WUz),
3.2 Learning sparse ensemble and aligning the image encoder
The previously defined matrix Uavd can be viewed linear projection of the image embedding onto a M + dimensional semantic space. While this space has high ambient dimension, the projected embeddings live in low-dimensional manifold that has rank less than or equal to that of the image embedding space. By enforcing a sparsity constraint on Wavd, we can select the most important dimensions among havd- We demonstrate that the selected subspace is also robust to natural distribution shifts. Intuitively, we imagine that the large distribution shift in the image embedding space only corresponds to a small shift in the semantic space, since the semantics of images should be invariant. We will later demonstrate with mutual information estimations.
With fixed Uavd, we learn W. with & regularization [| and the cross-entropy loss. Not only does sparse logistic regression select the important features, but it actually also finds the intuitive features. For example, on CIFAR-10, we demonstrate that the selected features are usually the ones that actually describe that class: for each class, we pick the three features with the largest coefficients, and show that the properly descriptive class features are chosen most often; the results are listed in table 9 in the appendix. After obtaining a sparse Wavd, we fix Uavd and the sporsity pattern of W avd: and finetune both the image encoder f, well as the entries in W avd- This process aligns with LP-FT (Kumar et al., 2022), which has theoretical justification for its robustness.
3.3 Text descriptive features are compressive and invariant
Beyond the improvement in performance alone, however, the core of our method relies on the empiri- cal evidence that text descriptive features have many benefits from an information--heorelie perspective. Specifically, we show here that the text descriptive features form more invariant and more compressive rep- resentations of the data than the naive image encoder features. This motivates their use, especially under distribution shift, where we see them outperform the alternatives.
We base our investigation upon two notions: the invariance principle and the information bottleneck. First, the invariance principle from causality (Pcarl, 1995) states that the predictors should only rely on the causes of the labels rather than the spurious features. Following this principle, several mutual information (MI) based OOD generalization works (Arjovsky et al., 2019; Zhao et al., 2022; Li et al., 2021; 2022; Zhao et al.,
5

---

Published in Transactions on Machine Learning Research (05/2024)
2019; Feng et al., 2021; Ahuja et al., 2021) propose that a good feature representation Z would have high mutual information with the label, Y(; but low MT with the domain index, T(Z; so as not to leak information about the domain itself. Closely related is the information bottleneck, which similarly states that a good representation will again have high MI with the label, but low MI with the input X() In recent years, several works have suggested that combining the invariance principle with the information bottleneck can lead to practical and provably strong OOD generalization (Ahuja et al., 2021; Li et al., 2022).
We demonstrate that the text descriptive features essentially obey both the tencts of the invariance principle and the information bottleneck: the extracted text features H have high MI with the labels, but substantially lower MI with both the domain index and the input itself. The features of our framework correspond to the following Markov chain:
10 +
Z U H
(1)
where y ~ Nyy.p~x,,z Z, h~ ~ H, 1 ~ y corresponds to realizations of the true labels, the input images, the image embeddings, the text descriptive features, and the predictions (the capital letters are random variables) respectively. Here W, U, h and H can be subscribed by avd, vd, ep as in section 3. We will use A for the domain index.
By the Data Processing Inequality (DPI, Cover (1999)), we immediately have that T(X;Y) 2 I(Z;Y) Z I(H;Y), Y). Additionally, however, we also observe for the text descriptive features I(H;Y) is nearly as large as ((Z;Y) (i.e., there is not much decrease in the information about the label), but I(H; A) and I(H; ()) are substantially lower than [(Z;A) A) and X() (i.e, the text descriptive features leak much less information about the domain index and the input).
To assess this, we conduct numerical evaluations on CIFAR-10 (Krizhevsky et al., 2009), CIFAR-10.1 (Recht et al., 2018), and 11.2 (Lu et al., 2020). We index these three datasets, denoting the index random variable as A. We compute the image embedding and the instantiated descriptive feature h for every image in these three test sets. To estimate mutual information, we use the SMILE estimator (Song & Ermon, 2019). The numerical estimation is presented in fig. 2. MT is estimated for two sets of text descriptive features: hep ~ Hop and havd ~ Havd. Importantly, Hop should be viewed as a post-processine protpproossing of Havd. Intuitively, we see that Y(: > I(Hava:Y) Y) > [Hepi Y) by DPI. We also see that I(Z A) > [H A) > [Hopi A), which suggests that the text descriptive features h are much more invariant to the distribution shift. The noticcable gap between [(Havd, Y) and T(Hop. explains why it is beneficial to work with text descriptive features beyond vanilla zero-shot classification.
From the information bottleneck perspective, Figure 2 also presents that [(X; Hava) < (X;; 2) by a large margin, we can then interpret Havd as a "better" compression of the input image X, in the sense that it preserves only information in X that is helpful for predicting Y. Of course, this also means that one cannot reconstruct X from Havd better than from Z, although this is an orthogonal goal to ours. Typically better input compressions lead to smaller generalization error. Under mild conditions one can bound the /21xxx2)+1o/(2/0) where n is the
generalization error of feature Z with probability at least 1 6: 8: GenErr S / /2 n
number of training samples (Shwartz-Ziv et al., 2018). Intuitively, if the features have small MI with the inputs, then the perturbation in the input space cannot perturb the features too much, hence constraining the expressiveness of the features. Since I(Havd: X) is significantly smaller than I(Z; X), we can expect a more predictable test performance (compared to the training performance). On the other hand, high ((Hava;)Y) Y) makes sure that the accuracy will not be too low. The synergy of the two notions elucidates the superiority of AVD in the few-shot setting.
In addition, we also measure lected by the E regularizer. By DPI, we immediately have [(Havd Y) 2 I(aand; Y), in fig. 2, we see that numerical evaluation shows nearly identical estimation for these two statistics, indicating that most information about the label is preserved by the sparsity regularization. Mcanwhile by DPI, we also im- mediately have [(Hava A) Z [flavd; A), and both are negligibly small. The interesting part comes in I(Hovdi X)= 2 I(flavd: Once again, DPI tells us the inequality should hold true, and numerical evaluation suggests that Havd has a noticeable drop in its mutual information with X, explaining why they generalize better. We remark that in measuring mutual information with variational methods, there are inevitable
X). ((Faad; and T T(Haadd A), where Aand represent the AVD se-
6

---

Published in Transactions on Machine Learning Research (05/2024)
3.0
1.0
18aami X
2.5
0.8
10 10
2.0
IHHdd
0.6
3 1.5
0.4
1.0
0.3
0.2
0.0
25
50
75
100
125
150
175
200
0.0
25
50
75
100
125
150
175
200
0 50
100
150
200
250
300
Epoch
Epoch
Epoch
Figure 2: The MI estimations at interest. The estimator is variational and we include the whole optimization trajectory to show convergence (hence we can focus on the estimations in the last epoch). Each experiment is run three times and two standard deviations are filled. Left: the MI between a different set of features and the labels. Middle: the MI between different set of features and the domain indices. Right: the MI between a different set of features and the input images.
Table Accuracies on ImageNet. (IN) and its variants. We compare LP vs SLR-AVD.
Shots
k
==
=
=
k=
A 32
Methods
LP
SLR-AVD
LP
SLR-AVD
LP
SLR-AVD
LP
SLR-AVD
LP
SLR-AVD
LP
SLR-AVD
31.51
40.56
44.06
54.16
54.66
63.19
62.33
68.23
67.55
71.40
71.15
73.67
IN-R
35.23
48.88
46.30
61.23
54.50
67.64
59.25
70.58
72.54
64.32
74.53
IN-A
22.52
29.81
27.26
36.96
32.34
42.09
34.88
44.41
45.15
39.19
47.89
IN-V2
26.91
35.12
37.13
47.07
45.92
55.02
52.50
59.15
62.52
61.23
64.75
IN Sketch
16.80
31.03
28.77
33.29
40.73
45.39
Net
19.38
34.11
36.02
42.80
43.67
49.17
Average
8.39
10.48
9.52
7.94
6.20
estimation and optimization errors for example, even though DPI tells (HHo A) (Havdi A), our measurement leads to its contrary, but both values are vanishingly small and hence the difference is insignif- icant. On the other hand, our focus here is the significant difference between Havd (as well as Hava) and Z, that is, AVDs are much more invariant and compressible compared to their image embedding counterparts.
4 Experiment
We demonstrate the improved performance of AVD three settings: zero-shot (no parameter update). linear probing (only the last layer is updated), and full model finetuning. Throughout the experiments, we focus the few-shot setting. We test method on ImageNet, ImageNet-R, ImageNet-V2, ImageNet- A, TmageNett-Sketch, and ObjectNet (Deng et al., 2009; Hendrycks et al., 2021a;b; Recht et al., 2019; Wang et al., 2019; Barbu et al., 2019), demonstrating the superiority of the sparsely learned visual de- scriptors ensemble. We abbreviate ImageNet as IN, and correspondingly their variation, for example, we write ObjectNet as IN-Object. By default, we use the ViT-B/16 model unless otherwise specified. The hand-crafted templates for ImageNet classes contain a set of seven prompts suggested the CLIP github repository
/wt.bl/aac/ooo/oo////aeeaaaaaaaa /oppeao/n/aieeeeee /CLIP): 1. "itap of a . 2. "a bad photo of the O.." 3. "a origami O." " 4. "a photo of the large O.' 5. "a O in video game." 6. "art of the O." 7. "a photo of the small O." This set usually outperforms the original 80 templates in Rad- ford et al. (2021).
For simplicity, we will use the following acronyms for different methods and datasets, also see table for a detailed comparison among each methods. Whenever we combine our method with WISE-FT (Wortsman et al., 2022), will be explicitly mentioned. We defer the hyperparamoter discussions the appendix.
ZS: Zero-shot classification using text embeddings of hand-crafted prompts ensembles.

---

Published in Transactions on Machine Learning Research (05/2024)
IN
IN-R
IN-A
0.5
0.7
0.7
0.6
0.6
0.4
0.3
0.4
0.4
0.3
0.2
0.3
32
16
32
32
IN-V2
IN-Sketch
ObjectNet
0.45
0.5
0.3
0.4
0.2
0.3
0.15
16
32
16
32
16
32
SUR.AVD
prompts
Image
features
Random projection
Figure 3: Few-shot experiments compare several baseline methods vs SLR-AVD. In each subfigure, the x-axis represents the number of shots per class, the y-axis represents test accuracy ImageNet (IN) variants and ObjectNet. Here we consider shot in f1,2, 4, 8, 16, 32} shots per class. SLR-AVD is more sample efficient in the in-distribution reference and also much more robust to several distribution shifts.
Table 2: Acronyms for several methods in consideration. In the column heads, CP: class prompts; VD: visual descriptors; Img: image embeddings.
Features
Parameter updates
CP
VD
Img
Linear
All
7S
ZS-VD
/
ZS-AVD
/
LP
SLR-AVD
FT
SLR-FT-AVD
ZS-VD, ZS-AVD: Zero-shot classification using visual descriptor and augmented visual descriptors, re- spectively.
LP: Linear probing using image embeddings.
SLR-AVD: Sparse logistic regression using AVDs.
FT: Finetuning the image encoder and classification head.
SLR-FT-AVD: Sparse logistic regression with AVD, and then finetune the linear head plus the image encoder with frozen sparsity patterns.
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Accuracies zero-shot, visual descriptors, and augmented visual descriptors on ImageNet (IN) and its variants. ZS-AVD uses descriptors from GPT4 and ZS-AVD2 uses descriptors from Llama2- 1aa-a2tt3B ZS-AVD outperforms all baselines across different datasets.
7S
ZS-VD
ZS-AVD
ZS-AVD2
Waflle-2
Wallle-
Waffee10
IN
68.78
65.89
69.52
69.15
64.36
62.27
60.24
IN-V2
62.23
59.19
62.97
62.86
57.95
56.39
54.54
IN-R
77.72
72.75
77.85
77.88
73.32
72.83
70.68
IN-A
50.64
46.11
50.87
51.07
46.4
44.51
50.65
IN-Sketch
48.38
44.84
48.91
48.45
44.43
42.90
41.57
ObjectNet
54.31
49.60
54.58
53.57
48.73
48.37
46.34
Table 4: WISE-FT weight interpolation for standard ppro-soo//fmetunnig (FT) and SLR-FT-AVD with
optimal a. Accuracies are on ImageNet (IN) and its variations.
Shot
k
k == 2
k=
Method
FT
SLR-FT-AVD
FT
SLR-FT-AVD
FT
SLR-FT-AVD
IN
68.88
70.31
69.59
71.21
70.48
72.09
Average t
1.43
1.62
1.61
IN-R
77.82
78.29
78.13
78.53
78.32
78.59
IN-A
50.09
51.29
50.43
51.51
52.11
52.64
IN-V2
62.32
63.74
63.07
64.37
63.50
65.30
IN -ketch
48.45
49.35
48.75
49.63
48.99
49.92
ObjectNet
54.52
54.94
55.01
54.99
55.77
55.41
Average 1
0.88
0.73
0.64
4.1 Tero-shot with AVDs
As mentioned in section 3.1, we can easily establish zero-shot matrices with AVDs. We set Wvd to be the aforementioned block diagonal form, and Wep to be an identity matrix. We merge them into W... Wadd = [Wad, Wep). Their performances are compared in table ZS-AVD outperforms every zero-shol baseline on all ImageNet variations. We find that simply using VD usually underperforms 28, indicating that the class names one of the strongest prompts. This observation is intuitive during contrastive training the class name itself is likely to show up in the caption the most often, compared to other visual descriptors. One can certainly try to improve 7S-VD results by more carefully prompting GPT-3, gathering descriptors from different data sources/searde engines. Pratt et al. (2022); Yang et al. (2022); Menon & Vondrick (2022) have studied the quality descriptors across different datasets and hyperparameters (e.g. temperature for sampling, etc) settings. Here, we do not further pursue this direction. Instead, we utilize our observation that simply using the merged prompts Wavd already surpasses the best zero-shot classifier. Notice here we have a parameter y that decides how much we weight the zero-shot model. Empirically find that setting Y 5 is sufficient for all datasets. We conduct small-scale experiments to further investigate the influence of difference choice of 7, the GPT prompts, and the GPT sampling hyperparameters. We find these choices typically do not lead to significant deviations in test accuracies unless the generated visual descriptors are too repetitive, see the appendix for details.
CIFAR-10 and its variations
Another interesting bascline zero-shot method we consider is WaffleCLIP (Roth et al., 2023), where the prompt contains the ground truth class name and p random tokens. Here, we compare to p 2, 5, 10, and notice that these random prompts are less informative compared to AVD.

---

Published in Transactions on Machine Learning Research (05/2024)
4.2 Comparison to linear probing
We compare SLR-AVD to LP with {1, 21,2,4,7 4, 8, 16, shots per class. Each experiment is conducted three times with independent random seeds. We report the averaged test accuracy on ImageNet and its distribution shift variations, see fig. 3 for details. Our proposed method outperforms linear probing on all tasks. Detailed accuracies are presented in table In a nutshell, our method outperforms linear probing by 8.39%, 10.48%, 9.52%, 7.94%, 6.54%, 6.20% on h = 1, 2, 4, 8, 16, 32 respectively. When training with large shot size (k 32, 64, 256, 1024), SLR-AVD still shows effectiveness, see fig. 16 in the appendix.
(1) AVD with & reg- ularization. (2) ANN classifier (A 3) with image embeddings. (3) ANN classifier (k 3) with AVD. (4) Randomly project the image emboddings onto I2000 and perform lincar probing. (5) VD. (6) Class prompls with & regularizer.
In addition to linear probing, we additionally consider several baselines in fig. 3:
Our proposed SLR-AVD outperforms all baselines. Specifically, we see that the incorporation of & sparsity is crucial for the improvement of the few-shot performance; the same time, the combination of both class prompts and VD is also important. The comparison to random projection also reveals that the benefit of sparse AVD is not simply due to its smaller dimension: the sparsely chosen AVDs are semantically meaningful, which comes with a stronger regularization effect.
Although learning with visual descriptors significantly outperforms linear probing in the few-shot setting, we should remark that ImageNet and its variations are usually considered "in-distribution" to the CLIP training data. In this case, the zero-shot model itself is usually a very strong baseline, and typically outperforms few-shot models, as can be observed by comparing the results in table 7 and table 1. WISE-FT serves as a strong method to improve both in-distribution and out-of-distribution accuracies. We can apply WISE-FT to any of our existing settings, including SLR-AVD and LP. In particular, we can train a lincar head (and/or image encoder, depending on the setting) Wieamed, and interpolate with the zero-shoi weight. Wzs by laking a convex combination a + (1 Weermed, for a E {ai....,ay, ai, wC can plot that weight ensemble's 's ID and OOD test accuracy. This procodure thus creates an ID-OOD frontier and along the curve, some ensemble excels at both ID and OOD distribution. In the ID-OOD curves in fig. 4, we show the plot of k = 4, 8, 16. SL---VD's ID-OOD curve overwhelms that of LP, indicating that SLR-AVD is better at both ID and OOD tasks.
J. a. We are free to vary a. Then for each
4.3 Comparison to finetuning
We compare WISE-FT where we additionally interpolate the image encoder, to a weight interpolation between SUR-FT-AVD and 7S-AVD (WISE-FT+SIRRAVDD). The ID-OOD frontier is presented in fig. 4 and the accuracies are reported in table 4.
On the ID task, SUSE-FF++LRR-TTAAEE spectively with A = 1,2,4 shot training data. Averaging over 5 distribution shift datasets, with optimal a, WISE-FTT+SLB--T-AV
outperforms vanilla WISE-FT by 1.43%, 1.62%, and 1.61% re-
outperforms vanilla WISE-FT by 0.88%, 0.73%, and 0.64% respectively for k k= 1,2,4. 4. The optimal is picked independently for each method on each dalaset.
4.4 Comparison to CoOp
CoOp and CoCoOp both tackle the few-shot learning problem from a prompt optimization perspective (Zhou et al., 2022b;a). Since the latter requires an additional neural adapter module and its benefit mostly lics in generalization to unseen classes, we only compare to CoOp in this work as it is more relevant to our setting, CoOp learns the prefix of "[prefix] {classname]" in the continuous token embedding space. The benefit of CoOp is that it operates in continuous space, hence one can optimize using standard backpropagation. On the other hand, due to the requirement of backprop, CoOp stores a large computation graph, hence memory-efficiency is a big advantage of SLR-AVD over CoOp.
When implementing CoOp, we choose a prefix of length 16 and do not use a suffix. The prefix is fixed for all classes We train with Adam for 20 epochs, setting the batch size to 512. This gives us comparable results to those of the original paper.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Img
WISE-FT+LP
SLR-AVD
WISE-FT+SLR-AD
60
60
accuracy (Average)
56
58
50
0OD accuracy (Average)
56
54
52
48
50-
52
46
44
0OD
48
50
42
54
56
62
64
56
68
70
72
63
65
89
71
73
68
70
72
74
accuracy (ImageNet),
accuracy (ImageNet),
accuracy (ImageNet),
Img
Img
WISE-FT Img
SLR-FT-AVD
WISE-FT+SLR-FTAVD
accuracy [Average)
0.154
0.592
6.9/5
0.150
0.592
0.050
0.590
0.5425
0.500
6.500
0.166
53575
0.550
eils ID accuracy (ImageNet), 1 shot
9.708
2a05 0.72
ID accuracy (lmageNet), 2 shot
ID accuracy (ImageNet), shot
Figure 4: Top: ID-OOD accuracy curve of WISE-FT+LP vs WISE.FT+SLR-AVD. ID is tested on Ima- geNet, and OOD averaged over ImageNet variations. Experiments with 84 16]-shots are presented. Each accuracy averaged over 3 runs. We can see that our proposed method overwhelms LP in all cases. Bottom: ID-0ÓD accuracy curve of WISE-FT performed. ID is tested on ImageNet, and OOD is averaged over 5 ImageNet variations. Experiments with [1,2, 4j-shots are presented. Each accuracy is averaged over 3 runs. We can see that our proposed method overwhelms WISE-FT in all cases.
WISE-FT--SLR-FT-AVD full parameter updates are
Table 5: Accuracies of CoOp and SLR-AVD on ImageNet. Both methods are incorporated with WISE-FT.
The results are reported with the best interpolation.
Shots CoOp SLR-AVD
2
4
8
16
32
69.54
69.73
70.14
70.55
71.11
71.85
69.83
70.33
71.18
72.37
73.45
74.34
A
+0.28
+ 0.61
1.04
+ 1.82
+ 2.33
+2.49
For a. fair comparison, we compare WISE-FT+CoOp to WISE-FT+SLR-AVD. The vision backbone used is ViT-B/16 for both methods. We use the ZS weight for CoOp WISE-FT interpolation. The results are reported in table 5, and we pick the interpolation that yields the best test accuracy. Our proposed method oulperforms CoOp in all cases.
4.5 Comparison to LaBo
LaBo (Yang et al., 2022) proposes a similar approach to ours, where they first achieve many descriptors from various sources, then perform feature selection based on submodular optimization, and finally they learn classification layer using the selected features. There are two inconveniences that come with submodular optimization based method: one is the users have to design proper submodular function, and the other is
11

---

Published in Transactions on Machine Learning Research (05/2024)
the feature selection and learning process are separated, imposing additional effort. For fair comparison, we compare to LaBo using CLIP VIT-I,/14 backbone ImageNet. We take the same set of concepts collected in Yang et al. (2022), randomly selected six features for each class, and train SLR-AVD using the subsampled features. While the original LaBo paper select 50000 features for 1000 classes, SLR-AVD selects 5961, 5890, 6678, 6959, and 6990 out of the total 7000 features, for k = 2, 4,8, 8, 16, respectively. Notice that the actual matrix avd E IR x000x 7000 is very sparse - roughly 98% to 99% percent of the entrics are zero. The results are presented in table SLR-AVD has outperformed both baselines.
Table 6: comparison among LP, LaBo, and SLR-AVD. The accuracies of LP and LaBo are gathered from Yang al. (2022), the image features are extracted from CLIP-L/14 for all models. This evaluation is done Net.
Shots
1
2
4
8
16
LP
42.25
55.71
64.80
71.23
75.08
LaBo
51.09
57.43
62.94
68.45
72.60
SLR-AVD
63.09
71.26
75.46
77.29
78.88
4.6 WILDS Benchmark
We further conduct experiments on the WILDS benchmark (Koh et al., 2021), specifically, iWildCam (Beery et al., 2021) and FMoW (Christie et al., 2018). Following the norm, on iWildCam, we evaluate based on the macro F1 for both ID) and OOD test. data; FMoW, evaluate based accuracy for IDD test data, and "worst geographical region" accuracy train with the full dataset instead of the few-shot data. The results are presented in table 7. We see that AVD does not always improve the performance in the zero-shot case, there is an improvement in iWildCam but a decrease in FMoW. On the other hand, SLR-AVD outperforms LP on iWildCam significantly, and perform on par with LP on FMoW. Meanwhile, learning with GPT-4 generated AVD does not have strong edge over learning with those generated by a less capable language model, IJama2-13B-cha (Touvron et al., 2023). A very compelling reason is that contrastive multi-modal objective function ignores subtle semantic differences, for example, CLIP treats texts like bag of words (Yuksckgonul et al., 2022). We leave further investigation to future research.
OOD test data, see Koh et al. (2021) for details. In this section,
4.7 Ablation study
In this section, we try understand what makes SLR-AVD outperform other basclines. In particular, would like to investigate two factors: the importance of sparse regularization and the language models that generate the AVDs. We conduct these results using the iWildCam dataset, the results are presented in table 8. We see that while & regularization does improve the performance on the ID data, it is not necessarily more robust to the OOD dataset, while & improves both ID and OOD performance. On the other hand, a less capable LLM can still generate reasonable AVD that match the performance of AVD generated by GPT-4.
Table Evaluation on WILDS benchmark. 7S-AVD, and SLR-AVD, use descriptors from GPT-4, ZS-AVD2 and SLR-AVD2 use descriptors from JJama2-13B-chas
zs
ZS-AVD,
ZS-AVD2
LP
SLR-AVD,
SLR-AVD,
WWilCCam ID
10.84
13.12
11.87
40.20
43.06
43.61
WWilCCam OOD
8.82
11.32
11.04
28.88
31.91
31.09
FMoW ID FMoW OOD
20.45
20.31
21.28
46.87
47.21
46.63
18.92
17.79
19.34
29.00
28.89
30.00
12

---

Published in Transactions on Machine Learning Research (05/2024)
Table 8: Ablation studies on the iWildCam dataset. SLR-AVD; trained with GPT-4 generated AVD, and ALR-AVD2 with Llama2-13B-chat generated AVD.
LP
fv-AD
SLR-AVD; SLR-AVD2
ID
40.20
42.95
43.06
43.61
OOD
28.88
26.93
31.91
31.09
5 Conclusion
Motivated by the invariance principle and information bottleneck, we present how to leverage descriptive features for image learning in the few-shot setting robustly. These descriptive features can be easily obtained from LLMs. Applying sparse logistic regression then successfully selects the important features, which turn out to be intuitive. Our proposed method outperforms linear probing and standard finetuning in both ID and OOD tasks, with or without combining with WISE-FT. This approach helps us further understand the CLIP embodding space and how the semantics serve as strong robust prior. Moving forward, it is important to understand and quantify the robustness of the visual descriptors' space and compare it to the image embedding space statistically. On the practical side, this work aligns image encoders to fixed text encoder; is valuable to study how to simultaneously align both encoders in a robust way.
References
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Channthrittophe Gagnon-Audet, Yoshua Bengio, loan- nis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for dustofidistiibution generalization. Advances Neural Information Processing Systems, 34:3438 34400, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint Xiv: 02899,,, 2019.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pp. 254 263. PMLR, 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint 2tf::2105004400 2021.
Lóonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.
Avrim Blum and John Langford. Pac-mdl bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Keruel 2003, Washinglon, DC, USA, August 24-27, 2003. Proceedings, pp. 344-354 357. Springer, 2003.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang, Prompt learning with optimal transport for vision-language models. arXiv preprint arXic:2210.011 01253, 2022.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Aaron Defazio, Francis Bach, and Simon Lacosto-Julien. Saga: fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pollern recognilion, pp. Teee, 2009.
Zhili Feng, Shaobo Han, and Simon S Du. Provable adaptation across multiway domains via representation learning. arXiv preprint 7XXi:2206606657 2021.
Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Toannou, and Graham W Taylor. Bounding generalization error with input compression: An empirical study with infinite-width networks. arXiv preprint ar XXtt:2207.09408, 2022.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint orX.v:2212.00638, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out- of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340 8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial ex- amples. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognilion, pp- 15262-15271 2021b.
Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sameer Singh, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, et al. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. arXiv preprint wrXiv:211.0083/2 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in- the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghmmathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform dustof-distributton. arXiv preprint arXiv:2202. 10054, 2022.
Bo Li, Yezhen Wang, Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for semi-supervised domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1104-1113, 2021.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7407, 2022.
Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness in Deep Learning, volume 5, pp. 15, 2020.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Lin, and Xinmei Tian. Prompt distribution learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5206 5215, 2022.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint Xrxi::2210. 07183, 2022.
Shay Moran and Amir Yehudayoff. Sample compression schemes for ve classes. Journal of the ACM (JACM), 63(3):1 100:1-10,22000 2016.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, 1995.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for vero-shot image classification. arXiv preprint or arxi::200 03320, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Benjamin Recht, Rebecca Roelols, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint SS8::18060004550 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pp. 5389 5400. PMLR, 2019.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint Xrxi022010. 057611,, 2020.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Wallling around for performance: Visual classification with random words and broad concepts. arXiv preprint arXiv:2306.07282 2023.
Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization in deep neural networks, 2018.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estima- tors. arXiv preprint drXiv:1910.0620 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xixiv preprint orXiv:2307.09288 2023.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made casy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03664 2023.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. Tn International Conference on Machine Learning, pp- 11205- PMLR, 2021.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7959-7971, 2022.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. arXiv preprint arXie:2211.11158 2022.
Mert Yuksekgomll, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what do about it? In The Eleventh International Conference Learning Representations, 2022.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant represen- tations for domain adaptation. In International conference on machine learning, pp. 7523-7532. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. The Journal of Machine Learning Research, 23(1):15356 15404, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- vilion, pp. 16816 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwci Liu. Learning to prompt for vision-lagguage models. International Journal of Compuler Vision, 130(9))2337-293) 2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompl tuning, XrXiv preprint ar Xiv: 14866, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2024)
A Miscellaneous
Hyperparameter For ImageNet and its variations, we fix a set 6804 augmented visual descriptors. The hyperparameters are swept over disjoint training and validation sets of size 20 per class for LPP and SLR-AVD. For & regularization, its non-smoothnes makes it notoriously hard for auto-diiferentiationn To circumvent the smoothness issue, we apply the GPU implementation (Wong et al., 2021) of variance- proximal gradient method SAGA (Defazio et al., 2014). We adopt the regularization path approach, in which the solver optimizes over 100 regularization strengths X1 A2 ... A100- Here we set tY to be the strength that returns a model that uses none of the features, and λι00 = 0.1 x λ1. For LP,, we always use & regularization, we use L-BFGS implemented by scikit-learn, and search for the regularization strength over 100 grids between 0.5 and 6. All the Äs are ovenly spread in the log-space', For FT and SLR-FT-AVD, we select hyperparamoters of size 4 per class. The batch size is fixed be 512 and the number of epochs is fixed to be 10. We always optimize with AdamW, and choose cosine rate scheduler with warm-ups. We randomly select learning rate in [le - 8, 3e - 5], weight decay in [0.1, 0.12], and warm up steps in {0, 50, 500), for 20 trials. The chosen parameters are then fixed throughout all experiments.
using training and validation
Details about U, W
We include a more detailed description of U, W:
(Urd) : (Ued)as
(Upp)1
Und cp_
Und
.ee
Uavd
where each U, Rd
(U (Cop)1c1
1 0
0
1
W vd
W
W avd [Wod W
where 1 E R.M.
0
Here assume each class has the number of clescriptors. The general case be easily derived. Wop E |iclii is diagonal matrix. W is block-diagonal with |C] number of rows; each of its block has row vector of size M,, which amounts to total of Eeec Mc = M columns.
What AVDs are selected On CIFAR-10, we show that the selected AVDs via sparse logistic regression are intuitive, see table 9.
In python mampy.ooggpaeemmaa..nngggo 1o22o)) 100)
17

---

Published in Transactions on Machine Learning Research (05/2024)
Table 9: Features selected when trained with & norm on C1FAR-10. The selected important features for each class are intuitive. Notice that the feature selection method does not restrict the candidates be that particular class's descriptors.
Classes
Features
airplancs which has anticollision lights
airplanes photo of airplanes
airplanes which has overhead storage bins
cars which has body kit cars which has bumpers cars which has wheel arch trim
cars
birds which has leg color birds which has flight silhouette birds which has eye color
birds
cats which has pink tongue cats which has pink nose cats which has slit pupils
cats
deer which has large facial glands deer which has long, tufted hair the neck and shoulders deer which has short, curved antlers
deer
dogs which has silky fur dogs which has pattern dogs which has floppy cars
dogs
frogs which has large, bulging eyes frogs which has ridged or wartylooking skin frogs which has fold of skin along the back
frogs
horses which has hooves horses which has temperament horses which has intelligence
horses
ships which has lifeboats ships which has bridge
ships
ships which has bow
trucks which has trailersway control trucks which has grille
trucks
trucks which has lift kits
B More experiment results
As recap, we use the following acronyms for different methods and datascts. Also see table 2. ZS: Zero-shot classification using text embeddings of hand-crafted prompts ensembles.
ZS-VD, ZS-AVD: Zero-shot classification using visual descriptor and augmented visual descriptors, re- spectively.
LP: Linear probing using image embeddings.
SLR-AVD: Sparse logistic regression using AVDs.
FT: Finetuning the image encoder and classification head.
18

---

Published in Transactions on Machine Learning Research (05/2024)
SLR-FT-AVD: Sparse logistic regression with AVD, and then finetune the linear head plus the image encoder with frozen sparsity patterns.
WISE-FT:
Weight ensemble using ZS and FT.
WISE-FT+LP: Weight ensemble using ZS and LP (so only the last linear layer trained).
WISE-FT+SLR-AVD: Weight ensemble using ZS-AVD and SLR-AVD (so only the last linear layer is trained).
WISE-FT-SIL-PT-AADD. Weight ensemble using SLR-FT-AVD and 7S-AVD. This is short for WISE- FTV-ST--VD--T..
IN: ImageNet.
IN-R: ImageNet-R.
IN-A: ImageNet-A.
IN-V2: ImageNet
IN-Sketch: ImageNet-Sketch.
The dataset-wise ID-OOD curves of LP vs SLR-AVD on IN-A, IN-R, IN-V2, IN-Sketch, and ObjectNet are listed figs. to 9, respectively.
FT
Paess shot
Cur feroodd
0OD sccuracy Uinqgueee
000 (mageeve
100 soc currey Imaagueerr...
47
57
-7
31
20
51
55
05
45
55
55
52
1D sccurscy (ImageNet),cgm Shot 1
accuracy (maggNNe), shot
ID accuracy (Imagekeet), shot
WSL/Ting Pratseiing
aesinese))
50
000
accuragy (ImageNet), shot
accuracy OmageNel,
accuracy (ImageNett).
Figure 5: ID-0OD curves of LP vs SLR-AVD IN-A. k 1, 2, 4, 8, 16, 32.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Caer shat tnd
CLr burooed
320ue 27 Umsggeee-1
75
60
50
83
19
000
31
20
41
65
51
55
05
71
55
54
57
64
10
accuracy (mageNet, Shot
accurscy ImageNet), shot
10 acturacy (Imagevee), shot
WSL/Ting fratawdin
Desinete))
WSG-T CU-Tunaan
accuracy dmageNet-0
69
0000 2220230
accuracy (ImageNet), shot
accuracy UmageNet),
accuracy (ImageNiet).cm shott
Figure 6: ID-OOD curves of LP vs SLR-AVD IN-R. k 1,2 4, 8, 16, 32.
MDd FT
CLR ferrodd
000 sccursiy lim g2N--22
61
80
54
51
34
31
20
41
45
51
55
05
46
55
55
52
ID accurscy (ImageNet),cgm Shot
accuracy (mageNee), shot
ID accuracy (Imageveet), shot
WSL/Ting
aesirese))
61
DOD accuraccy Umuigetet-v22 -2)
01
3 03
02
68
01
23
54
03
05
70
accurary (ImageNet), shot
accuracy OmageNel,
accuracy (ImageNett)
Figure 7: ID-0OD curves of LP vs SLR-AVD IN-V2. k= 1, 2, 8, 16, 32.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Ca shat And
000 Sccurcyy Uinazeeet 2lettc
43
37
31
30
26-
000
31
20
41
65
5L
55
05
71
55
55
64 10 accuracy (Imagevee), shot
52
10
sccuracy (mageNet, Shot
accurscy (mageNet), shot
WSE/Ting fratawing
Desinete)
may
8 44
45
82
60
16
000
05
70
accuracy (ImageNet), shot
accuracy UmageNet), shot
(Batcuryyy Omagakee) shot
Figure 8: ID-OOD curves of LP SLR-AVD on IN-Sketch. k 1,,,,,4,8 16,
302240 NSS-TA
NSC WPTHER
ABGIT
GL ang
WSGHT AMC
000 acarray Giaggeee ab0cc
object
04
47
29
900 accurrkyy
-7
29
000
35
accuracy (mageNee,
accuracy (maageNe),
accuracy (Imagetvet),
CUP3W0-rIN AD
Wokee mng
CLF aropprat
GLF CiPaaarees
000,800 sccurasy aww.raa//mmaaaaaaa Umapptttt (Imapeeet.t)eer
abject)
(ImageNet), shot
(mageNet)
44
abaccurocy 3magehee..
Figure 9: ID-OOD curves of LP vs SLR-AVD on ObjectNet. k= 1,,,,,,,,6,, 16,32.
The dataset-wise ID-OOD curves of WISE-FT WISE-FT+SIR-FT-ADD on IN-A, IN-R, IN-V2, IN- Sketch, and Net are listed in figs. 10 to 14, respectively.
21

---

Published in Transactions on Machine Learning Research (05/2024)
AIR ad
0.
accura (mnogates))
000 Scoursry (migetee))
6.20
2.04
0.00
0:32
c.00
0825 06003 80835 saw sxess soda ouars 2000 1.2
In-iistrbutton accuracy limageMet).
indistribution accuracy (Imageleet, shoto2
5,00
066
6.55
0.10
1.0 9000 6.00 6SE 0.0 12te 6n5 0009 In-detribution acourscy Lmmgeeett,
Figure 10: ID-OOD curves of WISE-FT vs WISE-FT-SLR-PT-AVD IN-A. k = 1,2.4.
w al
acouracy (ma0000000
8.00
0XP ecturary (mageeeett)
12
0.144
0,302
k.7
6.78
0:00
8.7
6.7
0.77
1020 nebes snics cae neiss wels vos
0998
8:50
6.70
0.10
9.00 8.0 t3t c71a aio
distribution accuracy (ImageNet). shot=1
inddstributio accuracy (Imagelietl, shot=2
nn-detribution acouracy (ImageNc), shot-4
Figure 11: ID-OOD curves of WISE-FT vs WISE-FT-SLEFT--AD IN-R. k 1,2.4.
C1P cnn
OPP Beciure Mourtar
Sraeeett and
and
Ourth an
A-013
0:00 UCCuracy [l Veeeeer
acourscy
0::00
4.009
1,00 cel00 Lairs octoo caloo ondor onlos and0 0710
5,00
6.00
066
6.85
0.10
5,00 5000 6.00 CSN 0X0 17t 608 0000
In-distribttion accuracy (ImageNet), shot=1
In-dssrriuttion accuracy (Imagelieti, shots22
In-detribution acourscy (ImageNet),
Figure 12: ID-OOD curves of WISE-FT vs WISE-ET-SIREFT-AVD IN-V2. & 1,2,4.
awcuracy ClmageN (ImapelleSeecc))) eShecc))
we mn
ODD eccumicy (ImageNN (tSeecc)))
WEE le
4/82
DAN
Bas s6l0e salio cas cols wods overs 7000 8.300
9.10 ditributionn accuracy (ImageNiett, sho0-2
3000
6981
068
6/50
5.00 0000 300 5008 6.00 6213 0710 5/08
distribution accuracy (ImageNet). shot=1
Indtsrbuution accuracy (ImageNet). shot0-
Figure 13: ID-OOD curves of WISE-FT WISE-FT-SLR-TT-VVD IN-Sketch. k 1,2,4
22

---

Published in Transactions on Machine Learning Research (05/2024)
ASB ad
accurscy (Cb/occMe))
000 eccurscy (Chjectte))
3.47
182 0603 8083 caw sxess soda eueis 2000 1.305
5.00
6.04
066
6.85
0.10
1,00 9000 8.00 L.X. 0.05 62t 6015 0000
Indiittibution accuracy limageNet).
In-disriautiio accuracy (ImageNeet,
acourscy LimageNet,,
Figure 14: ID-OOD curves of WISE-FT vs WISE-FT-SLR-PT-AVD Nej. A == 1,
The detailed accuracies of WISE-FT vs WISE-FT-SLR-FT-ADD with different choices of given in table 13 (ID) and table 12 (OOD). 0 corresponds to zero-shot accuracy, and 1 corresponds to full fine-tuned model. The results in the same setting with only the last linear layer trained are presented in table 10 (TD) and table 11 (OOD).
Table 10: Accuracies on ImageNet with difference choices of a. We compare LP vs SLR-AVD.
Shots
k
h: 2
k:
k
k:
16
k 32
Methods
LP
AVD
LP
AVD
LP
AVD
LP
AVD
LP
AVD
LP
AVD
0.0000
68.78
69.53
68.78
69.53
68.78
69.53
68.78
69.53
68.78
69.53
68.78
69.53
0.0001
68.79
69.54
68.83
69.54
68.85
69.54
68.87
69.55
68.92
69.55
68.94
69.55
0.0002
68.81
69.54
68.88
69.55
68.91
69.55
68.93
69.56
69.02
69.56
69.07
69.56
0.0004
68.86
69.56
68.96
69.58
69.03
69.58
69.09
69.59
69. 69.24
69.59
69.35
69.59
0.0008
68.94
69.59
69.14
69.63
69.26
69.64
69.40
69.64
69.65
69.66
69.84
69.68
0.0016
69.06
69.62
69.38
69.72
69.63
69.74
69.93
69.79
70.36
69.79
70.70
69.83
0.0032
69.22
69.72
69.77 73
69.86
70.17
69.93
70.73
70.00
71.40
70.07
72.01
70.08
0.0063
68.99
69.81
69.69
70.06
70.71
70.22
71.54
70.40
72.52
70.50
73.38
70.51
0.0126
67.31
69.83
68.04
70.33
70.35
70.75
71.65
71.09
73.10
71.25
74.22
71.27
0.0251
62.15
69.26
63.91
70.33
68.
71.18
70.42
71.88
72.45
72.23
74.12
72.36
0.0501
53.51
66.75
57.69
69.30
64.35
71.17
68.18
72.37
71.
73.08
73.30
73.44
0.1000
44.41
61.19
51.99
66.38
60.59
69.99
65.81
71.96
69.62
73.45
72.46
74.26
0,2000
37.91
53.68
47.94
62.06
57.59
67.75
64.13
70.95
68.60
73.09
71.83
74.34
0.3000
35.35
49.10
16.43 16.43
59.57
56.41
66.32
63.42
70.15
68.19
72.62
71.58
74.21
4000
34.05
46.60
45.63
57.93
55.83
65.43
63.06
69.52
67.99
72.22
71.44
74.06
0.5000
33.22
44.72
45.12
56.82
55.44
61.80
62.82
69.14
67.85
72.00
71.33
73.91
0.6000
32.67
44.79
55.98
55.18
64.30
62.66
68.89
67.75
71.81
71.27
73.85
0. 7000
32.27
42.47
44.52
55.39
54.99
63.92
62.54
68
67.70
71.67
71.23
73.77
0.8000
31.96
41.71
44.31
54.87
54.85
63.62
62.45
68.51
67.62
71.57
71.20
73.73
0.9000
31.69
41.11
44.17
54.47
541.4
63.39
62,39
68.36
67.59
71.48
71.17
73.71
1.0000
31.51
40.56
44.06
54.16
54..6
63.19
62.33
68.23
67.55
71.40
71.15
73.67
23

---

Published in Transactions on Machine Learning Research (05/2024)
Table 11: Accuracies on ImageNet variations with difference choices of a. We compare LP vs SLR-AVD. The results are averaged over all ImageNet variations.
Shots
k
h:
k
k
16
32
Methods
LP
AVD
LP
AVD
LP
AVD
LP
AVD
LP
AVD
LP
AVD
0.0000
58.66
59.03
58.66
59.03
58.66
59.03
58.66
59.03
58.66
59.03
58.66
59.03
0.0001
58.67
59.03
58.68
59.03
58.69
59.04
58.70
59.04
58.70
59.04
58.71
59.04
0.0002
58.68
59.02
58.68
59.04
58. 70
59.04
58.77 58.71
59.04
58. 70
59.04
58.73
59.04
0.0004
58.69
59.03
58.
59.04
58.71
59.04
58.73
59.03
58.75
59.05
58.81
59.05
0.0008
58.70
59.03
58.75
59.06
58.81
59.06
58.84
59.06
58.88
59.06
58.99
59.06
0.0016
58.72
59.06
58.81
59.08
58.90
59.09
59.03
59.11
59.07
59.08
59.23
59.11
0.0032
58.69
59.07
58.
59.13
58.97
59.18
59.15
59.19
59.20
59.16
59.47
59.21
0.0063
58.30
59.10
58.27
59.14
58.79
59.23
58.89
59.31
58.98
59.28
59.40
59.37
0.0126
56.77
58.90
56.41
59.16
57.40
59.30
57.56
59.45
57.79
59.49
58.34
59.59
0.0251
52.55
58.09
51.93
58.70
54.30
59.14
5-1.81
59.38
55.47
59.55
56.
59.78
0.0501
45.05
55.46
45.0
57.11
49.68
58.20
51.10
58.73
52.54
59.13
53.86
59.76
0.1000
36.63
50.25
39.71
53.88
45.30
56.12
47 77
57.21
50.03
58.05
51.87
59.25
0,2000
30.30
43.39
35.56
42.08
53.33
45.47
55.21
48.38
56.46
50.61
58.29
0.3000
27.83
39.70
33.94
17.00
10.79
51.76
14.55
5/102
17.73
55.60
50.12
57.70
4000
26.59
37.37
33.09
45.44
40.10
50.76
14.09
53.25
17.39
55.02
49.88
57.31
0.5000
25.82
35.84
32.57
44.43
39.67
50.12
43.79
52.74
47.18
54.63
49.71
57.04
0,6000
25.28
34.73
32.22
43.70
39.40
49.63
43.59
52.36
47.03
54.37
49.62
56.82
0. 0.7000
24.87
33.94
31.99
43.16
39.18
49.25
43.45
52.07
46.92
54.16
49.55
56.67
0.8000
24.59
33.33
31.80
42.72
39.03
48.95
43.34
51.86
46.83
54.02
49.49
56.54
0.9000
24.36
32.85
42.38
38.91
48.71
43.26
51.67
46.77
53.88
49.44
56.45
1.0000
24.17
32.42
31.53
42.08
38.80
48.51
43.19
51.54
46.72
53.79
49.41
56.35
Table 12: Accuracies on ImageNet variations with difference choice of a. We compare WISE-FT combined with FT and SLR--T--VD, respecively. The results averaged over 5 ImageNet variations.
Shot
k =
= 2
k h=
Method
FT
SLR-FT-AVD
FT
SLR-FT-AVD
FT
SLR-FT-AVD
0.00
58.39
59.07
58.39
59.21
58.39
59.33
0.02
58.40
59.09
58.40
59.22
58.45
59.39
0.04
58.40
59.11
58.44
59.27
58.53
59.45
0.06
58.42
59.11
58.46
59.31
58.62
59.49
0.08
58.42
59.12
58.48
59.33
58.69
59.58
0.10
58.44
59.14
58.51
59.38
58.73
59.63
0.20
58.46
59.20
58.65
59.48
59.07
59.97
0.40
58.50
59.29
58.82
59.68
59.40
60.24
0.60
58.55
59.38
58.97
59.80
59.60
60.37
0.80
58.56
59.44
58.98
59.75
59.68
60.19
1.00
58.58
59.49
58.98
59.74
59.50
59.88
24

---

Published in Transactions on Machine Learning Research (05/2024)
Accuracies on ImageNet with difference choice of a. We compare WISE-FT combined with FT and SLR-FT-AVD, respecively.
Table 13:
Shot
k = 1
k ==
k =
4
Method
FT
SLR-FT-AVD
FT
SLR-FT-AVD
FT
SLR-FT-AVD
0.00
68.33
69.71
68.33
70.30
68.33
71.13
0.02
68.33
69.73
68.37
70.33
68.44
71.19
0.04
68.37
69.76
68.45
70.35
68.53
71.24
0.06
68.39
69.78
68.50
70.11
68.60
71.30
0.08
68.39
69.82
68.53
70.49
68.68
71.36
0.10
68.43
69.83
68.55
70.51
68.80
71.12
0.20
68.51
69.92
68.74
70.68
69.20
71.64
0.40
68.59
70.04
69.04
70.91
69.89
71.93
0.60
68.70
70.18
69.29
71.12
70.28
72.09
0.80
68.78
70.27
69.47
71.21
70.48
71.96
1.00
68.88
70.31
69.59
71.21
70.42
71.70
Choosing y and LLM prompting We consider another prompt "Give me 100 useful visual features for distinguishing ( a photo", and use it with frequency penalty (FP) D. FP 0.1 in ② 3 uses the GPT3 prompts in the main text with FP. Unless otherwise specified, other experiments use 7 = M.+I and FP 0.1, and the GPT prompts in the main text. We find that the GPT3 prompt itself does not matter much as FP it is more important to generate more diverse set of VD. Note in the main text we set 5, this is because on ImageNet it is hard to guarantee the same M, across classes (due to excess number of classes), hence we use large y to enforee ZS-AVD relies mostly on the strong class prompts. In this ablation study, we enforce GPT to give 100 VDs per class so we can simply average over them.
Table 14: ZS ablation on and GPT prompts.
Y or prompts
1/(M, 1)
5
CIFARIO
91.51
91.19
91.16
91.25
91.42
90.44
CIFAR10.1
86.35
85.90
85.90
85.90
85.60
85.40
CIFAR10.2
83.80
83.10
83.10
83.20
84.20
82.50
Here we include comparison among SIR--VDD COOp+WISE-FT, Wall-CLIP-WISE-T. and sparse class prompts (SCP)+VV WISE-FT. For SCP, we have 1000 classes and 7 templates. We create a text embedding for each class with each template, and get in total 7000 class prompts. On the few-shot training data, we learn a sparse combination of these 7000 features with an lj-reuularized cross-entropy loss. Finally, perform WISE-FT with the zeroshot weights.
25

---

Published in Transactions on Machine Learning Research (05/2024)
Car Instared ArD
CJ Instaned Coop
CP Vallnd Wath
G. Frezured Spar prompla
0OD accuracy (ImageNee- avgl
23
OOD (Imagr
50
1
40
45
35
60-
0OD
45
50
55
60
65
70
54
59
69
60 62 63 64 65 66 68 69 70
DD accuracy (lmageNet), shot 1
10 accurary (ImageNet), shot
[D accuracy (mmggNNet) shot
OCD accuracy (imageNet: avg)
80
0OD accuracy (inageNett-aug
-9
0OD
69 70
72
70 71
73
70 71
74 (maggeNet), shot 32
(Imagelvet), shot
10
(ImageNet), shot
Figure 15: ID-OOD curves of SLR-AVD vs CoOp, Waffle, and the class prompts on average over ImageNet variations. & = 1,2,4,8, 16, 32.
Performance of SLR with AVD vs image features when & is large. The numbers are averaged over two random runs.
0.63
Imags fndtanes
68
Augeestii doorispees
Aughented uuual decotiiee
3.5
0.5
0.83
3.89
0.2
1004
1020
$420
341
0.0
0.2
42
3.25
0.2
64
256
Figure 16: LP vs SLR-AVD, across various ImageNet variations. k= k-1,2,44.,166,332.66
How sparse is SLR-AVD The average number of non-zero entries for each class is 447, 182, 177, 173, and 135 for t 1,2,4,8, 16, 32. The numbers rounded to the nearest. integers.
26

---

Published in Transactions on Machine Learning Research (05/2024)
3.0
1.0
18ammi X
2.5
0.8
10
HZ,
10Mcx
2.0
0.6
101 30
3 1.5
Iifee:
IH Y!
0.4
1.0
MR.
0.3
0.2
0.0
25
50
75
100
125
150
175
200
0.0
25
50
75
100
125
150
175
200
0
50
100
150
200
250
300
Epoch
Epoch
Epoch
Figure 17: The MI estimations at interest. The estimator variational and we include the whole optimization trajectory. Each experiment is run three times and two standard deviations are filled. Left: the MI between different set of features and the labels. Middle: the MI between different set of features and the domain indices. Right: the MI between different of features and the input images.
Experiments aggregation with k = 4 We include the results with A: 4 for method and several baselines in table 15. The MLP model has layers 512, 4500, 1000. This amounts to total of 512 4511 + 4500 * 1000 = 6804000 parameters, which equals to the total number of parameters in our SLR model (it has 6804 1000 parameters we also count the entries).
Table 15: An aggregation of several methods.
IN
IN-V2
IN-R
IN-A
IN-Sketch
ObjectNet
ZS
68.78
62.23
77.72
50.64
48.38
54.31
ZS-VD
65.89
59.19
72.75
46.11
44.84
49.60
ZS-AVD
69.52
62.97
77.85
50.87
48.91
54.58
MLP weight decay 0.01
52.55
49.23
29.08
44.39
27.15
29.71
MLP weight decay 0.1
52.62
49.33
29.07
44.39
27.2
29.79
LP
54.66
45.92
54.5
32.34
28.77
32.44
WISE-FT+LP
70.71
63.66
77.89
50.69
48.89
54.39
SLR-AVD
63.19
55.02
67.64
42.09
37.43
40.39
WISE-FT4SLR-AVD
71.18
64.2
77.98
51.07
49.37
54.59
FT
70.42
63.50
77.87
52.08
48.47
55.56
WISE-FT+FT
70.48
63.5
78.32
52.11
48.99
55.77
SLR-FT-AVD
71.70
64.79
77.95
52.61
48.94
55.10
WISE-FT+SLR-TTAVD
72.09
65.3
78.59
52.64
49.92
55.41
CoOp
69.36
62.77
76.54
50.43
47.96
53.68
WISE-FTTCoop
70.14
63.48
78.1
51.48
49.16
54.79
27


Reviews:
[{"review_id": "fjt9MErzFB", "reviewer": "Reviewer_QRYi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* Using the mutual information to explain why text descriptive features are better than other representations is interesting.\n\nWeakness\n1. **Not many new findings.** This is my main concern of this paper. Almost all the findings in this paper were presented in previous work. The zero-shot results have been shown in Pratt et al. (2022) and Menon & Vondrick (2022). The few-shot results have been demonstrated in Yang et al. (2023). The proposed method is almost the same as these works, and this paper just formalizes the ideas of previous work and replicates the results. The main difference between the proposed method and existing ones is the sparse logistic regression and some fine-tuning, but the author did not explain it well and lacked convincing ablations to verify its effectiveness (see weakness 3). The mutual information analysis is interesting, but the results have some issues (see weakness 4). The robustness claims have severe problems (see weakness 2). Overall, from my point of view, this paper does not provide additional important insights to the community.\n\n2. **Evidence for the robustness claim.** Usually, to prove the robustness of a model, the experiments should train/validate and test in different domains. Both in-domain (ID) and out-of-domain (OOD) performance need to be reported, and a robust model should have good in-domain, not drop too much on OOD (smaller gap), and have a better average domain performance than baselines.\n\n    The few-shot setting is not enough to justify the robustness claim. Robustness means the model is less likely to learn the spurious cues/biases of the in-domain data. However, in few-shot settings, the model does not learn much from the in-domain data; it still mainly leverages the priors (from LLM and CLIP) to make the prediction. The improvement over the linear probe is just because the text features have better priors, as shown in LaBo (Yang et al., 2023), not because the proposed method is more robust.\n\n    The proposed method is better than the linear probe in both in-domain and out-of-domain, but the domain gap doesn't get narrower, which means the proposed method may just be a stronger few-shot classifier, not a more robust classifier.\n\n    To support the robustness claim, the author must show the results using more in-domain data to train the model and needs to show the proposed model has a smaller in-domain, out-of-domain gap. In addition, using other more confounded datasets, such as Waterbirds (Sagawa et al., 2020) and [WILD](https://wilds.stanford.edu/datasets/) benchmark could be a better choice.\n\n3. **The evaluation and results are not solid.** This paper lacks direct comparison with existing work, like LaBo (Yang et al., 2023), in few-shot settings. Most improvements are marginal, especially compared with fine-tuning results (Table 4). Why in Table 4 only choose maximum $k=4$, not $32$, same as the other settings? I hypothesize that fine-tuning in small shots makes it hard for the model to update all parameters. The ablation of the sparse logistic regression is also missing. What if we learn a linear layer instead?\n\n4. **Inconsistent Mutual Information Results.** In the second paragraph of section 3.3, it says, \"a good feature representation $Z$ would have high mutual information with the label, $I(Z; Y)$, but low MI with the domain index, $I(Z;A)$.\" However, in paragraph 5, the results are \"$I(Z; A) > I(H_{avd}; A) > I(H_{cp}; A)$\" which means class prompts (cp) is more invariant to domain shift than augmented visual descriptors (avd). If this is true, why use avd as the feature?\n\n5. **The presentation of the paper can be improved.** Although the idea of the paper is simple, and I am very familiar with this field, I still find this paper hard to read. The method section introduces too many notions, and the method names in the results section are highly abbreviated, which makes it difficult to parse the results. I always need to go back and forth to understand the numbers. The figures in the paper can be improved. Figure 1 is below the standard of the TMLR publication. Figure 2 is not color-blind friendly, and I cannot map the lines to each label. Figures 3 and 4 are also hard to parse. In summary, this paper needs a lot of effort to improve its readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studied using text descriptions as feature exactors for few-shot image classification. The method is very simple. First, LLM (GPT-3) generates a list of textual descriptions for each class, along with the prompts containing the class names, as the information bottleneck. Given an image feature (encoded by CLIP), the dot product of the text features and image features is taken as the new feature representation of that image. Then, a logistic regression layer is trained on top of this feature as the classification head. The experiments show the proposed method has better few-shot ability than the selected baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "* Change the experimental setup and get new results to support the claim of robustness. (see weakness 2 and 3).\n* Explain the mutual information results (see weakness 4).\n* Improve the presentation of the paper (see weakness 5).", "broader_impact_concerns": "Discuss the potential bias inherited from GPT-3 and CLIP.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AZbwM4unIp", "reviewer": "Reviewer_3Luv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. Information-theoretic principles: The paper provides an analysis based on information-theoretic principles, such as compression and invariance, which provide a solid theoretical foundation for the proposed method.\n2. Experimental evaluations: The paper provides a comprehensive analysis of the proposed method through experiments on several image datasets, demonstrating its effectiveness compared to baselines.\n3. Improved generalization: The proposed method is shown to be robust to domain shifts, as the automatically generated visual features are less informative about the domain, leading to better generalization in out-of-distribution tasks.\n4. Interpretability: The suggested approach allows for the identification of the most informative features and their direct textual counterparts, as displayed in Table 6.️\n\n# Weaknesses\n1. Lack of ablation studies: A more detailed analysis of the contribution of each component of the proposed method (e.g., language model, ℓ 1 -regularized logistic regression) could help better understand its effectiveness and identify potential areas for improvement.\n2. Dependence on high-quality language models: The effectiveness of the proposed method relies on the quality of the language model used to generate the visual descriptors. Any errors or biases in the generated descriptions could impact the overall performance of SLR-AVD. An evaluation with openly-available models would also improve the reproducibility of the paper.\n3. Missing comparisons: Although the authors provide a comparison with CoOp in Section 4.4, the analysis would benefit from comparing also with more recent methods, such as CoCoop [1].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach to few-shot image classification using text descriptions of images. The proposed method, SLR-AVD, leverages multiple automatically generated visual descriptors for each class to improve classifier performance beyond zero-shot. The main insight is that these features are more compressive and invariant to domain shifts than traditional image embeddings, leading to better generalization in out-of-distribution tasks. The paper provides a thorough analysis of the proposed method and demonstrates its effectiveness through experiments on several image datasets, mainly ImageNet variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Robustness to LLM choice: Relying on closed models limits the reproducibility of the work. How much would the performance change if the descriptors are generated by openly-available models, e.g. Lllama2 or Mistral? Showing robustness to the LLM choice would strengthen the work.\n2. Comparison with recent prompt-based techniques: Although it is not critical to evaluate the proposed method, the analysis would greatly benefit from a more thorough comparison with prompt-learning techniques like CoCoop [1].\n\n[1] Zhou, Kaiyang, et al. \"Conditional prompt learning for vision-language models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "broader_impact_concerns": "A possible concern is about using LLMs to generate descriptors. Such a system would inherit any bias present in the original training data/model, this could potentially lead to biased/unfair classification.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1IY2zwS78L", "reviewer": "Reviewer_om1n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper investigates text descriptive features from an information-theoretic perspective, and uses selected descriptive features for a classification task in VLM, which is a new approach.\n- Theoretical understanding of text descriptive features is important for reliability. \n- Extensive studies on a few-shot setting strengthen the proposed method.\n\n**Weaknesses**\n- Ablation study on W^hat_avd : The proposed SLR-AVD method finetunes the entries in W^hat_avd. What are the reasons behind finetuning them? I thought this wouldn't be needed if the features were properly selected by sparse logistic regression. In that sense, 1) what is the performance when not W^hat_avd but only the image encoder is finetuned, and 2) when W_avd is used (and finetuned) instead of W^hat_avd?\n- Lack of information-theoretic understanding of sparsity on W : In Section 3.3, the authors explained the benefits of text descriptive features in the information-theoretic perspective. While the paper's proposed method, SLR-AVD, uses W^hat_avd (i.e. sparsity on W), it has not been theoretically compared to W. \n- How does the plot for I(H_vd;Y), I(H_vd; A), and I(H_vd; X) look like in Figure 2?\n- I suggest to standardize the term when describing the author's main method. For instance, 1) in Figure 3, I assume \"augmented visual descriptors\" might indicate SLR-AVD, and 2) in Table 1, AVD might indicate SLR-AVD, and 3) In Table 5, AVD should be WISE-SLR as described in page 7 (please correct me if I'm wrong). Also, method definitions in Table 2 should be explained in Section 4.\n- In Table 1, to highlight the method of using sparsity, AVD could be compared to SLR-AVD.\n- What is the contribution in a zero-shot setting? I assume SLR-AVD can only be applied in a few-shot setting. In that sense, the zero-shot setting only applies to VD and AVD. Since VD is the same as [1] and AVD is simply adding a standard prompt with VD, I wonder what the contribution is in the zero-shot setting. If there is a limited contribution in this setting, the experimental comparison in Section 4.1 could be removed or shortened.\n- Inconsistency of the zero-shot results in Table 1 and references [1,2] : In [1] (correspondingly VD in Table 1) and [2], the results are in par with ZS, while in Table 1 their results are lower much lower than ZS. \n- What are the plots for WISE-FT + AVD in Figure 4? Here AVD indicates finetuning both image encoder and W (not W^hat_avd).\n\n[1] S. Menon and C. Vondrick, Visual classification via description from large language models.\n\n[2] K. Roth et al., Waffling around for performance: Visual classification with random words and broad concepts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SLR-AVD, a few-shot method in a VLM classification task. The proposed method first generates multiple descriptions for each class using LLM, and selects important descriptions using sparse logistic regression optimization on descriptive features. Using sparse features is more robust to domain shifts than traditional image embeddings. The authors also show the robustness from an information-theoretic perspective. The authors compare the proposed method in the zero-shot and few-shot settings to demonstrate the effectiveness of their method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The proposals are mainly described in the Weaknesses. I request the authors to change (or add) the explanations/experiments/plots mentioned above.\n\n**minor**\n- Missing definition of notation blkdiag in Section 3.1.\n- Missing definitions of notation M, P, and d in Section 2.1.\n- In the second paragraph in Section 4.2, it might be Figure 3, not Table 1?\n- I think the last paragraph in Section 4.2 is more related to Section 4.3?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the core components of the article, such as the use of information-theoretic principles to build the approach (QRYi, om1n, 3Luv) and the extensive experimental validation (om1n, 3Luv).

At the same time, the reviewers raised several concerns, mostly regarding the technical contribution (i.e. QRYi) and the experimental validation itself. Specifically, the latter was considered lacking important analyses, such as in- vs out-of-domain gaps (QRYi, om1n), changes w.r.t. the number of shots (QRYi), missing comparisons with more recent baselines (QRYi, om1n, 3Luv), lack of extensive ablation studies (3Luv, om1n), and inconsistencies on the mutual information results (QRYi, om1n).

The authors' reply and the updated version of the manuscript addressed most of the concerns, especially those related to the experimental comparisons, ablations, and the number of shots. The only remaining concern is the discrepancy between the empirical results (Figure 2) and some inequalities of Section 3.3: $I(H_{avd}, Y)\geq I(\hat{H}_{avd}, Y)$,  

$I(H_{avd}, A)\geq I(\hat{H}_{avd}, A)$.



Nevertheless, all main concerns have been addressed and the previously mentioned misalignment is mild (i.e. leaning toward equality of the two mutual information). Thus, the AE recommends the acceptance of the work. As a suggested minor revision, the last part of Section 3.3 can be expanded to clarify the reasons behind the misalignment and/or why the differences are negligible.
Decision: Accept (poster)