Learning to Decompose Visual Features with Latent Textual Prompts
Authors: Feng Wang
Manling Li
Xudong Lin
Hairong Lv
Alex Schwing
Heng Ji
Venue: ICLR 2023 poster (2023)
Keywords: CLIP
vision-language learning
visual prompt

Abstract: Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations. Nonetheless, for downstream inference, CLIP-like models suffer from either 1) degraded accuracy and robustness in the case of inaccurate text descriptions during retrieval-based inference (the challenge for zero-shot protocol); or 2) breaking the well-established vision-language alignment (the challenge for linear probing). To address them, we propose Decomposed Feature Prompting (DeFo). DeFo leverages a flexible number of learnable embeddings as textual input while maintaining the vision-language dual-model architecture, which enables the model to learn decomposed visual features with the help of feature-level textual prompts. We further use an additional linear layer to perform classification, allowing a scalable size of language inputs. Our empirical study shows DeFo's significance in improving the vision-language models. For example, DeFo obtains 73.2% test accuracy on ImageNet with a ResNet-50 backbone without tuning any pretrained weights of both the vision and language encoder, outperforming zero-shot CLIP by a large margin of 15.0%, and outperforming state-of-the-art vision-language prompt tuning method by 7.6%.

Published as a conference paper at ICLR 2023
LEARNING TO DECOMPOSE VISUAL FEATURES WITH LATENT TEXTUAL PROMPTS
Feng Wang', Manling Li², Xudong Lin Hairong Lv Alexander G. Schwing" & Heng Ji
Tsinghua University University of Illinois at Urhana-Champaign SColumbia University
ABSTRACT
Recent advances in pre-training vision- -aaiuug models like CLIP (Radford et al., 2021) have shown great potential in learning transferable visual representations. Nonetheless, for downstream inference, CLIP--ike models suffer from either 1) degraded accuracy and robustness when inferring by retrieving textual class names (the zero-shot protocol); or 2) breaking the well-established vision-language alignment (linear probing). To combine the best of both worlds, we propose Decomposed Feature Prompting (DeFo). DeFo maintains the dual-model archi- tecture yet leverages learnable embeddings as textual input and performs classi- fication with an additional linear layer. As result, we find DeFo be able extract decomposed visual features with the help of textual prompts and to allow a scalable size of language inputs. Our empirical study shows DeFo's significance in improving the vision- laiion-a models. For example. DeFo obtains 73.2% test accuracy on ImageNet with a ResNet-50 backbone without tuning any pretrained weights of both the vision and language encoder, outperforming zero-shot CLIP by a large margin of 15.0%, and outperforming state-of-the-art vision-language prompt tuning by 7.6%.
INTRODUCTION
Language-guine visual pretraining has gained a lot of attention and shows great promise in learning transferable image representations. By establishing a connection between images and natural lan- guage, recent vision-language models are able to turn visual inference over restricted number of classes into zero-shot open-vocabulary inference (Radford al., 2021; Jia et al., 2021; Pham al., 2021).
One of the recent successes for zero-shot inference is the contrastive language-iman -maaa pretraining (CLIP) model (Radford et al., 2021). It uses 400 million image-text pairs to learn an alignment between visual and textual representations obtained from a vision encoder and a language encoder respectively. In downstream applications, CLIP-like models (Radford et al., 2021; Jia et al., 2021; Pham et al., 2021) then perform zero-shot inference by hard-target retrieval, i.c., they directly compute the distance between vectorial image representation obtained from the vision encoder, and representations of text prompts (e.g., "a photo of an airplane" or "a photo of an automobile") obtained from the language encoder. The target class (e.g., "airplane" or "automobile") correspond- ing to the text prompt with the smallest distance to the vector representing the image constitutes the zero-shot inference result. When annotations are given, simple linear probing (i.e., removing the language encoder, fine-tuning of the vision encoder and training of classifier on top of the vision encoder) further improves the results (Radford al., 2021). Moreover, context optimization (CoOp) (Zhou et al., 2021) 1) replaces the hand-crafted prefix or suffix (e.g., "a photo of a"") of the text prompts by trainable embedding vectors.
However, the zero-shot CLIP and CoOp infer using hard textual targets, i.e., the class names, which results in two main challenges. First, class names in text prompts (e.g., "airplane" or "automo- bile"), as used in zero-shot CLIP and CoOp inference, do not permit to accurately summarize the semantic information of an image. Therefore, inference is very sensitive to the words chosen for class names. We refer to this challenge expressive sensitivity. Empirically, this challenge causes zero-shot CLIP and CoOp to struggle to achieve as competitive results as linear probing with the same image encoder when downstream training data is available (e.g., 58.2% accuracy vs. 72.3%

---

Published as a conference paper at ICLR 2023
on ImageNet (Deng et al., 2009)). Moreover, this sensitivity can be observed by modifying class names. Fore example, for zero-shot inference on CIFAR-10 (Krizhevsky et al., 2009), CLIP ob- tains an accuracy of 63.7% when the original class names used. Notably, simply replacing or extending the class names with suitable synonyms' (e.g.. "plane" and "car" rather than "airplane" and "automobile") can improve accuracy to 79.6%, which highlights the challenge of expressive sensitivity.
Second, despite the fact that hundreds of millions of pretraining samples cover a large number of concepts that can possibly appear in downstream datasets, zero-shot inference continues to struggle to recognize rare objects. We refer to this as the conceptual sensitivity. For example, zero-shot CLIP is only 38.5% accurate when classifying EuroSAT satellite images (Helber et al., 2019). which much lower than the result of a supervised ResNet-50 (He et al., 2016) encoder (93.4%). Also, zero-shot CLIP with ResNet-50 encoder achieves less than 90% accuracy on MNIST (LeCun, 1998), which can even be outperformed by a simple logistic regression model. While linear probing is straightforward way to improve results, removing of the language encoder breaks the vision- language alignment that is learned from the pretraining data, and therefore degrades few-shot and transfer learning performance.
In this paper, we propose Decomposed Feature Prompting (DeFo), which turns the hard-target- retrieval paradigm of CLIP and CoOp into dual-model feature prompting. Specifically, DeFo 1) provides to the language encoder a set of learnable embedding sequences which are independent of the hard semantic targets; and 2) performs classification by tuning an additional layer. As result, DeFo does not rely on the textual representations of class names being classification targets, which addresses the issues of expressive sensitivity and conceptual sensitivity. Meanwhile, DeFo main- tains the dual-model architecture, which enables the model to leverage the language information, so that few-shot and transfer learning performance can be boosted.
DeFo results show the significance of addressing the sensitivity challenges of CLIP-like models. For example, with a ResNet-50 backbone, DeFo achieves 73.2% test accuracy on ImageNet without modifying any pretrained weight of the image and text encoders, outperforming vanilla CLIP by a large margin of 15.0% and outperforming CoOp by 7.6%. In a variety of visual contexts, DeFo attains average accuracy of 79.9% over 11 image classification benchmarks, which is 21.0% higher than that of zero-shot CLIP and 6.2% higher than CoOp.
2 RELATED WORK
Pretraining-fnetuninng has long been a dominant paradigm of transfer learning in machine learning. computer vision, and natural language processing. Generally, pretraining vision encoder by gener- ative objectives (Bao et al., 2021; Hc et al., 2022) or discriminative objectives (He et al., 2020; Chen et al., 2020; Grill et al., 2020: Caron et al., 2021) at the scale of one to ten million images (Deng ct al., 2009) is sufficient to yield good visual representations and strong predictive performance in downstream visual tasks. However, without the supervision from other modalities, such pretrained models require task-specific finetuning (Bao et al., 2021; He et al., 2022; O Pinheiro et al., 2020; Wang et al., 2022a; Lin al., 2022a) or linear probing He et al. (2020); Chen et al. (2020) for reasonably domain-adapted predictions.
The contrastive language-image pretraining (CLIP) (Radford et al., 2021) method instead jointly pretrains a vision encoder and text encoder on 400 million curated image-text pairs, with a con- trastive objective (Gutmann & Hyvärinen, 2010) that matches the visual and textual representations. In downstream applications, CLIP achieves competitive results in various vision or vision-language tasks such as image classification (Zhou et al., 2021; Gao et al., 2021), dense prediction (Rao al., 2022), video-language tasks (Luo et al., 2021; Lin et al., 2022b; Wang et al., 2022b). image manip- ulation (Patashnik et al., 202 1), and multimedia event extraction (Li et al., 2022).
Following the success of CLIP, the ALIGN (Jia et al., 2021) model leverages noisy dataset of 1.8 billion image-text pairs to scale up vasion-language representation learning, and the BASIC (Pham et al., 2021) model further scales up this approach in terms of data and model size. Based on the success of CLiP-like vision-language pretraining, a series of follow-up inference approaches are proposed to improve classification results. For example, Zhou et al. (2021) propose CoOp to learn
We use WordNet (Fellhaum, 2010) to find synonyms.
2

---

Published as a conference paper at ICLR 2023
context information in downstream datasets, and Gao et al. (2021) propose CLiPeAdapter to learn Nomain-adaptatiinn for vision-language models. Further, following CoOp, Zhou et al. (2022) propose CoCoOp to enhance the performance in unseen classes; and similarly, following CLIP Cdipeed Zhang et al. (2021) propose Tip-Adapter to explore non-parametric adaptation layers. Despite the progress these methods (Zhou al., 2021; Gao et al., 2021; Zhou al., 2022) have achieved in downstream predictive performance, they do not change CLIP's s inference paradigm of retrieving class names, Hence, the challenges of expressive sensitivity and conceptual sensitivity remain.
METHODOLOGY
As shown Figure 1, our DeFo fol- lows the dual-model architecture of CLIP, i.c. we use a vision encoder and language encoder which map the visual inputs and textual inputs into the same latent space. However, in DeFo, the language encoder plays different role from that in the zero- shot CLIP. Specifically, CLIP directly constructs hard targets for classifica- tion by feeding the language encoder with & textual queries (e.g.. "a photo where
gradient
logits
t 12 k
logits
Linear
weights
R2
weights
t
kdd
Tr
nud
f
V
+wwshs3
12 nxm.dd
1 Wxbx3
"A photo of cat"
[v]] [vi] [vP]]
"A photo of dog" (k queries)
Image
Image
[v]] [v
[vP]
of cat", "a photo of dog"..... the number of classes and each query corresponds to specific one. As explained in Section I, this in- ference protocol leads to expressive sensitivity and conceptual sensitiv- ity challenges which incurs degrada- tion of accuracy and robustness.
(n queries)
Zero-Shot CLIP
DeFo (ours)
Figure An architectural comparison between our DeFo and CLIP. "V" and "I." denotes vision and language encoder respectively and their weights are fixed. DeFo leverages se- quences of trainable embedding vectors ([v,I) as textual in- put and maps decomposed visual features by linear layer.
In contrast, in DeFo, change the existing paradigm of hard-target retrieval while maintaining the vision- lasgua-l encoder architec- ture learn decomposed visual features. Specifically, DeFo aims to utilize the language encoder to construct a projection matrix that maps the visual features from the d-dimensional CLIP latent space to new n-dimensional feature space. To this end, we feed the language encoder with n trainable text queries and then perform classification by an additional linear layer. By jointly tuning both the text queries and the classification layer, DeFo is able to learn textual prompts of detailed visual features and robust feature mapping for classification.
Overall, DeFo has two main benefits compared to CLIP-like models. First, compared with hard- target-based inference protocols such the zero-shot CLIP and CoOp (Zhou et al., 2021), DeFo removes the expressive and conceptual sensitivity challenges which significantly improves accuracy and robustness of downstream performance (see Table and 4). Next, compared with linear probing which discards textual information, the optimization of the projection matrix in DeFo is bounded by the text encoder which results in the need for much fewer training samples achieve good performance (see Table 2). Moreover, also note that in DeFo the number textual queries n is independent of the number of classes k; the query size is scalable fit specific downstream tasks. Next, we detail the DeFo and compare it to existing methods.
3.1 MODEL INFERENCE
As shown in Figure/ 1, DeFo uses a visual encoder gV -xxxxx3 + Rd and a language encoder gr. Rmx Rdd to extract image and text representations, respectively. For this, the visual inputs are 3-channel images of shape w x h, and the language inputs are sentences with m words where each word is embedded into d, dd-dimensional vector. Both the visual and textual features are then mapped into a d-dimensional latent space, i.e., we get an image representation vector fr E IRd and text representation vectors ft. f? f?...... ...... ... fr E Rd, where n denotes the number of query sentences used for the encoder gr. By applying the dot product between fa and each of the f` (note that both
3

---

Published as a conference paper at ICLR 2023
fi and f are & normalized vectors, i.e., Il l2 =1ff+ll 1), get n-dimensional vector, where the i-th element measures the similarity between the image and the i-th text query.
CLIP and CoOp directly use this vector to predict the label of the image, because each text query in their settings corresponds to a specific class. Formally, CLIP and CoOp have n = k, where is the number of classes to be inferred, and the probability of the image belonging the i-th class is computed by
(xp((T Pi Σj=1 exp(
where ( denotes the dot product and T is a temperature coefficient.
Instead, DeFo decouples the text queries from specific classes. Specifically, use scalable num- ber of queries, ie., the number n is not limited to be equal to k, and perform classification by an additional linear layer that maps the n-dimensional feature vectors to k-dimensional vectors. The probabilities are then computed by the softmax of the k-dimensional vector. Note that only this linear classification layer and the textual queries are trainable in DeFo. We fix the weights of both the text encoder and the image encoder to maintain the vision- lasoualanggugee alignment.
3.2 TRAINABLE TEXT EMBEDDINGS
The language encoder gr, receives sequences of de-dimeesional embedding vectors as input. When natural language is used, each word in the vocabulary first needs to be encoded into de dddimenniional embedding. In DeFo, we skip the process of designing hand-crafted prompts with natural language. Instead, we directly optimize the word embeddings via back-propagation. Specifically, n independent sequences of text embeddings where each sequence consists of m de-dimensional vectors in the form of "[y'] [v²] ... [v'']". The total "textual" input of DeFo can be written as a tensor X,. E Ruxmxd. Note that here we assign the same length m to each query for easy comprehension and implementation. In practice, the design of DeFo's input more flexible and the length of each query is not required to be identical.
initialize
By optimizing XL. DeFo makes CLIP-like vision-language models free from both hand-crafted prompts and annotations such as class names. In this way we address the issues of expressive and conceptual sensitivity caused by using class names as hard targets.
3.3 COMPARISON TO EXISTING METHODS
As illustrated in Figure 1, zero-shot CLIP has no trainable parameters. The textual queries are com- posed by hand-crafted prompt and class names that describe the semantic targets of the categories. The linear- linear-probing probing CLIP uses only the vision encoder for classification. Without the assistance of textual representations, this method has to utilize an additional linear layer to map the visual fea- tures from the latent space (d-dimensional) to the output space (k-dimensional), which introduces N = d x & additionally trainable parameters. CoOp (Zhou et al., 2021) mostly follows the archi- tecture zero-shot CLIP, yet replaces CLIP's hand-crafted prompt by sequence of trainable text embeddings, with N = k x m X de learnable parameters for class-seecific prompts.
Intuitively, both CoOp and our DeFo use trainable text embeddings as inputs of the language encoder. Both methods differ in that the number of textual queries n is independent from the number of classes i for DeFo, and the queries are not composed using class names. Therefore, DeFo has a scalable size of additionally learnable parameters. Specifically, it introduces in total N = n x (m de + k)) trainable parameters, which scales linearly with the number of queries n. For example, with able parameters while attaining 72.3% accuracy, which outperforms CoOp (65.6%) who has 8.2M learable parameters and CLIP- CLIP-Adapter (63.6%) who has 1M learnable parameters.
256 and m 16 in ImageNet (k 1000), DeFo introduces 2.4M learn-
In addition, compared to linear probing which directly maps the d-dimensional latent features to out- put logits, DeFo also uses linear layer but maps n-dimensional features. In this way. DeFo is able to first project visual features with the assistance of n textual representation vectors, which provides DeFo with significantly better few-shot performance and interpretability than linear probing.
4

---

Published as a conference paper at ICLR 2023
Table I: Test accuracy ImageNet (%). Results with i taken from Zhou et al. (2021), and those with & are taken from Zhang et al. (2021). Our results are marked in gray The best results are
bolded. The results without using text encoder are de-emphasized.
Method
RN-50
RN-101
ViT-B/32
VIT-B/16
Zero-Shot CLIP (Radford ct al., 2021)
58.2
61.5
62.0
66.9
Linear-Probing CLIP
72.8
75.5
76.0
79.5
Prompt Ensembling
60.41
62.5T
63.7
68.71
CoOp (Zhou et al., 2021)
65.6
67.8
68.0
72.4
CoCoOp (Zhou et al., 2022)
65.1
67.1
Target Optimization (our ablation)
71.4
73.2
74.0
78.1
CLIP-Adaptes (Gao al., 2021)
63.6
65.4
66.2
71.1
Tip- Aip-Aeppter (Zhang et al., 2021)
62.0
64.8
65.6
70.8
DeFo (ours)
73.2
75.5
76.2
80.2
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
4.1.1 BASELINE MODELS
DeFo based on CLIP (Radford et al., 2021) for an easy comparison to the other baselines (Zhou et al., 2021; 2022; Gao et al., 2021). For CLIP, we mainly explore two inference protocols, zero-shot and linear probing. Zero-shot CLIP requires no extra training data and it infers by directly matching image representation to the text representation of class names with hand-crafted prompts. Linear- probing CLIP drops the text encoder and instead attaches a randomly initialized linear layer to the image encoder, and then tunes only this linear layer with downstream training data for domain- adapted classification.
CoOp (Zhou al., 2021) and CLIP-Adapter (Gao et al., 2021) succeed in improving CLIP inference performance so they serve as the primary baselines to our DeFo. To give more comprehensive
results, also compare DeFo ImageNet the recent baselines CoCoOp (Zhou et al., 2022) and Tip-Adapter (Zhang et al., 2021), which are direct extensions for CoOp (Zhou et al., 2021) and CLIP-Adapter (Gao al., 2021). Note that we do not expect CoCoOp and Tip-Adapter to yield better results than their base models CoOp and CLIP-Adapter because they are proposed to address different problem (discussed in Section 2). We report the results of Tip-Adapter without its further fine-tuning (Zhang et al., 2021) and all the baselines follow the pre-proee pro-processing of CoOp for fair comparison. Further, in this paper, we develop another bascline called "Target Optimization", which uses learnable embedding vectors class names combined with hand-crafted prompt prefix. Target Optimization can be regarded as an ablated version of DeFo, which helps to understand the importance of the learnable embeddings.
4.1.2 DATASETS
We follow prior methods to select 11 publicly available datasets, i.e., ImageNet (Deng et al., 2009), Food101 (Bossard al., 2014), OxfordPets (Parkhi al., 2012), Caltech101 (Fei-Fei al., 2004), SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012), StanfordCars (Krause et al., 2013), FGVCAireraft (Maji al., 2013), DTD (Cimpoi al., 2014), Flowers102 (Nilsback & Zisserman, 2008), and EuroSAT (Helber et al., 2019). The categories in these 11 datasets include natural objects, scenes, human actions and fine-grained features such as textures and satellite imagery, which could cover general semantic targets of visual understanding tasks.
For the domain--eneralizatiin study, we also evaluate the models on four ImageNet-varian datasets, namely, ImageNet-v2 (Recht et al., 2019), ImageNet-Advvesarr (Hendrycks et al., 2021b), ImageNet-RRtention Retention (Hendrycks et al., 2021a), and ImageNet-Sketee Sketch (Wang et al., 2019). These four datasets do not have training images and their categories correspond to ImageNet (Deng al., 2009). We train on ImageNet and test on these variant datasets to evaluate camainngeneralizattinn performance.
5

---

Published as a conference paper at ICLR 2023
Table Few-shot accuracy on ImageNet (%). n-shot denotes training with n samples per class. t: Note that the Zero-Shot CLIP uses no training data of ImageNet. We put this result to the column "Full" for easy comparison. Our results are marked in gray The best results are bolded.
Method
L Encoder
Full
I-shot
2-shot
4-shot
8-shot
16-shot
Zero-Shot CLIP
V
58.2
Linear Prob. CLIP
X
72.8
23.6
32.2
40.8
48.9
54.3
CoOp
/
65.6
59.2
59.4
59.7
61.0
63.3
CoCoOp
65.1
57.4
57.8
58.2
58.5
59.0
ELIP-Adapter
58.2
58.6
59.4
60.4
61.3
Tip-Adapter
V
57.1
57.8
58.6
59.9
61.0
DeFo (ours)
73.2
59.4
59.7
60.3
61.7
64.0
4.1.3 TECHNICAL DETAILS
The experiments are built upon CLIP pretrained models. During training, the weights of both image and text encoders are frozen. In this paper, we explore both few-shot and full-dataset training. The few-shot setting follows CLIP (Radford et al., 2021) and CoOp (Zhou et al., 2021). ), i.e., training with 1, 2, 4, 8, and 16 samples per class that are randomly selected from the training set. By default, we use simple data augmentation of random crop and flip, and train with SGD optimizer with a mini- batch size of 32, 2e-3 learning rate, 0.9 momentum, and 0.01 weight decay (following CoOp (Zhou al., 2021)) for 50 epochs. For full-dataset training on ImageNet, we use a batch size of 256 and a learning rate of 0.01, which yields similar accuracy to the default setting but significantly reduces training time.
The number of text queries (n) is naturally fixed to the number of classes (k) for zero-shot CLIP, CoOp and Target Optimization. We set the length of learnable prompt to 16 words for CoOp. and set the length of learnable class name to two words for Target Optimization. The query size of DeFo scalable in terms of both the length and quantity of text, so we have flexible choices. We empirically find that a larger query size (the number of text queries n)) generally yields better predictive performance, in particular for large- For example, with similar number of text queries, i.e., 1000 for CLIP and 1024 for DeFo, DeFo outperforms the zero-shot CLIP by 14.1 1% (top-1 acc.) on ImageNet, while this improvement can be further boosted to 15.0% by using 2048 queries in DeFo.
datasets such as ImageNet (Deng et al., 2009).
When training on full ImageNet, we use n = 2048 text queries and m = 16 words (following CoOp (Zhou et al., 2021)) fully exploit its learning capacity. For few-shot training on ImageNet, we use a smaller query size, i.e., datasets, the text length set to 16, and we find that a smaller number of queries could be sufficient to yield good performance. Specifically, considering the scale of each dataset, we set n = 1024 for SUN397 (Xiao al., 2010), 512 for StanfordCars (Krause et al., 2013), Food101 (Bossard et al., 2014), and UCF101 (Soomro et al., 2012), = 256 for Caltech101 (Fei-Fei et al., 2004), Flowers 102 (Nilsback & Zisserman, 2008), and FGVCAircraft (Maji al., 2013), and n 128 for OxfordPets (Parkhi et al., 2012), DTD (Cimpoi et al., 2014), and EuroSAT (Helber et al., 2019).
1024 and m 4, to prevent over-fitting. For the other 10
For CoOp, we follow default setup to initialize the trainable text embeddings from randomness, as we find that the random initialization and manual initialization (e.g., initialize from "a photo of a²) yield almost the same performance. When training on full datasets, this phenomenon also works for DeFo and Target Optimization, so randomly initialize the parameters well. For few-shot training of DeFo, we initialize the first & text queries by the class names with random prefix, and fix the corresponding weights (W E & of the classification layer to identity matrix. In this way we further reduce the number of trainable parameters and make use language supervision via the text encoder, which consequently yields robust performance when training data is limited.
4.2
MAIN RESULTS
4.2.1 COMPARISON ON IMAGENET
We first compare our DeFo with the baselines on ImageNet under both full-dataset training and few-shot settings. As shown in Table 1, by training on the entire ImageNet data, method ob-

---

Published as a conference paper at ICLR 2023
Table Domain transfer accuracy on ImageNet variants (%). Our results are marked in gray
Method
Encoder
ImageNet-v2
ImageNet-A
ImageNet-R
ImageNet-S
Zero-Shot CLIP
51.5
21.7
56.0
32.9
Lin-Prohe CLIP
X 52.1 (+0.6)
13.6 (-8.1)
35.5 (-20.5)
21.8 (-11.1)
CoOp
/
53.3 (+3.8)
22.4 (+0.7)
55.9 (-0.1)
33.5 (+0.6)
DeFo (ours)
/
58.4 (+6.9)
21.7 (+0.0)
55.8 (-0.2)
33.2 (+0.3)
Table Average test accuracy (%) on 11 datasets. Results with f are taken from (Gao al., 2021).
Our results are marked in gray The best results are bolded. The results without using text encoder de-emphasized.
Method
RN-50
RN-101
ViT-B/32
ViT-B/16
Zero-Shot CLIP (Radford et al., 2021)
58.9
59.9
61.6
65.2
Linear-Probing CLIP
79.2
81.9
74.7
80.0
CoOp (Zhou et al., 2021)
73.7
76.2
75.5
79.7
Target Optimization
76.1
78.2
76.3
80.8
CLIP-Adapter (Gao al., 2021)
74.6
DeFo (ours)
79.9
82.5
80.8
82.8
tains the highest test accuracy with both ResNet and Vision Transformer backbones. Notably, with ResNet-50 image encoder, DeFo outperforms the zero-shot CLIP by 15.0%. It is also ob- served that by using better prompts (i.e., Prompt Ensembling and CoOp), the accuracy is improved by relatively small margin. This result demonstrates the issues of expressive sensitivity, i.e., the annant-nnotated class names cannot define or well describe the semantic information the images in each category, even if the prompt has been optimized. Notably, using a simple prompt but opti- mizing the class names (Target Optimization) yields more competitive performance (e.g.. 71.4% vs. 65.6%)
Overall, our DeFo continues to yield superior performance than the baselines for both full-dataset and few-shot training shown in Table The linear-probing protocol achieves close accuracy to DeFo with sufficient training samples (e.g., 72.8% vs. 73.2%). However, its drawback is obvious when training data is limited. Typically, as reported in Table 2, the linear probing protocol with one sample per class yields only 23.6% accuracy, which is much lower than that of zero-shot CLIP (58.2%), CoOp (59.2%), and our DeFo (59.4%).
4.2.2 GENERALIZED PERFORMANCE
evaluate the domain-transfer performance by 16-shot training on ImageNet and testing on ImageNet-v2 (Recht et al., 2019), ImageNet-Adversart (Hendrycks al., 2021b), ImageNet- Retention (Hendrycks et al., 2021a), and ImageNet-Sketch (Wang et al., 2019). As shown in Table 3, compared with the baseline of zero-shot CLIP, DeFo attains 6.9% higher accuracy ImageNet-v2. Also, DeFo yields a similar level of transfer performance as zero-shot CLIP and CoOp did on the other three datasets. In contrast, the linear probing protocol incurs significantly degraded perfor- mance on ImageNet-A, -R, and -S, as it forgoes assistance of language information.
For wider range of classification tasks, further evaluate DeFo shown in Table 4, our DeFo achieves the highest average test accuracy over the 11 benchmarks with different image encoders. A specific comparison CLIP and CoOp on each of the datasets is also provided in Figure 2. We note that CLIP favors the common and generic objects such as the images in Food101, OxfordPets, and 1o1tech for which our DeFo outperforms CLIP by < 10% accuracy and CoOp even fails to improve upon CLIP on Food101. However, when comes to fine-grained feature recognition tasks such as classifying the type of aircraft (Maji et al., 2013), CLIP and CoOp are shown to be very sensitive to the objects. Consequently, DeFo outperforms CLIP by 25.4% accuracy and outperforms CoOp by 11.2% on this dataset. The different robustness between CLIP and DeFo the 11 datasets indicates the issue of sensitivity challenge for CLIP, and indicates that DeFo successfully addresses this issue by decomposing and then combining the visual features.
total of 11 datasets. As

---

Published as a conference paper at ICLR 2023
456.2
CoOp
Accuracy over CLIP
DeFO (ours)
40-
+28.7
+31.1
124.3
+25.4
+19.4
20
+15.0
+15.0
17.5
+5.3
+4.9
Food101
OxfordPets
Calroch 101
ImageNet
SUN397
UCF101
StddCars
FGVCA
DTD
Flowers 102
EAro
Figure 2: Accuracy improvements over zero-shot CLIP. On all the 11 classification benchmarks, our method outperforms the CLIP and CoOp baselines by non-trivial margins.
Learned queries:
Well-matched images:
Leamned queries:
rell-matched images:
" "toward red [s] love"
"snow - [s] [s]" (Oxford Pets)
(a)
(Caltech-101)
(d)
(b) "ring- [s] aboard [s]" (Caltech-101)
(e) "dogs- [s] ends [s]" (Oxford Pets)
"[s] sunflowers with [s]"
"an [s] stripe- [s]"
(c)
(f)
(Caltech-101)
(Caltech-101)
Figure 3: Interpretation (nearest words) of the learned text embeddings of DeFo. We highlight the key words and replace the symbols and meaningless words by "[s]". We surprisingly find that our DeFo is able to learn detailed visual features such as color (a), shape (b), texture (c), and context (d). Also, DeFo is able to directly learn precise semantic target (f, sunflower is category of Caltech- 101) generalized semantic target (c).
4.3 INTERPRETATION OF TEXT QUERIES
One benefit of CLIP-like models is that they are able to provide interpretable visual predictions, as the visual features highly aligned with the representations of natural language. A simple way to interpret the learned word embeddings, i.c., the n sequences of m embedding vectors in Xx, is searching the nearest natural words within the vocabulary by measuring their Euclidean distance. However, as this approach directly maps the continuous embedding vectors into discrete codes of words, the interpreted sentences do not necessarily "make sense" and may contain meaningless words or symbols, which is also observed in prior work (Zhou et al., 2021).
Nonetheless, we still find very interesting evidence from the interpretation of DeFo. We observe that some of the interpreted query sentences include meaningful key words that describe specific visual features such color, shape, and texture. As illustrated in Figure (a)-(c), in Caltech-101 dataset (Fei-Fei et al., 2004), DeFo learns the words "red", "ring". and "stripe", while the well- matched (based on the consine similarity in the latent space) images in the dataset look consistent with human understanding of these features. For example, DeFo matches the word "red" with the objects such as a lotus flower and a bird in this color. For the word "ring", we can find the ring or circle shapes in the corresponded images. Also, DeFo is able extract background information such as "snow" (see Figure/ (d)). And surprisingly, DeFo sometimes directly learns the semantic targets that are closely related to the categories of the dataset. For example, it learns the word "dogs" which is a parent category in OxfordPets (Parkhi et al., 2012), and the word "sunflowers" which is exact category in Caltech-101 (Fei-Fei et al., 2004).
8

---

Published as a conference paper at ICLR 2023
Despite the fact that this interpretation approach is not nggorous enough, because the text features learned by DeFo possibly exceed the existing vocabulary. still provides very strong evidence that DeFo features are meaningful. We hope this result will yield greater insights in a follow-up study on interpretable vision-language inference.
4.4 ABLATION STUDY
In this section, we present additional results ablate the gains of DeFo. First, the size of the textual input, including the number of queries n and the length yn of each query sentence, may affect the performance. We compare the accuracy of DeFo model with different n and m and summarize the results in Table 5a and 5b, where we use ResNet-50 image encoder and train our model on the entire ImageNet data. As reported in the two tables, smaller size of textual input slightly reduces the performance within an acceptable margin. For example, with n 256 queries it yields 72.5% accuracy on ImageNet, which is only 0.7% lower than that of the default setup of = 2048. Also, 16, and further
using 4 words per query yields only 0.5% lower accuracy than that of increasing m (e.g., m = 32) cannot obtain clear improvements. These results indicate that DeFo is robust to its syper-parameters, and DeFo improvements of accuracy do not rely large-scale trainable parameters.
There is another concern that DeFo uses additional classification layer which introduces more parameters, where n and & denote the number of queries and classes, respectively. To ablate the gain of the additional parameters, their performance with DeFo. Specifically, we attach a k x k-dimensional linear layer to the logits DeFo if we set
add the same layer on top of CLIP and CoOp, and compare
of CLIP and CoOp, their model size identical experiment on ImageNet as well, and the results are summarized in Table 5c, As is shown, with the help of the linear layer, the accuracy of CLIP improved 62.4%, which is still significantly lower than that of DeFo. Notably, the "CLIP linear" model is equivalent to our DeFo model with n k and fixing the textual inputs to the queries used in CLIP. This indicates that the classification layer yields a very limited improvement (< 4%), and the superior performance of DeFo mainly comes from learning decomposed features.
k. We conduct this
Table 5: Ablation studies of DeFo. Our results with default setup are marked in gray
(a) Accuracy with m=16.
(b) Accuracy with n=2048.
(c) Gain linear layer.
Queries (n)
Acc.
Length (m)
Acc.
Model
Acc.
256
72.3
2
72.5
CLIP
58.2
512
72.5
72.7
CLIP + linear
62.0
1024
72.9
8
73.0
CoOp
65.6
2048
73.2
16
73.2
CoOp + linear
69.8
32
73.2
DeFo (n=1000)
72.9
CONCLUSION
In this paper, we identify two main issues of existing vision-language inference protocols, i.e., the expressive sensitivity and the conceptual sensitivity. To address them, we propose DeFo which main- tains the dual-mode architecture but infers by decomposed visual features. Specifically, it leverages trainable text prompts and decouples visual features from hard semantic targets. We demonstrate the significance of DeFo by showing its two benefits. First, DeFo gets rid of the textual descriptions of class names and instead infers via a linear classifier, which yields superior performance in the full- dataset scenarios compared with zero-shot CLIP and CoOp. Next, DeFo keeps the language encoder, which we find is able to bound the projection of visual features and therefore achieves competitive results in few-shot learning and domain transfer. Overall, DeFo provides new vision-language learning and inference paradigm, i.c., prompting the decomposed visual features, which we hope is of practical importance in fully exploiting the learning capacity of vision-language models.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. arXiv
preprint arv: 10:::2206.r0 08254, 2021.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, 2014. 5,6
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 2
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014. 6,
Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 2,
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, 2004. 5,
Christiane Fellbaum. Wordnet. In Theory and applications of ontology: computer applications. Springer, 2010.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 2021. 7,
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your latent new approach to self-supervised learning. In NeurlPS, 2020. 2
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. JMIR, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. 5.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a. 5,7
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b. 5,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and laniun-language representation learning with noisy text supervision. In ICMI. 2021. 1,2
10

---

Published as a conference paper at ICLR 2023
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for tne-grained categorization. In ICCV Workshop, 2013. 5, 6
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009. 2
Yann LeCun, The mnist database of handwritten digits. hup::/yann.. lecun. com/exdb/nmistl/ 1998. 2
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Fhih-Fu Chang. Clip-event: Connecting text and images with event structures. In CVPR, 2022. 2
Xudong Lin, Fabio Petroni, Gedas Bertasius, Mareus Rohrbach, Shih-Fu Chang, and Lorenzo Tor- resani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13853-13865 2022a.
Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, and Shih- Fu Chang. Towards fast adaptation of pretrained contrastive models for multi-channel video- language retrieval. arXiv preprint wrXiv:2206.020S8 2022b. 2
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860 2021. 2
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.331555, 2013. 5, 6,
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 5, 6
Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. NeurlPS, 2020. 2
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012. 5, 6,
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, 2021. 2
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv:2111.10050 2021. 1,2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021. 1, 5,
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guiss dense prediction with context-aware prompting. In CVPR, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019. 5,7
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2012. 5, 6
Feng Wang, Huiyu Wang, Chen Wei, Alan Yuille, and Wei Shen. Cp2: Copy-paste contrastive pretraining for semantic segmentation. ECCV, 2022a. 2
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurlPS, 2019. 5,

---

Published as a conference paper at ICLR 2023
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang. Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. arXiv preprint arXiv:2205.10747, 2022b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 5, 6
Renrui Zhang, Rongyao Fang. Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 2021. 3,5
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. arXiv preprint arXiv:2 2021. 3, 6, 8
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022. 3,5 5
12


Reviews:
[{"review_id": "jrTfputadJ", "reviewer": "Reviewer_MDxU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper point out two challenges in the downstream inference of pre-training vision-language models: expressive sensitivity and conceptual sensitivity.  To handle the problems, the paper proposes a new dual-model feature prompting methd, named as Decomposed Feature Prompting (DeFo). By providing an independent set of learnable embedding and tuning an additional layer for classification, authors claim that the model trained by DeFo significantly addresses the sensitivity challenges of CLIP-like models.\n", "summary_of_the_review": "The proposed approach is shown to be effective, but the lack of some experiments may lead to the limited contribution of the proposed method. If the author adds more meaningful experiments, I think it will make the paper more interesting, and I am very happy to revise my score.\n", "strength_and_weaknesses": "Strength：\n1. The two problems pointed out by the paper: do exist and deserve the attention of the community.\n2. The authors take time to implement and evaluate several prominent baselines. Experimental evaluation shows competitive performance.\n3. This paper is well written and easy to follow.\n\nWeakness:\n1. The paper does not discuss the computational complexity of the proposed methods. How to efficiently complete the fine-tuning of the pre-trained model is also a direction worthy of attention. I look forward to seeing the authors discuss a comprehensive comparison of DeFo's training time and other methods, such as CoOp and CLIP-adapter.\n2. Some important ablation study may be missing.  Since the authors point out that using class labels to generate text embeddings may bring challenges with expressive sensitivity. So a very straightforward idea is that we can directly set an independently learnable parameter as the prototype of each category to calculate the cosine similarity with image embeddings. or adopt the exponential-moving-average (EMA) manner [3]. These above-mentioned methods do not use text encode. Therefore, it is not necessary to carry out the forward of the text encoder every iteration during training. If these method can also achieve very good results, then I feel that the novelty and effectiveness of DeFo may be challenged. Therefore, I think it is very necessary to supplement this experiment. Look forward to the author discussing in following version.\n3. The comparison of some other important baseline is missing, such as Tip-adapter [1] and CoCoOp [2].\n4. I look forward to the author's discussion of the additional learnable parameters introduced in addition to CLIP's pre-trained model, and compare the number with other methods. Because if too many parameters are introduced, the performance improvement may come from overfitting of too many parameters. If the authors would like to compare the number of additional parameters of DeFo with CoOp and CLIP-adapter, I think it may be very helpful for us to comprehensively evaluate and compare these methods.\n5. We expect that the model can not only achieve good performance on a single dataset, but also have the potential to transfer beyond a single dataset. I suggest authors to add discussion about the perfomance of DeFo for domain generalization. Maybe the setting in Section 4.2 of [3] is a good formulation. This may strengthen the contribution of the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qqwij5l5D_", "reviewer": "Reviewer_xRfX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address two issues regarding the Clip-like model: 1) degraded accuracy and robustness when inferring by retrieving textual class names (the zero-shot protocol); 2) breaking the well-established vision-language alignment (linear probing). To combine the best of both worlds, this paper proposes Decomposed Feature Prompting (DeFo), which maintains the dual-model architecture yet leverages learnable embeddings as textual input and performs classification with an additional linear layer. The empirical study shows DeFo’s performance in improving the vision-language models.", "summary_of_the_review": "In general, this paper is well written. I suggest that this paper can be accepted after supplementing some contrastive results against SOTA method and more discussions in the experimental part.  \n\n", "strength_and_weaknesses": "Strengths:\n1）Prompts Learning is a meaningful direction, and this paper provides a valuable discussion of this direction to some extent.\n2) The authors propose a novel and efficient method to address two issues regarding the Clip-like model.\n3) The DeFo achieves 73.2% test accuracy on ImageNet with a ResNet-50 backbone without tuning any pretrained weights of both the vision and language encoder, outperforming zero-shot CLIP by a large margin of 15.0%, and outperforming state-of-the-art vision-language prompt tuning by 7.6%.\n4）This paper is easy to follow and the motivation is well explained.\n\nWeaknesses:\nThere are some issues that need to be improved: 1) The introduction and related work section lack a detailed discussion of the recent method CoCoOp. 2) A comparison with the SOTA method CoCoOp is lacking in the experimental section. For example, in Tab1, Tab2, Tab3, Tab4 and Fig2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "M0Dm6d93fW", "reviewer": "Reviewer_gg5B", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a simple method of prompt learning for Vision-Langauge model (in particular the CLIP). It is an extension based on existing work CoOp, where it leverages learnable embeddings as textual input and performs classification on this basis. Empirically, DoFo shows great improvement, where it reaches 73.2% on ImageNet 1K benchmark. In overall, it is a simple yet effective way for the VL prompt learning.", "summary_of_the_review": "In overall this is a good extension on the previous CoOp, where the authors aim to improve the prompt learning for a pre-trained VL model, in particular the CLIP. The proposed method is simple: by leveraging set of learnable parameters as textual embedding, it assist the classification task especially on ImageNet chanllenge. In essense, DeFo is largely based on the exisitng technique of CoOp, yet it's presented empirical results largely exceed the previous. My biggest concern arises from the aspect of technical contribution and the larger impact of CoOp alike work: \n\n1. The reviewer understands that simple technique is possibly to lead to bigger impact, so it is indeed not an issue for a intuitive&direct solution as a good submission. Yet my hesitation comes from the consideration that the presented insight of DeFo remains less significant: it seems that the biggest modification is making the learnable embedding as input for textual encoder yet it seems the important baseline ``Target Optimization'' also reaches a good results in such a similar schema. But again, I understand and also agree that the complexity of technique does not reflect contribution so I prone to suggest acceptance for the submission. \n\n2. Impact of CoOp alike work: another of my concern comes from the real world impact of prompt learning and its variations in recent years. Indeed, prompt learning, especially for LLMs show promissing applications for language comprehending tasks, yet this seems to be less important on the Vision side, which is mainly because that the pre-trained Vision model is that large enough that requires prompt learning. E.g., tiny model like MobileNet can even perform better than CLIP or other Vision tasks much more efficiently and better. A more desperate need of prompt engineering is for the Vision-Language Tasks that require more comprehending and high-level understanding, e.g., image/video question answering, captioning and cross-modal retrieval. In such context, prompt engineering for CLIP is rather limited and does not reflect much extension yet as current efforts seem more focusing on ImageNet challenge for simple classification task. But admittedly, the reviewer still agree that DeFo is in overall a good submission and has it's contribution for the community.", "strength_and_weaknesses": "Strength:\n1. Good and clear motivation, good paper writing with easy understanding scope&contribution.\n2. Great empirical results. \n\n\nWeaknesses:\n1. Limitation of technical novelty but it may not be a big concern from my perspective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster