XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding
Year: 2023

Abstract:
Recently, Contrastive Visual-Language Pretraining (CLIP) has demonstrated remarkable capability in various Visual Language Understanding (VLU) tasks. Yet, most CLIP-based methods require tasks-specific designs and sufficient training data. In this paper, we introduce a simple yet efficient paradigm for lowresource VLU named XtremeCLIP, which involves very few trainable parameters to improve the generalization ability of the trained models. In our XtremeCLIP framework, we reformulate a series of VLU tasks as a unified open-book affinity-matching problem. Furthermore, to handle the insufficient supervised signals in small datasets, we adopt contrastive learning to utilize the implicit sorting information of ground-truth labels to provide more supervised cues. Extensive experiments over multiple datasets on visual entailment, visual question answering, and image classification show that XtremeCLIP consistently outperforms existing baselines in low-resource settings. 1

heading: 1 Introduction
text: Pre-trained Visual-Language models such as XVLM (Zeng et al., 2021) and CLIP (Radford et al., 2021) have been proposed to unify visual and textual representations in the same embedding space and shown great potential for Visual Language Understanding (VLU). Conventional fine-tuning approaches (Clark et al., 2020; Lee et al., 2020; Wang et al., 2023) heavily depend on the time-consuming and labor-intensive process of data annotation, which are bothersome in low-resource scenarios. In the literature, Ben Zaken et al. (2022); Song et al. (2022) propose partial-parameter fine-tuning to preserve the pre-trained knowledge of these models. Yao et al. (2021); Song et al. (2022); Tsim-
∗Corresponding author 1The source code is publicly available in the EasyNLP
framework (Wang et al., 2022). URL: https://github. com/alibaba/EasyNLP.
poukelli et al. (2021) reformulate visual grounding and visual question answering as a “fill-inblank” problem by hand-crafted prompts. Gao et al. (2021); Zhang et al. (2022) utilize lightweight adapters (Houlsby et al., 2019) to retain the knowledge of CLIP. Besides, Zhou et al. (2022b,a); Zhu et al. (2022) address image classification tasks by utilizing textual representations describing image categories.
Despite the success, we suggest there are still some drawbacks in existing works. i) The discrete prompt paradigm requires labor-intensive promptengineering, while the soft template paradigm results in an unstable training process. ii) Adapters or partial-parameter fine-tuning methods may underperform due to their relatively large number of tunable parameters, requiring additional training data to achieve satisfactory results. iii) The aforementioned methods are task-specific in design, implying that their effectiveness may be derived from task-specific architectures. Hence, it is vital for us to design a more unified parameter-efficient tuning approach in order to solve various VLU tasks.
In this paper, we present XtremeCLIP, an extremely parameter-efficient tuning method for solving various VLU tasks based on CLIP (Radford et al., 2021). XtremeCLIP reformulates a series of VLU tasks uniformly into an open-book affinitymatching problem. Here, we adopt a knowledgebase prototype matrix to record the salient characteristics for each class by visual-textual fusion features, then perform affinity matching between image-text pairs and prototypes of each class. We further utilize the implicit sorting information of ground-truth labels by contrastive learning to provide more supervised cues from low-resource training sets. During model training, all parameters of textual and visual encoders in CLIP are fixed. Hence, XtremeCLIP is extremely parameterefficient. We conduct extensive experiments on a visual entailment (VE) benchmark (i.e., SNLI-VE), a
6368
visual question answering (VQA) benchmark (i.e., VQA v2), and three widely used image classification (IC) benchmarks (i.e., EuroSAT, DTD, and FGVC). Results show XtremeCLIP consistently outperforms baselines in low-resource scenarios.

heading: 2 XtremeCLIP: The Proposed Method
text: The model architecture and training procedure of XtremeCLIP are in Figure 1. First, a knowledgebase prototype matrix is constructed (Snell et al., 2017) by combining visual and textual features, designed to serve as a repository of the key characteristics for each class. Then, open-book affinity matching is performed between the image-text instance and the prototypes for each class.

heading: 2.1 Prototype Matrix Construction
text: Given a set of N image-text training instances: D = {(imgi, txti), li}Ni=1, where li denotes the ground-truth label, txti denotes the corresponding textual description of the image imgi. Image-text pairs are encoded using visual V and textual T encoders of CLIP:
v1 = V(imgi), v2 = T (txti)), v1, v2 ∈ Rd (1)
A fusion function F is employed to obtain uniform image-text representations that capture the interactions between visual and textual information:
F(v1, v2) = [v1, v2, v1+v2, v1−v2, v1×v2] (2)
where F(v1, v2) ∈ R5d. These fusion features are used to construct the knowledge-base prototype
matrix denoted as WP by averaging them per their ground-truth labels:
Mc = ∑N i=1 I(li = c) · F(V(imgi), T (txti))∑N
i=1 I(li = c) (3)
WP = [M1, · · · ,MC ] , (WP ∈ RC×5d) (4) where C denotes the number of classes, Mc denotes the prototype of the c-th class and c ∈ 1 · · ·C, I(·) denotes the indicator function, and [·] denotes the concatenation operator.

heading: 2.2 Open-book Matching

heading: Prototype Matching for VE and VQA. In
text: VE or VQA, affinity matching is performed between the fusion feature of a given image-text pair and the prototypes for each class: Pi = F(V(imgi), T (txti)) ·W⊤P . Prototype Matching for IC. In traditional IC tasks, only images are provided without corresponding textual descriptions. We obtain textual descriptions (prompts) for all classes, followin Radford et al. (2021). Given an image imgi and the textual descriptions of all image categories {tc|c = 1 · · ·C}, the predicted probability (denoted as Pi,c) of imgi w.r.t. the c-th image category is as follows: Pi,c = F(V(imgi), T (tc)) ·M⊤c . Thus, the entire probabilistic distribution Pi is: Pi = [Pi,c|c = 1 · · ·C].

heading: 2.3 Training Paradigm
text: XtremeCLIP has only one set of tunable parameters, namely the Prototype Matrix denoted by Wp. Its fusion function, visual, and textual encoders are solely utilized for constructing the prototype
matrix, with all parameters frozen during the training phase. In XtremeCLIP, the model is trained using the Cross-Entropy (CE) loss given Pi. The sample-wise CE loss is defined as follows:
LCE = − ∑
c∈{1···C} li,c · logPi,c (5)
where li,c denotes the ground-truth label w.r.t. the c-th class. However, the model can hardly achieve satisfactory performance with only supervised signals from CE in low-resource tasks. Given that instances’ affinity with ground-truth classes should be ranked higher than other classes, this implicit sorting information can be utilized to guide the model to recognize instances’ ground-truth classes via contrastive learning (Zhong et al., 2020). We define the affinity of the ground-truth category (i.e., the prototype matching probability, denoted as Pi,l) as positive samples and other affinities in Pi as negative samples. Following Liu et al. (2022); Liu and Liu (2021), the sample-wise Contrastive Learning (CL) loss is computed as:
LCL =
C∑
c=1
max(0, Pi,l − Pi,c). (6)
The total loss function for XtremeCLIP, namely L, is defined as: L = LCE + LCL.

heading: 3 Experiments

heading: 3.1 Experimental Settings
text: We briefly describe the experimental settings and leave more details in Appendix. Datasets. SNLI-VE (Xie et al., 2018) is utilized for visual entailment, consisting of image-text pairs whereby a premise is defined by an image. VQA v2 (Goyal et al., 2017) is utilized for visual question answering, containing questions about images. Here, we only consider the yes/no samples. Questions with open answers require decoder models and are not the focus of this paper For IC, EuroSAT (Helber et al., 2019) contains satellite images consisting out of 10 categories. DTD (Cimpoi et al., 2014) contains describable textures images with 47 classes. FGVC (Maji et al., 2013) contains images of 102 aircraft model variants. Baselines. In our work, we compare XtremeCLIP with zero-shot CLIP (Radford et al., 2021); finetuning paradigms including standard fine-tuning, mixout (Lee et al., 2020), pre-trained weight decay (weight decay) (Lee et al., 2020) and Layerwise Learning Rate Decay (LLRD) (Clark et al.,
2020); partial-parameter fine-tuning paradigms including BitFit (Ben Zaken et al., 2022) and BiNor (Song et al., 2022); and adapter-based methods including CLIP-Adapter (Gao et al., 2021) and Tip-Adapter (Zhang et al., 2022). Backbone. For fair comparison, all baselines and our approach adopt the ViT-B/16 (ViT-Base with the patch size 16 × 16) version of CLIP. Other versions of CLIP are also experimented with.

heading: 3.2 Experimental Results
text: VE&VQA results in low-resource settings. Table 1 presents the results of XtremeCLIP and baselines, in low-resource VE and VQA. The finetuning paradigms perform worse than partial finetuning paradigms in all settings, which demonstrates conventional fine-tuning paradigms are datahungry and not suitable for low-resource VLU tasks. XtremeCLIP consistently outperforms partial fine-tuning and adapter-based methods, showing that reformulating VLU tasks as prototype affinity matching can efficiently utilize visual-textual information with much fewer trainable parameters. Few-shot IC. Table 2 presents the performance of XtremeCLIP and baselines, in few-shot IC. Finetuning paradigms are still not suitable for fewshot image classification. Unlike BiNor and BitFit, CLIP-Adapter and Tip-Adapter specifically utilize adapters to learn from low-resource datasets meanwhile preserving the knowledge of CLIP, thus performing the best among baselines. Although XtremeCLIP has fewer trainable parameters than baselines, it still performs the best thanks to the supervised cues provided by contrastive learning and our task modeling approach. Ablation study. We replace the prototype matrix of XtremeCLIP with a randomly initialized matrix i.e., XtremeCLIP w/o. proto). We also detach the contrastive loss from XtremeCLIP (i.e., XtremeCLIP w/o. cl), or replace the fusion feature with the concatenation of visual and textual features (i.e., XtremeCLIP w/o. fusion). Table 3 presents the results of XtremeCLIP and its ablations. Detaching contrastive loss drops the performance, as contrastive learning provides more supervised cues. Reformulating VLU tasks as prototype affinity matching is somehow an open-book retrieval problem, which can augment model performance (Chen et al., 2022). Replacing the fusion feature drastically drops performance for VLU, which demonstrates the importance of interaction
between visual and textual information. Model-scale study. We test various CLIP versions with results in Table 4. The settings are the same as in ablation study. It shows that XtremeCLIP can be effectively adapted to different CLIPs and consistently has good performance. Data-scale study. Figure 2 presents the influence of the number of training instances on XtremeCLIP. As the number of training data increases, the accuracy of XtremeCLIP on VE and FGVC signifi-
cantly increases while full fine-tuning only slightly increases, which demonstrates XtremeCLIP has higher data-using efficiency.

heading: 4 Conclusion
text: We propose XtremeCLIP, a simple and efficient paradigm that reformulates VLU tasks as a prototype affinity matching problem. We adopt contrastive learning to leverage implicit sorting information from ground-truth labels, providing more supervised cues to handle insufficient supervised signals in small datasets. Experimental results demonstrate that XtremeCLIP consistently outperforms all baselines in low-resource scenarios.

heading: Limitations
text: In this paper, the proposed XtremeCLIP framework is mainly focused on CLIP-based deterministic VLU tasks. In future work, we will extend XtremeCLIP to other Pre-trained Vision-Language models and apply XtremeCLIP to generative tasks such as image captioning, visual grounding or visual relation extraction.

heading: Acknowledgments
text: This work was supported by the National Natural Science Foundation of China under Grant No. 62202170 and Alibaba Group through the Alibaba Innovation Research Program.

heading: A Case Study
text: Figure 3 presents the probability distributions of several images before and after fine-tuning of our approach. The constructed knowledge-base prototype matrix indeed captures the salient characteristics of categories. Based on the knowledge, images can be correctly classified even in zero-shot learning. After fine-tuning, the performance of XtremeCLIP is further boosted.

heading: B Experimental Details

heading: B.1 Training Corpora
text: We collect the pre-processed IC training corpora (i.e. FGVC (Maji et al., 2013), EuroSAT (Helber et al., 2019) and DTD (Cimpoi et al., 2014)) from the open-sourced project of (Zhang et al., 2022) on Github 2. The hand-crafted prompt templates that describe the category names for EuroSAT, DTD, and FGVC are listed in Table 5. During model training, we randomly select 8 and 16 images of each category for few-shot IC.
For visual entailment and visual questionanswering tasks, we download the pre-processed SNLI-VE (Xie et al., 2018) and VQA v2 (Goyal et al., 2017) from the open-sourced project XVLM (Zeng et al., 2021) on Github 3 and randomly select 2000, 5000, and 10000 samples from each dataset for low-resource VLU tasks.
The statistics are listed in Table 6.

heading: B.2 Experimental Details of Our Approach
text: We employ ViT-B/16 from OpenAI CLIP 4 as the default underlying model. We train XtremeCLIP by AdamW algorithm with β1 = 0.9, β2 = 0.999, ϵ = 1e − 4. The training is processed on an NVIDIA Tesla A100 GPU. We run XtremeCLIP 20 epochs for VE and VQA with a batch size of 16 and it takes around 20 minutes; and 100 epochs for IC with a batch size of 16 and it takes around 60 minutes.
2https://github.com/gaopengcuhk/TipAdapter/DATASET.md
3https://github.com/zengyan-97/x-vlm 4https://github.com/openai/CLIP

heading: B.3 Experimental Details of Baselines
text: For full fine-tuning paradigms (i.e. Mixout (Lee et al., 2020), pre-trained weight decay (weight decay) (Lee et al., 2020), layerwise Learning rate decay (LLRD) (Clark et al., 2020)) and partial parameter fine-tuning paradigms (i.e. BiNor (Song et al., 2022), BitFit (Ben Zaken et al., 2022), Linear Probe (Radford et al., 2021)), we set the learning rate for CLIP parameters as 5e− 7 and the learning rate for classification head as 2e− 3 after the grid search. We train full fine-tuning baselines and partial fine-tuning baselines by AdamW algorithm with β1 = 0.9, β2 = 0.999, ϵ = 1e − 4. We run the aforementioned baselines 20 epochs for VE and VQA with a batch size of 16, and 100 epochs for IC with a batch size of 16.
For CLIP-Adapter (Gao et al., 2021) 5 and TipAdapter (Zhang et al., 2022) 6, we directly take their open-sourced codes on GitHub. Though TipAdapter is proposed for few-shot IC only, by replacing the image features with the visual-textual fusion features of the input image-text pairs when constructing the instance retrieval matrix, it can be directly utilized for other VLU tasks as well.
To adapt CLIP-Adapter to VE and VQA, we respectively apply visual and textual adapter to the visual V and textual encoder T of CLIP to learn adaptive visual and textual features, then weight sum the adaptive visual and textual features with the original visual and textual feature from CLIP, following the original paper. Thereafter, we get the visual-textual fusion representations by the fusion function F . Finally, we perform image-text pair classification with the classification head as in Gao et al. (2021).

heading: B.4 Fusion Function Ablation
text: Table 7 shows the results of XtremeCLIP with various fusion functions, including traditional higher order and element-wise exponential operations. The results indicate that the selected fusion func-
5https://github.com/gaopengcuhk/CLIP-Adapter 6https://github.com/gaopengcuhk/Tip-Adapter
tion, namely F in Eq. 2, is both simple and highly effective, outperforming the others.

heading: B.5 Additional Comparasion
text: Table 8 presents the image classification results of XtremeCLIP and the baseline methods, namely CoOp (Zhou et al., 2022b), and Linear Probe (Gao et al., 2021), which are solely utilized for image classification. The results demonstrate that reformulating image classification as an open-book matching paradigm indeed helps XtremeCLIP consistently outperform CoOp and Linear Probe.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The fifth section "Limitation"
3 A2. Did you discuss any potential risks of your work? The fifth section "Limitation"
3 A3. Do the abstract and introduction summarize the paper’s main claims? The first section "Introduction"
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts?

heading: Left blank.
text: B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments?

heading: The third section "Experiments"
text: 3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The third section "Experiments" and the Appendix B2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? the Appendix B2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The third section "Experiments"
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The third section "Experiments"
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we will extend XtremeCLIP to other Pre-trained Vision-Language models and apply XtremeCLIP to generative tasks such as image captioning, visual grounding or visual relation extraction