Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization

## Abstract

Abstract Efficient fine-tuning of vision-language models (VLMs) like CLIP for specific downstream tasks is gaining significant attention. Previous works primarily focus on prompt learning to adapt the CLIP into a variety of downstream tasks, however, suffering from task overfitting when fine-tuned on a small data set. In this paper, we introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization, while a self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR . Specifically, trainable orthogonal matrices are injected seamlessly into the transformer architecture and enforced with orthogonality constraint during the training, benefiting from the norm-preserving property and thus leading to stable and faster convergence, while keeping the pre-trained weights frozen.
To alleviate deviation from fine-tuning,
a self-regularization strategy is further employed to retain the generalization of the model during the training within a bypass manner. In addition, to enrich the sample diversity for downstream tasks under the small dataset scenario, we first explore attentive CutOut data augmentation to boost the efficient fine-tuning, leading to better model fitting capacity for specific downstream task. Then we support the theoretical analysis on how our approach improves the specific downstream performance and maintains the generalizability. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario on par with the elaborated prompt learning methods. We conduct extensive experiments to demonstrate that our method explicitly steers pretrained weight space to represent the task-specific knowledge and presents competitive generalizability under base-to-base/base-to-new , cross-dataset transfer and domain generalization evaluations.

## 1 Introduction

Large-scale pre-trained vision-language models (VLMs) have been emerging as prevalent cornerstones in a wide spectrum of downstream vision and vision-language tasks, including few-shot image recognition Zhou et al. (; ); Zhang et al. (); Gao et al. (); Khattak et al. (); Zhu et al. (); Samadh et al. (); Manli et al. (); Cheng et al. (); Wang et al. (), object-detection Feng et al. (); Gu et al. (); Bangalath et al. (); Zang et al. () and segmentation Ding et al. (); Boyi et al. (); Rao et al. (); Wang et al. ().
Leading models like CLIP Radford et al. () and ALIGN Jia et al. () demonstrate remarkable generalizability by training with aligning image-text pairs from large web corpora using contrastive loss, thereby encoding open-vocabulary concepts within a joint vision-language embedding space. Despite the effectiveness of these VLMs in zero-shot recognition, fine-tuning them for specific downstream tasks while preserving their strong zero-shot capabilities remains a significant challenge. Designing manual text prompts for different tasks requires substantial human effort and expert knowledge, which is often infeasible for achieving optimal performance in data-efficient settings Brown et al. ().

Recently, prompt learning Zhou et al. (; ) serves as an exceptional paradigm to achieve this objective, however, tending to prioritize task-specific knowledge and resulting in task overfitting issues Park et al. (); Khattak et al. (), where the fine-tuned model struggles to generalize well to new/unseen tasks under data-efficient settings.
To address this dilemma, alternative approaches must be explored. Drawing inspiration from empirical observations that hyperspherical similarity effectively encodes semantic information Chen et al. (); Liu et al. (; ) and that hyperspherical energy Liu et al. () can characterize the pairwise relational structure among neurons, we hypothesize that well-pretrained models like CLIP should maintain consistent levels of hyperspherical energy even after fine-tuning. An intuitive approach is to use a suitable regularizer to preserve hyperspherical energy levels during the fine-tuning phase. However, ensuring that the difference in hyperspherical energy is minimized remains a challenge. Inspired by recent orthogonal transformation methods Qiu et al. (); Liu et al. (), we propose that the pretrained pairwise hyperspherical energy can be preserved by leveraging orthogonal transformation for all neurons with the same operation. This approach utilizes the invariance property of orthogonal transformation, meaning norm-preserving during fine-tuning, to maintain consistent hyperspherical energy levels.

Motivated by the preservation of hyperspherical energy through orthogonal transformation, we introduce Orthogonality Learning to adapt pretrained VLMs (e.g., CLIP) to specific downstream tasks (e.g., few-shot image recognition) without altering their hyperspherical energy, thanks to the norm-preserving property during fine-tuning. This approach differs from common methods that heavily rely on prompt learning. Furthermore, previous works Lin et al. (); Liu et al. (; ) have shown that small hyperspherical energy leads to better generalization, and orthogonal transformation is a suitable and flexible solution for achieving this, especially in classification task. Our main idea is to apply the same orthogonal transformation to neurons so that pairwise angles are maintained within the hypersphere of CLIP.
Although prevalent adaptation methods for pretrained weights, such as LoRA Hu et al. (), achieve fine-tuning by adding small component matrices, they still suffer from low training convergence and generalizability degradation.

Figure: Figure 1: The pipeline comparison for tuning or adapting VLMs into downstream tasks. Our contribution is to introduce a new fine-tuning pipeline by orthogonal tuning, that boost the CLIP and CoOp with competitive base/novel accuracy performances when compared with existing methods (results are computed by average 11 datasets).
Refer to caption: x1.png

In this paper, we propose a novel and efficient fine-tuning method using Orthogonality Learning, motivated by the preservation of hyperspherical energy through orthogonal transformation, shown different paradigm with exisitng works in Fig. 1 (a). To mitigate deviation from orthogonal constraint during training, we introduce a Self-Regularization strategy using the initial pretrained weights as an anchor point, thus dubbed OrthSR. Our method keeps the pretrained weights frozen while applying orthogonal fine-tuning and regularization simultaneously. In the dual-branch transformer architecture of the CLIP model, we inject trainable orthogonal matrices and enforce orthogonal constraints (such as using Cayley parameterization Helfrich et al. (); Lezcano-Casado & Martınez-Rubio ()). This ensures each injected layer matrix is orthogonal with a determinant of 1.
We investigate orthogonal fine-tuning in both image and text encoder of CLIP to demonstrate training efficiency and generalizability preservation of our method, distinguishing it from prompt tuning and low-rank matrix decomposition methods. The norm-preserving property of orthogonal transformations helps maintain hyperspherical energy levels, benefiting of stable convergence, robustness, and generalization. This enables seamless integration of task-specific knowledge into pretrained VLMs, allowing the trainable matrices to be merged with frozen weights during deployment without adding inference latency, while we shows evaluation superiority over previous methods in Fig. 1 (b). To prevent significant deviations from the pretrained model, we employ a Self-Regularization strategy that guides the model to stay close to the anchor point, supported by the pretrained model within a bypass manner.
This simple yet effective approach sustains orthogonal fine-tuning with initial anchor regularization, avoiding deviations from the zero-shot generalizability manifold severely. Besides, we utilize attentive CutOut data augmentation to enrich the data diversity, enhancing the task-specific knowledge of fine-tuned model (e.g., few-shot image recognition) under data-efficient setting. This leads to better model fitting capacity for specific downstream task, serving as implicitly increasing the sample diversity.
Unlike previous works Qiu et al. (); Liu et al. (), we focus on adapting VLMs to high-level task-specific scenarios (e.g., recognition) rather than fine-tuning generative models. Additionally, we devise a suitable regularization strategy to retain the strong generalizability that elucidates the training efficiency and generalizability preservation of our method.

Extensive experiments demonstrate the effectiveness of our OrthSR by evaluating on representative benchmarks: base-to-base/base-to-new, cross-dataset transfer and domain generalization. In the base-to-base/base-to-new setting, our method improves the new class of baseline model by 13.3% on average across 11 datasets, by 0.95% for cross-dataset setting and 1.80% on average across the four datasets for domain generalization setting, all of which presents competitive performance over the existing SoTAs. In summary, our contributions can be summarized as follows:

- •
We introduce a novel and efficient orthogonal fine-tuning method to adapt the VLMs into task-specific knowledge while maintaining strong generalizability. Due to the norm-preserving property, this fine-tuning leads to stable and faster convergence and exhibits superiority over the prompt tuning methods.
- •
To further mitigate the deviation from the pretrained model, we design a Self-Regularization strategy to enforce the fine-tuned model distilling informative zero-shot generalization information of the pretrained logits.
- •
Attentive CutOut data augmentation is employed to enhance the task-specific knowledge when fine-tuning the VLM under data-efficient setting.
- •
Extensive experiments are conducted to validate the effectiveness and effciency of our method, for the first time, we boost the CLIP and CoOp with weight decomposition tuning to obtain on par or even superior performances over existing methods.

## 2 Related works

Vision language models.
Recently, with a significant upsurge of large-scale pretrained vision-language models (VLMs) Yuan et al. (); Zhang et al. (); Jia et al. (); Cherti et al. (); Radford et al. (); Touvron et al. (), text and image embeddings have been trained jointly to be aligned with the large-scale image-text pairs corpora.
Driven by contrastive loss in a self-supervised manner, VLMs like CLIP Radford et al. (), ALIGN Jia et al. (), LiT Zhai et al. (), FLIP Li et al. () and Florence Yuan et al. () have elucidated remarkable performance. For instance, CLIP Radford et al. () and ALIGN Jia et al. () utilize approximately 400 million and 1 billion image-text pairs, respectively, to accomplish their multi-modal alignment training, benefiting a wide spectrum of downstream vision and vision-language tasks, including few-shot image-level recognition Zhou et al. (; ); Zhang et al. (); Gao et al. (); Khattak et al. (); Zhu et al. (); Samadh et al. (); Manli et al. (); Cheng et al. (); Wang et al. (), object detection Feng et al. (); Gu et al. (); Bangalath et al. (); Zang et al. () and segmentation Ding et al. (); Boyi et al. (); Rao et al. (); Wang et al. (). Despite strong generalizability towards zero-shot recognition tasks of these VLMs, effectively transferring them to downstream tasks without degrading their inherent generalization ability remains a challenging problem.

Efficient tuning for vision language models. With the emergence of VLMs, efficiently adapting these models to specific downstream tasks with limited data samples has garnered significant interest.
Prompt Tuning is firstly proposed in the NLP field Liu et al. (); Gao et al. (); Li & Liang (); Lester et al. (), which attempts to learn task-specific prompt templates.
Recently, in the computer vision community, CoOp Zhou et al. () pioneers the study by tuning the contextual tokens in text branch of CLIP into a set of learnable tokens to few-shot image recognition, which is further improved by CoCoOp Zhou et al. () through a Meta-Network Munkhdalai & Yu () paradigm to address the overfitting issue on base classes while generalizing better on unseen classes. To efficiently adapt large pretrained Vision Transformers, VPT Jia et al. () and Visual Prompting Bahng et al. () both insert trainable tokens into the input space of transformer model.
To leverage additional prompt learning for dual-branch models like CLIP, a plethora of works  Khattak et al. (; ); Cho et al. (); Zang et al. (); Park et al. (); Zhu et al. (); Lu et al. (); Wang et al. () have been proposed to learn these prompts towards a way that treats them as continuous learnable vectors while keeping the original model parameters frozen to retain the strong generalizability. Very recently, Test-Time Prompting Shu et al. (); Samadh et al. () emerges with the objective of enforcing consistency regularization between multiply views of a test sample by minimizing their averaged entropy.
Another line of work Brown et al. (); Devlin et al. (); He et al. () focuses on tuning VLMs over the pretrained weights. Adaptation methods Houlsby et al. (); Hu et al. (); Poth et al. () have become increasingly ubiquitous. The LoRA series Hu et al. (); Liu et al. (); Dettmers et al. () is widely used to finetune pretrained model weights using low-rank matrix optimization.
Our method shares a similar principle with LoRA for adapting pretrained model weights, but introduces a novel Orthogonality Learning approach. This not only enhances performance for specific downstream tasks (e.g., few-shot recognition) but also improves robustness and generalization with more efficient convergence.
Orthogonality regularization. Orthogonality has been commonly adopted to introduce orthogonal regularization to improve the robustness of Deep Neural Networks Liu et al. (); Brock et al. (); Huang et al. (); Xie et al. (); Huang et al. (); Lezcano-Casado & Martınez-Rubio (); Arjovsky et al. (); Wisdom et al. (); Qi et al. (); Li et al. (), that norm-preserving property can avoid exploding or vanishing gradients during training Bengio et al. (); Glorot & Bengio (), leading to faster convergence and encouraging robustness and generalization. This objective can be reached by a simple Cayley parameterization Helfrich et al. (); Lezcano-Casado & Martınez-Rubio (). Recently,OPT Liu et al. () introduces an orthogonal transformation applied to the neural weights to maintain the minimum hyperspherical energy. Furthermore, OFT Qiu et al. () extend this orthogonal paradigm to finetune the text-to-image diffusion models by employing Cayley parameterization constraint during the finetuning. In this paper, we further explore the utilization of orthogonal finetuning on CLIP for specific downstream tasks while proposing different regularization strategies to enhance generalizability on novel/uneen classes.

## 3 Methodology

### 3.1 Preliminaries

Contrastive Language-Image Pre-training (CLIP).
CLIP consists of two parallel encoders, image and text encoders, represented by $\mathcal{\theta}_{CLIP}=\{{\theta_{v},\theta_{t}}\}$. The image encoder $\mathcal{F}_{v}$ can be either a CNN He et al. () or a ViT Vaswani et al. (); Dosovitskiy et al. () for mapping input image into a image embedding, and the text encoder $\mathcal{F}_{t}$ is a Transformer Devlin et al. () for mapping input text into a text embedding, respectively. During pre-training, CLIP utilizes two parallel encoders to separately encode image and text into corresponding vectors in jointly aligned embedding space, and then adopts contrastive loss to pull together the cosine similarities of the correct image-text vector pairs while pushing away the cosine similarities of incorrect pairs. After pretrained on large-scale image-text pairs corpora, CLIP is capable of computing the text-image similarity and can be generalized to downstream tasks, like zero-shot image recognition, without fine-tuning. Specifically, the input image ${X}$ is first divided into $M$ patches and then projected into patch tokens, and a global class token $[CLS]$ is prepended to the patch token sequence, obtaining ${X}^{~{}}_{0}=\{CLS,e_{1},e_{2},...,e_{M}\}$ where $e_{i}$ standds for the ${i}^{th}$ patch. Those patch tokens will be encoded by transformer blocks inside the image encoder $\mathcal{F}_{v}$ by ${f}^{~{}}_{v}={\mathcal{F}}_{v}({X}^{~{}}_{0}:\theta_{v})$. Given the labels $\{[class]_{c}\}^{C}_{c=1}$ for the $C$ categories for classification where $[class]_{c}$ represents the class
name of the $c^{th}$ class, a hand-crafted text prompt like ‘a photo of a $[CLS]$’ will be embedded within the class label $[class]_{c}$ This results in $\mathcal{Y}^{~{}}_{0}=\{SOS,t_{1},t_{2},...,t_{L},c_{k},EOS\}$ where $SOS$ and $EOS$ denote the start and end token embeddings while $t_{i}$ and $c_{k}$ are specific word embedding corresponding to the text prompt and the class label, respectively. The text encoder $\mathcal{F}_{t}$ will encode $\mathcal{Y}^{~{}}_{0}$ via transformer blocks to produce text feature embeddings as ${f}^{~{}}_{t}=\mathcal{F}_{t}(\mathcal{Y}^{~{}}_{0}:\theta_{t})$. During zero-shot inference, the prediction probability on image ${X}$ will be computed as ${p}(y_{i}|X)=\frac{exp(sim(f_{t}\cdot f_{v})/\tau)}{\sum^{C}_{i=1}exp(sim(f^{~
{}}_{t}\cdot f^{~{}}_{v})/\tau)}$, where $\tau$ is a learned temperature coefficient and $sim$ denotes the cosine similarity computation, respectively.

Context Optimization (CoOp) Zhou et al. () proposes to leverage tunable text prompt by replacing the cumbersome and fixed hand-crafted prompt, that can be learnt from data. Now, the tunable prompt is constructed with $M$ learnable continues context vectors as $w=\{w_{1},w_{2},...,w_{M},c_{k}\}$, where $w_{i}$ represents the $i^{th}$ tunable vector and $c_{k}$ denotes the $c^{th}$ class name $[class]_{c}$. The finally fine-tuned training objective of CoOp is to optimize the contextual vectors $w_{i}$ only by minimize the cross-entropy loss between the ground-truth $\hat{y}$ and the model prediction $y$ as:

$$ ${p}(y_{i}|X)=\frac{exp(sim(f^{~{}}_{t}(:w)\cdot f^{~{}}_{v})/\tau)}{\sum^{C}_{ i=1}exp(sim(f^{~{}}_{t}(:w)\cdot f^{~{}}_{v})/\tau)},\quad\mathcal{L}_{\text{ ce}}=-\log{p(\hat{y}=y|{X})}$ (1) $$

### 3.2 Orthogonal fine-tuning

Traditionally, fine-tuning VLMs into specific downstream scenarios typically embraces small learning rate with gradient descent optimizer to update the model, This scheme implicitly constrains risky deviation from pretrained model, aiming to finetune the model via implicitly minimizing $\|\bm{M}-\bm{M}_{0}\|$ where $\bm{M}$ is the fine-tuned model weights and $\bm{M}_{0}$ is the pretrained model weights. Towards this strategy, there are still various ways to finetune a pretrained VLM. For example, LoRA Hu et al. () employs an additive low-rank matrix with constraint for model weights update, i.e., $\text{rank}(\bm{M}-\bm{M}_{0})=r^{\prime}$ where $r^{\prime}$ is set to be relatively smaller number than the pretrained ones. Differently, Orthogonal transformation targets at inducing a constraint for the pairwise similarity between neurons Liu et al. (); Qiu et al. (): $\|\text{HE}(\bm{M})-\text{HE}(\bm{M}_{0})\|=0$ ,where $\text{HE}(\cdot)$ denotes hyperspherical energy of a weight matrix. In this paper, we draw attention to the Feed-Forward-Networks (FFN) within the transformer architecture of CLIP, shown in Fig 2. Suppose a fully-connected layer with $\bm{W}=\{\bm{w}_{1},\cdots,\bm{w}_{n}\}\in\mathbb{R}^{d\times n}$ where $\bm{w}_{i}\in\mathbb{R}^{d}$ is the $i^{th}$ neuron ($\bm{W}_{0}$ is the pretrained weights). We expect to acquire the output vector $\bm{z}\in\mathbb{R}^{n}$ by $\bm{z}=\bm{W}^{\top}\bm{x}$ where $\bm{x}\in\mathbb{R}^{d}$ is the input vector. When introducing the orthogonal fine-tuning as minimizing the hysperical energy difference between the fine-tuned and pretrained model:

$$ $\min_{\bm{W}}\left\|\text{HE}(\bm{W})-\text{HE}(\bm{W}_{0})\right\|~{}~{} \Leftrightarrow~{}~{}\min_{\bm{W}}\bigg{\|}\sum_{i\neq j}\|\hat{\bm{w}}_{i}- \hat{\bm{w}}_{j}\|^{-1}-\sum_{i\neq j}\|\hat{\bm{w}}^{0}_{i}-\hat{\bm{w}}^{0}_ {j}\|^{-1}\bigg{\|}$ (2) $$

Figure: Figure 2: Overview of our proposed pipeline, OrthSR. The top shows our fine-tuning pipeline by applying orthogonal tuning into the Feed-Forward-Network of both image and text encoder ($\mathcal{F}_{v}$ and $\mathcal{F}_{t}$) of CLIP model which is trained with Self-Regularization strategy. On the left of bottom, orthogonal matrix injection is explained by injecting orthogonal matrix into the pretrained weights with orthogonalization constraint (such as Cayley parameterization). On the right of bottom, pretrained CLIP is utilized to highlight the most-discriminative image regions and then apply cutout operation to obtain cutout image $X_{cutout}$ which will be input to the fine-tuned model together with original $X$.
Refer to caption: x2.png

where $\hat{\bm{w}}_{i}=\frac{\bm{w}_{i}}{\|\bm{w}_{i}\|}$ is the $i^{th}$ normalized weight, and the hyperspherical energy of a fully-connected layer $\bm{W}$ is defined as $\text{HE}(\bm{W}):=\sum_{i\neq j}\|\hat{\bm{w}}_{i}-\hat{\bm{w}}_{j}\|^{-1}$. This objective can be optimally minimized to be zero. To achieve this target, we introduce the orthogonal transformation into the pretrained weights, $\bm{W}=\bm{A}\bm{W}_{0}$ in which $\bm{A}\in\mathbb{A}^{d\times d}$ is an orthogonal matrix, meaning that the determinant is $1$ or $-1$ of the initial matrix by imposing rotation or reflection, respectively. Now we can formulate the forward pass of FFN from $\bm{z}=(\bm{W}_{0})^{\top}\bm{x}$ to:

$$ $\bm{z}=\bm{W}^{\top}\bm{x}=(\bm{A}\cdot\bm{W}_{0})^{\top}\bm{x},~{}~{}~{}\text {s.t.}~{}\bm{A}^{\top}\bm{A}=\bm{A}\bm{A}^{\top}=\bm{I}$ (3) $$

where $\bm{W}$ denotes the fine-tuned weight matrix and $\bm{I}$ is an identity matrix. During the fine-tuning, we optimize the added $\bm{A}$ while keeping the pretrained weights $\bm{W}_{0}$ frozen. To finetune the model from $\bm{W}_{0}$, we initialize the orthogonal matrix $\bm{A}$ to be identity matrix $\bm{I}$, sharing similar principle with LoRA to set zero initialization of the additive matrices. Moreover, this allows us to gradually inject task-specific knowledge
into the fine-tuned model driven by cross-entropy loss.

Motivated by previous works Liu et al. (); Lezcano-Casado & Martınez-Rubio (); Helfrich et al. () discussing about differential orthogonalization methods, we focus on taking utilization of Cayley parameterization. The Cayley transform produces a representation of orthogonal matrices without $-1$ eigenvalues using skew-symmetric matrices (i.e., $\bm{C}^{\top}=-\bm{C}$) as follows:

$$ $\bm{A}=(\bm{I+C})^{-1}(\bm{I-C}),\bm{C}=(\bm{I+A})^{-1}(\bm{I-A})$ (4) $$

wherein we find this special orthogonal group is able to obtain competitive performances when adapting CLIP for downstream tasks (e.g., few-shot image recognition). Based on the orthogonal fine-tuning above to adapt the VLM into downsream scenario, we find there exists a potential risky error bounding such that the fine-tuned model presents inferior generalizability on new/unseen classes, shown in our experimental part. After applying the Neumann series to analyze: $\bm{A}=(\bm{I+C})^{-1}(\bm{I-C})$ can be written as: $\bm{A}\approx\bm{I}+2\bm{C}+\mathcal{O}(\bm{C}^{2})$, We empirically observe that this approximation results in instability of the fine-tuning Singla & Feizi (), which degrades the zero-shot generalization of the pretrained model, showing different phenomena with previous work Qiu et al. () on fine-tuning generative models.

### 3.3 Self-Regularization

This inspires us to investigate the regularization strategy to carefully constrain the fine-tuned model not deviating far away from the pretrained one. Therefore, we further design a Self-Regularization strategy to regularize the fine-tuned model through pretrained model with a bypass manner since the pretrained weights are frozen. As shown in Fig 2, the text prompts are processed by frozen text encoder $\mathcal{F}_{t}$ to obtain text embedding ${f}_{t}$, while we can also compute new text embedding $f_{t}(:,A_{t})$ which is encoded by orthogonal tuning text encoder after injecting orthogonal matrix to each FFN layer, $\mathcal{F}_{t}+A_{t}$. Here, we want to optimize the additive $A_{t}$ for the text encoder. At the same time, we input original image to the image encoder, and obtain $f_{v}$ encoded by frozen $\mathcal{F}_{v}$ and $f_{v}(:,A_{v})$ from $\mathcal{F}_{v}+A_{v}$, enabling $A_{v}$ tunable only. Further, the pretrained and fine-tuned logit are computed as follows:

$$ $f_{zs_logit}=sim(f_{t}\cdot f_{v}),\quad f_{logit}=sim(f_{t}(:,A_{t})\cdot f_ {v}(:,A_{v}))$ (5) $$

Then, we adopts the cross-entropy loss to train the model given the class label $\hat{y}$ as:

$$ ${p}(y_{i}|X)=\frac{exp(sim(f^{~{}}_{t}(:,A_{t})\cdot f^{~{}}_{v}(:A_{v}))/\tau )}{\sum^{C}_{i=1}exp(sim(f^{~{}}_{t}(:,A_{t})\cdot f^{~{}}_{v}(:,A_{v}))/\tau) },\quad\mathcal{L}_{\text{ce}}=-\log{p(\hat{y}=y|{X})}$ (6) $$

To further impose regularization from the pretrained anchor point,Then Kullback-Leibler loss $\mathcal{L}_{kl}$ is used to distill informative zero-shot knowledge from the anchor point so as to alleviate deviation far away from the pretrained mainfold wthin a bypass manner, as follows:

$$ $\mathcal{L}_{kd}=\mathcal{D}_{kd}(f_{logit},f_{zs_logit})$ (7) $$

where $\mathcal{D}_{kd}(f_{logit}||f_{zs_logit})=\sum\limits_{x\in X}(g(f_{logit})
log{\frac{g(f_{logit})}{g(f_{zs_logit}})})$, $g(\cdot)$ denotes softmax function.

### 3.4 Cutout augmentation

As shown in Fig 2, we utilize the pretrained model to infer the similarity map by computing the cosine similarity between image patch tokens and $[CLS]$ text token, named as attentive CutOut. Then it produces a map that each patch responses to $[CLS]$ text token and then reshape them into the same shape of the input image. During the training, we randomly select a cutout region size to zero the top-$K$ image patches, where $K$ ranges from $[l,L]$. To enforce randomness to image encoder so that the model can pay more attention to other less-discriminative image regions, we generate random and different erasing size for each training iteration. Specifically, let $X_{cutout}$ be the cutout image. We input it into the image encoder with $\mathcal{F}_{v}+A_{v}$ and obtain $f_{v_cutout}(:,A_{v})$. After that, following the aforementioned way, we then calculate the cutout logit $f_{cutout_logit}$ as:

$$ $f_{cutout_logit}=sim(f_{t}(:,A_{t})\cdot f_{v_cutout}(:,A_{v}))$ (8) $$

Similarly, we acquire the cutout classification and Kullback-Leibler loss in terms of the cutout image $X_{cutout}$ as:

$$ $\mathcal{L}_{\text{cutout_ce}}=-\log{p(\hat{y}=y|{X_{cutout}})},\quad\mathcal {L}_{cutout_kd}=\mathcal{D}_{kd}(f_{cutout_logit},f_{zs_logit})$ (9) $$

In this way, we enforce the fine-tuned model pay more attention to other less-discriminative image regions that response weak to the text embedding but still contains informative cues to help model learn task-specific knowledge under the data-efficient setting, which serves as diversifying samples.

### 3.5 Training objective

Overall, the training losses of our method consist of two parts, one for the image classification loss including global image classification loss and cutout image classification loss, while the other one includes two corresponding distillation loss. We expect that introducing orthogonal tranformation into CLIP model fine-tuned for specific downstream tasks is able to retain strong generalizability preservation. Hence, the overall loss $\mathcal{L}_{final}$ can be written as:

$$ $\mathcal{L}_{\text{final}}=\lambda_{1}(\mathcal{L}_{ce}+\mathcal{L}_{cutout_ ce})+\lambda_{2}(\mathcal{L}_{kd}+\mathcal{L}_{cutout_kd})$ (10) $$

where $\lambda_{1}$ and $\lambda_{2}$ are loss balancing hyper-parameters, weighting the task-agnostic and task-specific knowledge learning.

### 3.6 Theoretical Analysis

In this section, we provide theoretical analysis for the generalization error bound of  OrthSR.

We define the following optimization objectives according to Eq. 10:

$$ $\min_{\Theta\in\mathbb{R}}\underset{\mathcal{L}_{CE}}{\underbrace{\frac{1}{N}{ \textstyle\sum_{i=1}^{N}}\mathcal{L}\left(\hat{s}_{i}^{S}\left(\Theta\right),y _{i}^{gt}\right)}}+\lambda\underset{\mathcal{L}_{KD}}{\underbrace{\mathcal{L} \left(\hat{s}^{S}\left(\Theta\right),\hat{s}^{T}\right)}},\vspace{0pt}$ (11) $$

where $\Theta$ represents learnable orthogonal matrices $\{A_{v},A_{t}\}$ of the proposed method, and we use $S$ and $T$ here to denote the fine-tuned model and pre-trained  anchor model. Now we further analyze the effectiveness of  OrthSR by computing the generalization error bound. This bound computes the bias between the generalization error $\varepsilon\left(\Theta\right):=\bm{\mathbb{E}}_{\left(\hat{s}^{S},y^{gt}
\right)\sim\mathcal{D}}\mathcal{L}\left(\hat{s}^{S}\left(\Theta\right),y^{gt}\right)$ and empirical error $\bar{\varepsilon}_{\chi}\left(\Theta\right):=\frac{1}{N}{\textstyle\sum_{i=1}^
{N}}\mathcal{L}\left(\hat{s}_{i}^{S}\left(\Theta\right),y_{i}^{gt}\right)$, where $D$ is the real data distribution and $\bm{\mathbb{E}}\left(\cdot\right)$ denotes the expectation function.

###### Theorem 1 .

Assume that $\Theta^{*}$ is the solution to Eq. equation 11. Then we have that for any $0<\epsilon<1$ with probability $1-\epsilon$,

$$ $\epsilon(\Theta^{*})-\bar{\epsilon}_{\chi}(\Theta^{*})\leq X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}+\frac{C^{\prime\prime}}{\lambda^{2\alpha}\sqrt{N}}.$ $$

where $X^{*}=\mathrm{max}_{r\in\mathbb{N}_{N}}\left|\mathcal{L}\left(\hat{s}_{r}^{S}
\left(\Theta\right),y_{r}^{gt}\right)\right|$ and $\alpha>0$.

The first term of the upper bound converges with the increasing of the number of training data $N$, that can be achieved by our proposed attentive CutOut data augmentation instead of using extra data. We can also find that the second term converges to 0 with the increasing of $\lambda$, which means the our self-regularization $\mathcal{L}_{KD}$
within a bypass manner effectively improves the generalization ability of our method.

###### Theorem 1 .

## 4 Experiments

### 4.1 Experimental settings

Datasets: For evaluation in terms of both base-to-base and base-to-new class generalization, we conduct our method on publicly available 11 image recognition datasets: ImageNet Russakovsky et al. () and Caltech101 Fei-Fei () for generic objects classification, Oxford_Pets Parkhi et al. (), StanfordCars Krause et al. (), Flowers102 Nilsback & Zisserman (), Food101 Bossard et al. () and FGVCAircraft Maji et al. () for fine-grained classification, SUN397 Xiao et al. () for scene recognition, DTD Cimpoi et al. () for texture classification, EuroSAT Helber et al. () for satellite imagery recognition and UCF101 Soomro et al. () for action recognition. Following the existing methods Zhou et al. (); Khattak et al. (; ); Cho et al. (); Zang et al. (); Park et al. (); Zhu et al. (); Lu et al. (); Wang et al. (), we also evaluate our method on cross-dataset transfer and domain generalization. For cross-dataset transfer, we adopt ImageNet as the source and the remaining 10 datasets as target variants,
while for domain generalization, we also use ImageNet as source and ImageNetV2 Recht et al. (), ImageNet-Sketch Wang et al. (), ImageNet-A Hendrycks et al. () and ImageNet-R Hendrycks et al. () as targets.

Implementation details:
For all the experimental settings, we follow the common strategy of CoOp Zhou et al. () and CoCoOp Zhou et al. () for the fair comparison, including the dataset splits, default data augmentation, training schedule, shot of samples, backbones, length of context tokens (i.e., $M$ is 16 in this paper), etc. The $K$ is set to be 3 and averaged for all the experiments, reporting base and novel class accuracy and their harmonic mean (HM), respectively. We apply CLIP-ViT-B/16 as our pretrained backbone model to train for 5 epochs with a batch size of 4, and a learning rate of 1e-5 via SGD optimizer on a single Nvidia-A100-GPU, unless other stated. The hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ are set to be 1.5 and 1.2 by default, left for hyper-parameters sensitivity ablations in Appendix A.

Baseline: To validate the effectiveness of proposed OrthSR, we compare our approach against the following methods, including: (1) zero-shot CLIP Radford et al. (), which provides the basic baseline model for comparison without any prompt learning or adaptation finetuning; (2) commonly used single-modal prompt tuning methods to demonstrate superiority of our novel finetuning method, such as CoOp Zhou et al. () which constructs another baseline model for us using tunable context vectors for the input text prompt, CoCoOp Zhou et al. (), PLOT Chen et al. () and UNIGRAM Li et al. (), and VPT Jia et al. (); and multi-modal prompt tuning methods: MaPLe Khattak et al. () and PromptSRC Khattak et al. (). Note that the original paper of PLOT Chen et al. () adopts a weaker backbone model ResNet-50 He et al. (), here we change it to ViT-B/16 to implement for a fair comparison. Moreover, we also implement VPT which applies prompt tuning for image encoder, IVLP which applies independent prompt tuning for both image encoder and text encoder, all of which establish the basic comparisons.

**Table 1: Performance for base-to-base/base-to-new on 11 datasets. We train our model with a subset of the classes (base classes) in a 16-shot setting and evaluate on the test set including base classes and new classes, while HM denotes the harmonic mean of base and novel performance to show the generalization trade-off Xian et al. (), HM=(2 $\times$ base $\times$ new )/(base + new). The highest results are highlighted in Bold.**
| Dataset |  | CLIP | CoOp | CoCoOp | MaPLe | RPO | PLOT | PromptSRC | UNIGRAM | VPT | IVLP | OrthSR | Gain |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Radford et al. () | Zhou et al. () | Zhou et al. () | Khattak et al. () | Lee et al. () | Chen et al. () | Khattak et al. () | Li et al. () | (Base) | (Base) | (Ours) | $\Delta$ |  |
| Average on<br>11 datasets | Base | 69.34 | 82.69 | 80.47 | 82.28 | 81.13 | 77.20 | 84.26 | 80.34 | 80.81 | 81.83 | 84.16 | +1.47 |
| New | 74.22 | 63.22 | 71.69 | 75.14 | 75.00 | 60.38 | 76.10 | 75.92 | 70.36 | 73.63 | 76.55 | +13.3 |  |
| HM | 71.70 | 71.66 | 75.83 | 78.55 | 77.78 | 67.76 | 79.97 | 78.07 | 74.68 | 77.10 | 80.02 | +8.36 |  |
| ImageNet | Base | 72.43 | 76.47 | 75.98 | 76.66 | 76.60 | 75.97 | 77.60 | 76.60 | 70.93 | 76.80 | 78.10 | +1.63 |
| New | 68.14 | 67.88 | 70.43 | 70.54 | 71.57 | 69.23 | 70.73 | 70.69 | 65.90 | 70.40 | 70.35 | +2.47 |  |
| HM | 70.22 | 71.92 | 73.10 | 73.47 | 74.00 | 72.44 | 74.01 | 73.53 | 68.32 | 73.46 | 74.02 | +2.10 |  |
| Caltech<br>101 | Base | 96.84 | 98.00 | 97.96 | 97.74 | 97.97 | 96.53 | 98.10 | 98.07 | 97.86 | 97.53 | 98.17 | +0.17 |
| New | 94.00 | 89.81 | 93.81 | 94.36 | 94.37 | 82.86 | 94.03 | 95.11 | 93.76 | 93.57 | 94.03 | +4.22 |  |
| HM | 95.40 | 93.73 | 95.84 | 96.02 | 96.03 | 89.17 | 96.02 | 96.57 | 95.77 | 95.51 | 96.06 | +2.33 |  |
| Oxford<br>Pets | Base | 91.17 | 93.67 | 95.20 | 95.43 | 94.63 | 93.45 | 95.33 | 94.94 | 94.81 | 95.50 | 95.60 | +1.95 |
| New | 97.26 | 95.29 | 97.69 | 97.76 | 97.50 | 79.76 | 97.30 | 97.94 | 96.00 | 97.97 | 97.70 | +2.41 |  |
| HM | 94.12 | 94.47 | 96.43 | 96.58 | 96.05 | 86.06 | 96.30 | 96.42 | 95.40 | 96.72 | 96.64 | +2.17 |  |
| Stanford<br>Cars | Base | 63.37 | 78.12 | 70.49 | 72.94 | 73.87 | 61.41 | 78.27 | 73.50 | 72.46 | 73.27 | 79.40 | +1.28 |
| New | 74.89 | 60.40 | 73.59 | 74.00 | 75.53 | 42.69 | 74.97 | 75.38 | 73.38 | 74.17 | 73.87 | +13.4 |  |
| HM | 68.65 | 68.13 | 72.01 | 73.47 | 74.69 | 50.37 | 76.58 | 74.43 | 72.92 | 73.72 | 76.54 | +8.41 |  |
| Flowers<br>102 | Base | 72.08 | 97.60 | 94.87 | 95.92 | 94.13 | 95.62 | 98.07 | 95.20 | 95.39 | 96.47 | 97.60 | +0.00 |
| New | 77.80 | 59.67 | 71.75 | 72.46 | 76.67 | 56.03 | 76.50 | 76.21 | 73.87 | 72.90 | 75.53 | +15.8 |  |
| HM | 74.83 | 74.06 | 81.71 | 82.56 | 84.50 | 70.56 | 85.95 | 84.65 | 83.26 | 83.04 | 85.16 | +11.1 |  |
| Food101 | Base | 90.10 | 88.33 | 90.70 | 90.71 | 90.33 | 88.45 | 90.67 | 90.84 | 89.88 | 90.47 | 90.50 | +0.40 |
| New | 91.22 | 82.26 | 91.29 | 92.05 | 90.83 | 85.28 | 91.53 | 92.12 | 87.76 | 91.97 | 91.17 | +8.91 |  |
| HM | 90.66 | 85.19 | 90.99 | 91.38 | 90.58 | 86.84 | 91.10 | 91.48 | 88.81 | 91.21 | 90.83 | +5.64 |  |
| FGVC<br>Aircraft | Base | 27.19 | 40.44 | 33.41 | 37.44 | 37.33 | 29.63 | 42.73 | 32.25 | 33.10 | 34.20 | 41.93 | +1.49 |
| New | 36.29 | 22.30 | 23.71 | 35.61 | 34.20 | 16.17 | 37.87 | 38.00 | 30.49 | 34.00 | 36.87 | +14.5 |  |
| HM | 31.09 | 28.75 | 27.74 | 36.50 | 35.70 | 20.92 | 40.15 | 34.89 | 31.74 | 34.10 | 39.24 | +10.4 |  |
| SUN397 | Base | 69.36 | 80.60 | 79.74 | 80.82 | 80.60 | 78.56 | 82.67 | 80.43 | 79.66 | 81.00 | 82.47 | +1.87 |
| New | 75.35 | 65.89 | 76.86 | 78.70 | 77.80 | 72.34 | 78.57 | 77.91 | 72.68 | 78.40 | 79.33 | +13.4 |  |
| HM | 72.23 | 72.51 | 78.27 | 79.75 | 79.18 | 75.32 | 80.52 | 79.15 | 76.01 | 79.68 | 80.87 | +8.36 |  |
| DTD | Base | 53.24 | 79.44 | 77.01 | 80.36 | 76.70 | 69.87 | 83.37 | 73.62 | 79.15 | 79.50 | 82.40 | +2.96 |
| New | 59.90 | 41.18 | 56.00 | 59.18 | 62.13 | 53.63 | 62.97 | 62.38 | 50.76 | 50.10 | 65.33 | +24.1 |  |
| HM | 56.37 | 54.24 | 64.85 | 68.16 | 68.61 | 60.68 | 71.75 | 67.56 | 61.85 | 61.47 | 72.88 | +18.6 |  |
| EuroSAT | Base | 56.48 | 92.19 | 87.49 | 94.07 | 86.63 | 87.39 | 92.90 | 86.26 | 93.01 | 91.30 | 93.27 | +1.08 |
| New | 64.05 | 54.74 | 60.04 | 73.23 | 68.97 | 67.63 | 73.90 | 71.38 | 54.89 | 68.53 | 79.00 | +24.2 |  |
| HM | 60.03 | 68.69 | 71.21 | 82.35 | 76.79 | 74.30 | 82.32 | 78.12 | 69.04 | 78.29 | 85.54 | +16.8 |  |
| UCF101 | Base | 70.53 | 84.69 | 82.33 | 83.00 | 83.67 | 72.71 | 87.10 | 82.00 | 82.67 | 84.13 | 86.33 | +1.64 |
| New | 77.50 | 56.05 | 73.45 | 78.66 | 75.43 | 41.51 | 78.80 | 78.06 | 74.54 | 77.90 | 78.87 | +22.8 |  |
| HM | 73.85 | 67.46 | 77.64 | 80.77 | 79.34 | 52.84 | 82.74 | 79.98 | 78.39 | 80.90 | 82.43 | +14.9 |  |

### 4.2 Comparison with other methods

Base-to-base/base-to-new generalization.
In this section, we compare the results of our approach over the ones that commonly use prompt learning or LoRA finetuning. As can be seen in Table 1, our approach obtains 84.16% , 76.55% and 80.02% Acc. for the averaged 11 datasets in terms of validation on base, new and HM.

**Table 2: Performance comparison on the domain generalization.**
|  | Source | Target |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | ImageNet | -V2 | -S | -A | -R |
| CLIP | 66.73 | 60.83 | 46.15 | 47.77 | 73.96 |
| $LoRA_{CLIP}$ | 69.70 | 62.67 | 38.70 | 39.67 | 69.93 |
| CoOp | 71.51 | 64.20 | 47.99 | 49.71 | 75.21 |
| CoCoOp | 71.02 | 64.07 | 48.75 | 50.63 | 76.18 |
| VPT | 70.72 | 58.22 | 44.67 | 43.00 | 71.86 |
| UPT | 72.63 | 64.35 | 48.66 | 50.66 | 76.24 |
| MaPLe | 70.72 | 64.07 | 49.15 | 50.90 | 76.98 |
| OrthSR | 70.83 | 63.8 | 49.3 | 51.37 | 77.4 |

More importantly, our method surpasses the comparative $LoRA_{CLIP}$ with 2.74%, 6.15% and 4.95% of base, novel and HM evaluation, which further demonstrates the OrthSR is capable of not only efficiently adapting to task-specific task but also leading to generalizability preservation, thanks to the norm-preserving property of orthogonal finetuning. And these results further presents the prevalent $LoRA$ method potentially tends to prioritize task-specific knowledge and results in task overfitting issues while ours has no such issues, especially for the few-shot image recognition task. Meanwhile, our approach reports consistent superorities beyond the conventional prompt learning methods, VPT and IVLP, better illustrate the effectiveness of our approach. When compared with competing MaPLe Khattak et al. () and PromptSRC Khattak et al. () which utilize complex strategies to enhance prompt tuning, our method still behaves better generalizability, obtaining highest accuracy on evaluation with 76.55% for new classes and 80.02% for HM.

Cross-dataset transfer. For evaluating the cross-dataset tranfer, we train our approach on ImageNet Russakovsky et al. () and then directly evaluate it on the other datasets without any domain-specific finetuning or adaptation. We compare cross-dataset performance with existing methods in Table 4. In comparison with CoOp Zhou et al. () and CoCoOp Zhou et al. (), our proposed OrthSR presents better generalization performance in 9/10 and 5/10 datasets, respectively. Importantly, our approach exceeds $LoRA_{CLIP}$ in 9/10 datasets and shows obvious advantages among these dataset, which further demonstrates that our methods retains stronger zero-shot generalizability. Meanwhile, compared with the prompt tuning methods MaPLe Khattak et al. () and PromptSRC Khattak et al. (), we obtain 7/10 and 6/10 better generalization performance while not introducing any tunable parameters after training (0 v.s. 3.55MB and 0 v.s 46KB, respectively) and no complicated training strategy tailored to struggle with the generalizability preservation.

Domain generalization. Table 3 reports the results of OrthSR and other methods on out-of-distribution datasets. Following the common methods, we train our model and directly evaluate on other datasets. We can observe that our method consistently surpasses $LoRA_{CLIP}$ on all datasets, while obtaining 3/4 superiority with CoOp and CoCoOp. Interestingly, prompt-based VPT illustrates inferior performance in 4/4 datasets to ours, while ours gains 2/4 better generlization evaluation beyond MaPLe Khattak et al. (). This suggests that our orthogonal tuning with simple yet effective cross-regularization enables the finetuned model favor better generalization for datasets with domain shifts.

### 4.3 Ablations and analysis

Orthogonal tuning choice of encoder. In Table 3, we conduct experiments to to showcase which encoder, i.e., image encoder or text encoder, should be introduced with the proposed orthogonal tuning. As can be observed that only utilizing single encoder of CLIP model presents lower performance on both base, novel and HM metrics while both encoders equipped with orthogonal finetuning obtain the best result, compared among row1/2/3.

Loss ablation. Compared among row 1/4/5 in Table 3, we found that removing logits distillation loss causes significant degradation on the Novel/New classes and HM metrics, which illustrates that there are some kind of deviation away from the pretrained model, proving that necessitates regularization to guide the finetuning. After using logits distillation, $\mathcal{L}_{kl}$, we get improved on both the Base and Novel classes, by 0.64% and 1.46%, respecitvely. Note that we derive such distillation guidance from the pretrained model only in a bypass manner, instead of seeking for extra data synthesis or heavy large-language model prior knowledge auxiliary.

Complexity analysis. Since our proposed orthogonal tuning method shares similar idea with $LoRA$ adapting VLMs into downstream scenarios via pretrained weights finetuning, it is necessary to demonstrate the computation cost during the training and inference phases. We therefore test and summarize the number of trainable parameters (#Params) and inference latency (#fps) in Table 5. We can see that though our approach needs the most number of trainable parameters since we leverage both two encoders to be injected with orthogonal tuning matrices for each fully-connected layer within Feed-Forward-Network, our approach needs the same inference latency with the baseline, CoOp, achieving the fastest 645 fps while having significantly better few-shot recognition and generalization performance. More ablative studies please refer to our Appendix A.

**Table 4: Performance comparison on the cross-dataset transfer setting.**
|  | Source | Target |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ImageNet | Caltech101 | Oxford_Pets | StanfordCars | Flowers102 | Food101 | FGVCAircraft | SUN397 | DTD | EuroSAT | UCF101 |
| $LoRA_{CLIP}$ | 69.70 | 91.70 | 89.13 | 59.53 | 68.77 | 82.13 | 23.80 | 65.03 | 44.83 | 45.53 | 65.83 |
| CoOp | 71.51 | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 |
| CoCoOp | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 |
| MaPLe | 70.72 | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 |
| PromptSRC | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 |
| OrthSR | 70.83 | 94.07 | 89.63 | 65.63 | 71.40 | 86.53 | 24.13 | 67.23 | 46.73 | 42.33 | 69.17 |

**Table 5: Complexity analysis over various methods. We report the number of trainable parameters (#Params) and frames per second (#fps).**
| Methods | CoOp | CoCoOp | VPT | PLOT | MAPLE | OrthSR |
| --- | --- | --- | --- | --- | --- | --- |
| #Params | 2,048 | 35,360 | 13,824 | 8,192 | 3,555,072 | 43450368 |
| #fps | 645 | 37 | 152 | 583 | 282 | 645 |

## 5 Conclusions

This paper proposes a novel and efficient method for adapting pretrained VLM weights, OrthSR, for specific downstream tasks (e.g., few-shot image recognition). To explore an effective fine-tuning approach not suffering from task overfitting issues under a data-efficient setting, we propose an orthogonal fine-tuning method for efficiently updating pretrained weights. Optimized by the constraint with Cayley parameterization during training, the fine-tuned CLIP model is capable of maintaining minimal and same-level of hyperspherical energy as the pretrained model owing to norm-preserving property, leading to better robustness and generalizability for task-specific scenarios. Meanwhile, a self-regularization strategy is designed to enforce the model not to deviate far away from the pretrained one within a bypass manner. Additionally, we first explore attentive CutOut data augmentation to enable the fine-tuned model to learn better task-specific knowledge on a small data set. Finally, extensive experiments demonstrate the training efficiency and generalizability preservation of our approach and showcase competitive performance on three generalization evaluations, shedding new light on the future works for this few-shot tuning task.

Limitations and future improvements. Despite the competitive generalization performance our approach obtains, there are still several limitations to be further delved into exploration. First, our method presents marginal advantages on cross-dataset transfer or domain generalization evaluations, although we exhibit competitive performance under base-to-base/base-to-new setting. Moreover, there are still future improvements on how to efficiently lower the tunable parameters during the training phase, and remaining an interesting direction on how to leverage theoretical analysis to decompose or disentangle the VLMs to seek out the potential manifold space that allows us to inject task-specific knowledge without sacrificing zero-shot generalizability.

## Appendix A Appendix / supplemental material

### A.1 More implementation details

Besides the implementation details in our main paper, we provide more details in Table 6.

**Table 6: Hyperparameter setting used in our experiments.**
| Hyperparameters | Values |
| --- | --- |
| Batch Size | 4 |
| Input Size | $224\times 224$ |
| Input Interpolation | ”Bicubic” |
| Input Pixel Mean | $[0.48145466,0.4578275,0.40821073]$ |
| Input Pixel STD | $[0.26862954,0.26130258,0.27577711]$ |
| Transforms | [”random resized crop”, ”random filp”, ”normalize”] |
| Optimizer | SGD |
| Learning Rate | 0.00001 |
| LR Scheduler | ”cosine” |
| Warmup Epoch | 1 |
| Warmup Type | ”constant” |
| Warmup LR | 1$e$-6 |
| Backbone | ViT-B/16 |
| Number of Textual Prompts | 4 |
| Number of Visual Prompts | 4 |
| Learnable Prompt Length | 2 |
| Fixed Prompt Length | 2 |
| weight of cross-entropy loss $\lambda_{1}$ | 1.5 |
| weight of Kullback-Leibler loss $\lambda_{2}$ | 1.2 |
| patch number for Cutout inference (ViT-B/16) | randomly sample one from $[5,6,7,8,9]$ |
| Prompt Initialization | ”a photo of a” |
| Precision | ”fp16” |

### A.2 Evaluation metrics

Among all our experiments, we report $top_{1}$ accuracy for each dataset. In base-to-base/base-to-new generalization, the $top_{1}$ accuracy is measured on base classes and new classes, respectively. We then calculate the harmonic mean (HM) between the base and new class accuracy to show the generalization trade-off Xian et al. (), using $HM=\frac{2\times base\times new}{base+new}$. In domain generalization, and cross-dataset transfer settings, we measure $top-1$ accuracy on the test set of each dataset with the same split provided by CoOp Zhou et al. () following other related works.

### A.3 More dataset descriptions

We throughly conduct our method on publicly available 15 image recognition datasets across 4 common generalizability evaluation settings: ImageNet Russakovsky et al. () and Caltech101 Fei-Fei () for generic objects classification, Oxford_Pets Parkhi et al. (), StanfordCars Krause et al. (), Flowers102 Nilsback & Zisserman (), Food101 Bossard et al. () and FGVCAircraft Maji et al. () for fine-grained classification, SUN397 Xiao et al. () for scene recognition, DTD Cimpoi et al. () for texture classification, EuroSAT Helber et al. () for satellite imagery recognition and UCF101 Soomro et al. () for action recognition; datasets with apparent domain shifts ImageNetV2 Recht et al. (), ImageNet-Sketch Wang et al. (), ImageNet-A Hendrycks et al. () and ImageNet-R Hendrycks et al. (). We make a summary in terms of data statistics in Table 7.

**Table 7: Summary of all 15 datasets. N/A denotes that we do not use the corresponding training or validation sets, which will be used to conduct generalizability evaluation only.**
| Dataset | Domains | #Classes | #Train | #Val | #Test |
| --- | --- | --- | --- | --- | --- |
| ImageNet | generic classification | 1000 | 1.28M | N/A | 50,000 |
| Caltech101 | generic classification | 100 | 4,128 | 1,649 | 2,465 |
| OxfordPets | fine-grained classification | 37 | 2,944 | 736 | 3,669 |
| StanfordCars | fine-grained classification | 196 | 6,509 | 1,635 | 8,041 |
| Flowers102 | fine-grained classification | 102 | 4,093 | 1,633 | 2,463 |
| Food101 | fine-grained classification | 101 | 50,500 | 20,200 | 30,300 |
| FDVCAircraft | fine-grained classification | 100 | 3,334 | 3,333 | 3,333 |
| SUN397 | scene recognition | 397 | 15,880 | 3,970 | 19,850 |
| UCF101 | action recognition | 101 | 7,639 | 1,808 | 3,783 |
| DTD | texture recognition | 47 | 2,820 | 1,128 | 1,692 |
| EuroSAT | satellite recognition | 10 | 13,500 | 5,400 | 8,100 |
| ImageNetV2 | generic classification | 1000 | N/A | N/A | 10,000 |
| ImageNet-Sketch | sketch classification | 1000 | N/A | N/A | 50,889 |
| ImageNet-A | generic classification | 200 | N/A | N/A | 7,500 |
| ImageNet-R | generic classification | 200 | N/A | N/A | 30,000 |

### A.4 Loss balancing hyper-parameters sensitivity ablations

In our main paper, the overall training loss $\mathcal{L}_{final}$ is:

$$ $\mathcal{L}_{\text{final}}=\lambda_{1}(\mathcal{L}_{ce}+\mathcal{L}_{cutout_ ce})+\lambda_{2}(\mathcal{L}_{kl}+\mathcal{L}_{cutout_kl})$ (12) $$

In this section, we conduct ablative studies on hyper-parameters, $\lambda_{1}$ and $\lambda_{2}$ in Fig 3. The figure shows that the overall training is robust to both the hyper-parameters, $\lambda_{1}$ and $\lambda_{2}$.

Figure: Figure 3: Ablations in terms of $\lambda_{1}$ and $\lambda_{2}$.
Refer to caption: x3.png

## Appendix B Theoretical Proof

Following previous works Chen et al. (); Ni et al. (),
this section provides detailed proofs for the Theorem in Sec. 3.6. Notably, we propose to utilize attentive CutOut data augmentation to implicitly increase the sample number and make use of pre-trained model as generalization  anchor to maintain the generalization error bound, which is different from  Chen et al. ().
We introduce the following lemmas for proving our Theorem.

Lemma 1(McDiarmid’s Inequality Vershynin ()). Consider independent random variables $v_{1},v_{2},\cdots,v_{n}\in\mathcal{V}$ and a function $\phi:\mathcal{V}^{n}\to\mathbb{R}$. Suppose that for all $v_{1},v_{2},\cdots,v_{n}$ and ${v_{i}}^{\prime}\in\mathcal{V}\left(i=1,2,\cdots,n\right)$, the function satisfies

$$ $\left|\phi\left(v_{1},\cdots,V_{i-1},V_{i},V_{i+1},\cdots,V_{n}\right)-\phi \left(v_{1},\cdots,V_{i-1},{v_{i}}^{\prime},V_{i+1},\cdots,V_{n}\right)\right| \leq c_{i},\vspace{0pt}$ (13) $$

and then it holds that

$$ $\mathcal{P}\left\{\phi\left(v_{1},v_{2},\cdots,v_{n}\right)-\mathbb{E}_{v_{1}, v_{2},\cdots,v_{n}}\left(\phi\left(v_{1},v_{2},\cdots,v_{n}\right)\right)>\mu \right\}\leq e^{-\frac{2\mu^{2}}{{\textstyle\sum_{i=1}^{n}c_{i}^{2}}}}.\vspace {0pt}$ (14) $$

The proof of Theorem 1. is given as follows.

Theorem 1. Assume that $\Theta^{*}$ is the solution to OrthSR. Then we have that for any $0<\varepsilon<1$ with probability $1-\varepsilon$,

$$ $\epsilon(\Theta^{*})-\bar{\epsilon}_{\chi}(\Theta^{*})\leq X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}+\frac{C^{\prime\prime}}{\lambda^{2\alpha}\sqrt{N}}.$ $$

where $\epsilon(\Theta^{*})$ is the true error. $\bar{\epsilon}_{\chi}(\Theta^{*})$ is the empirical error. $X^{*}$ is the upper bound of the loss function $L$. $N$ is the number of training samples. $\lambda$ is our introduced regularization parameter. $\alpha>0$. $\delta$ is a probability parameter. $C^{\prime\prime}$ encompasses constants from the Rademacher complexity bound.

###### Proof.

The generalization error is defined as:

$$ $\epsilon(\Theta)=\mathbb{E}_{(x,y)\sim D}\left[L(s_{\Theta}(x),y)\right]$ $$

where $\Theta$ represents the model parameters, $L(s_{\Theta}(x),y)$ is the loss function, and $D$ is the true data distribution.

The empirical error is:

$$ $\bar{\epsilon}_{\chi}(\Theta)=\frac{1}{N}\sum_{i=1}^{N}L(s_{\Theta}(x_{i}),y_{ i})$ $$

where $\chi=\{(x_{i},y_{i})\}_{i=1}^{N}$ is the training set, and $N$ is the sample size.

We use McDiarmid’s inequality to control the deviation between empirical error and true error. The inequality states:

$$ $P\left(f(X_{1},\ldots,X_{n})-\mathbb{E}[f(X_{1},\ldots,X_{n})]>t\right)\leq \exp\left(-\frac{2t^{2}}{\sum_{i=1}^{n}c_{i}^{2}}\right)$ $$

where $X_{1},X_{2},\ldots,X_{n}$ are independent random variables, and $f(X_{1},\ldots,X_{n})$ is a function of these variables. When one sample in the training set changes, the maximum change in the empirical error is:

$$ $\Delta=\bar{\epsilon}_{\chi}(\Theta)-\bar{\epsilon}_{\chi^{\prime}}(\Theta)$ $$

The change in empirical error is bounded by $\frac{c}{N}$, where $c$ is the upper bound on the difference in the loss function:

$$ $|L(s_{\Theta}(x),y)-L(s_{\Theta}(x^{\prime}),y^{\prime})|\leq c$ $$

Applying McDiarmid’s inequality with the bound $\frac{c}{N}$, we obtain the following bound:

$$ $P\left(\epsilon(\Theta)-\bar{\epsilon}_{\chi}(\Theta)>t\right)\leq\exp\left(- \frac{2Nt^{2}}{c^{2}}\right)$ $$

We introduce the Rademacher complexity $R_{N}(L)$, which measures the complexity of the model:

$$ $R_{N}(L)=\mathbb{E}_{\sigma,\chi}\left[\sup_{\Theta\in\mathcal{H}}\frac{1}{N} \sum_{i=1}^{N}\sigma_{i}L(s_{\Theta}(x_{i}),y_{i})\right]$ $$

The generalization error bound becomes:

$$ $\epsilon(\Theta)\leq\bar{\epsilon}_{\chi}(\Theta)+2R_{N}(L)+X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}$ $$

where: $\bar{\epsilon}_{\chi}(\Theta)$ is the empirical error. $2R_{N}(L)$ is the Rademacher complexity term. $X^{*}\sqrt{\frac{2\ln(1/\delta)}{N}}$ is the variance term that decreases as the sample size $N$ increases.
To further reduce the generalization error, we introduce the regularization term $L_{KD}$ (Knowledge Distillation Loss) in Eq. 10, which limits the complexity of the model. The objective function of our OrthSR is:

$$ $\min_{\Theta}\left(L_{CE}+\lambda L_{KD}\right)$ $$

where $L_{CE}$ is the cross-entropy loss for measuring the fit of the model. $L_{KD}$ is the knowledge distillation loss, reducing the difference between student and teacher models. $\lambda$ controls the trade-off between the two losses.
To understand why the Rademacher complexity $R_{N}(L)$ is reduced under the regularization term, we analyze how regularization influences the hypothesis space $\mathcal{H}$ and, consequently, the complexity of the loss function class.

The Rademacher complexity $R_{N}(L)$ measures the richness of the loss class $\mathcal{L}=\{L(s_{\Theta}(x),y):\Theta\in\mathcal{H}\}$ by evaluating how well it can fit random noise. It is defined as:

$$ $R_{N}(L)=\mathbb{E}_{\sigma,\chi}\left[\sup_{\Theta\in\mathcal{H}}\frac{1}{N} \sum_{i=1}^{N}\sigma_{i}L(s_{\Theta}(x_{i}),y_{i})\right],$ $$

where $\sigma_{i}$ are independent Rademacher variables taking values $\pm 1$ with equal probability.

Regularization introduces a penalty term $\lambda L_{KD}$ in the objective function:

$$ $\min_{\Theta}\left(L_{CE}+\lambda L_{KD}\right).$ $$

This penalty discourages complex models by imposing a cost on large parameter values or deviations from the teacher model in knowledge distillation. As a result, the effective hypothesis space $\mathcal{H}_{\lambda}$ becomes smaller or more restricted because models with high complexity are penalized.

Mathematically, stronger regularization (larger $\lambda$) enforces tighter constraints on $\Theta$, effectively reducing the norm or other measures of complexity of the model parameters. We assume that through regularization, the model parameters satisfy the following constraint:

$$ $\|\Theta\|\leq\frac{C}{\lambda^{\beta}},$ $$

where $C$ and $\beta>0$ are constants.

Under this constraint, and assuming that the loss function $L$ is Lipschitz continuous with Lipschitz constant $L_{0}$, the Rademacher complexity can be bounded as:

$$ $R_{N}(L)\leq\frac{L_{0}C^{\prime}}{\lambda^{\beta}\sqrt{N}},$ $$

where $C^{\prime}$ is another constant.

Substituting this bound into the generalization error bound, we have:

$$ $\epsilon(\Theta^{*})-\bar{\epsilon}_{\chi}(\Theta^{*})\leq X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}+\frac{1}{\lambda^{\alpha}}\cdot R_{N}(L)\leq X^{*}\sqrt{ \frac{2\ln(1/\delta)}{N}}+\frac{L_{0}C^{\prime}}{\lambda^{\alpha+\beta}\sqrt{N }}.$ $$

To ensure consistency in the exponents of $\lambda$, we set:

$$ $\alpha=\beta>0.$ $$

Therefore, the generalization error bound becomes:

$$ $\epsilon(\Theta^{*})-\bar{\epsilon}_{\chi}(\Theta^{*})\leq X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}+\frac{C^{\prime\prime}}{\lambda^{2\alpha}\sqrt{N}},$ $$

where $C^{\prime\prime}=L_{0}C^{\prime}$ is a constant.

This inequality shows that $R_{N}(L)$ decreases as $\lambda$ increases, since $\alpha>0$. By reducing $R_{N}(L)$ through regularization, we tighten the generalization error bound:

$$ $\epsilon(\Theta^{*})-\bar{\epsilon}_{\chi}(\Theta^{*})\leq X^{*}\sqrt{\frac{2 \ln(1/\delta)}{N}}+\frac{C^{\prime\prime}}{\lambda^{2\alpha}\sqrt{N}}.$ $$

In summary, the regularization term reduces the Rademacher complexity $R_{N}(L)$ by limiting the capacity of the hypothesis space $\mathcal{H}$. This reduction leads to better generalization performance by preventing overfitting and tightening the generalization error bound.

∎

###### Proof.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Figure 3, I don't observe any significant differences across the various hyperparameter values. Does this suggest that setting $\lambda_1$ or $\lambda_2$ has only a marginal impact on performance, potentially indicating that certain components of the loss function may be less useful?
2. How does the proposed method perform on the few-shot learning task?
soundness: 3
strengths: 1. The idea of introducing orthogonal constraints during training seems interesting, which may provide useful insights for future research.
2. The experimental results are comprehensive and effectively validate the superiority of the proposed methods across various datasets.
3. The authors provide theoretical support to show the effectiveness of the proposed method (Of note: I did not carefully check the proofs).
summary: This paper presents OrthSR, featuring a novel and efficient orthogonal fine-tuning method to adapt vision-language models (VLMs) to downstream tasks, along with a self-regularization technique to preserve generalizability. Specifically, the authors incorporate trainable orthogonal matrices within the transformer architecture and introduce an attentive CutOut data augmentation strategy to enhance data diversity. Comprehensive experimental results across various datasets demonstrate the effectiveness of the proposed approach.
weaknesses: 1. The proposed method introduces a significantly larger number of trainable parameters (10x–1000x more) compared to other methods, yet achieves only limited performance improvements, raising questions about its overall cost-effectiveness. Additionally, could you report other efficiency metrics, such as training time and FLOPs, for the proposed method compared to other baselines?
2. The performance improvements are marginal. For example, in the base-to-new generalization evaluation on certain datasets, the proposed approach underperforms compared to PromptSRC. The performance gains on two other tasks are also limited.
3. The motivation for introducing the attentive CutOut is unclear, and its connection to other components of the proposed method is not well explained. 
4. A more detailed ablation study would be helpful. For example, how does the performance change when applying only the attentive CutOut?
5. Given the complexity of the proposed method, its reproducibility is limited. It is recommended that the authors provide the code implementation for this work.

Minor issues:
1. Line 31, classficiation -> classification.
2. Lines 303-306, kd -> kl?
3. Some of the notations in the methods section could be clarified. For instance, what is $f_t(:w)$ in Equation (1)? Additionally, could you explain what $k$ represents in $c_k$?
4. Line 279, the authors mention that “We empirically observe that this approximation results in instability of the fine-tuning”. Are there some related experiments?
5.Could you please clarify what the last row in Table 1 represents? Which baseline are you using for the performance gain comparison?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In Section 4.2, the paper claims that the proposed method surpasses the comparative $LORA_{CLIP}$, but I couldn’t find $LORA_{CLIP}$ in the baseline methods in Table 1. I also didn’t see any introduction for $LORA_{CLIP}$ in the baseline introduction, and could only find it in Table 2 and 4. Did the paper use a different name for it, or is there something that I am missing?

2. For efficient fine-tuning of CLIP, an impactful line of work includes the adapter tuning methods, such as CLIP-Adapter [1] and Tip-Adapter [2], which were not chosen as the comparison baseline. Why did the paper decide to leave these out?

3. In the ablation analysis, it seems that the model performance drops significantly when the cutout augmentation is removed. However, it seems that cutout could also be applied on other baselines, but was not used. Does this lead to unfair comparison?

4. The paper seems to be in improper format and is not using the ICLR template.

References:

[1] Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., ... & Qiao, Y. (2024). Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2), 581-595.

[2] Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., ... & Li, H. (2021). Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930.
soundness: 3
strengths: 1. Comprehensive Experiments:

The paper provides a comprehensive comparison of their method against a number of competitive prior works on 11 diverse datasets. The empirical results show that OrthSR achieves comparable results with best previous methods. The experimental design, following standard protocols in the field, is rigorous and well-justified. 
 
2. Sound Theoretical Proof:

The paper offers sound theoretical analysis showing that the proposed objective function achieves a low expected generalization error with sufficient data samples and a good choice of hyper-parameter \lambda, adding to the method’s credibility.
summary: The paper proposes OrthSR, a novel fine-tuning method inspired by orthogonal learning, to improve CLIP’s generalization capability on downstream tasks. Specifically, the method consists of three aspects: 1) Injecting orthogonal matrices into both visual and textual encoders while keeping the pre-trained weights frozen to adapt the model to specific tasks; 2) Keeping the zero-shot knowledge in a skip-connection manner to prevent the model from deviating too much on the downstream datasets, and 3) Adopting attentive CutOut augmentation to enrich data diversity. Theoretical analysis gives a bound on the generalization error on the overall training objective, and comprehensive experimental results show slight improvement over previous methods in base-to-base/base-to-new generalization, cross-dataset transfer and domain generalization tasks.
weaknesses: 1. Unclear Motivation:

While the paper argues that its aim is to enhance the robustness of models like CLIP, it is never clearly demonstrated how robustness is measured in the scope of the discussion, and why OrthSR is capable of improving over previous methods. Adding orthogonal matrices is an interesting attempt, but there is limited discussion on its similarities and differences compared to LoRA, a main target of comparison in the paper that also tries to fine-tune the pre-trained weights. Moreover, the section on cutout augmentation seems to be somewhat unrelated to the formulation of the proposed approach. 

2. Limited Improvements:

While the proposed approach manages to achieve good results over a wide range of datasets and tasks, the improvements don’t seem significant enough. For the base-to-base/base-to-new results in Table 1, the gain in performances is compared to CoOp but not the second-best method PromptSRC (which can be misleading). On average, OrthSR only outperforms PromptSRC by 0.05% in terms of harmonic mean. For other tasks, the improvement in performance seems even slighter, which heavily weakens the contribution of the paper.

3. Excessive Training Cost:

While the paper claims at first hand that part of the focus is on efficient fine-tuning, the training cost inevitably increases when both branches of the CLIP encoder are involved, as is also shown in Table 5. Even though the counterpoint is that OrthSR achieves the highest inference speed, the number of training parameters is more than 10 times over the next most costly method, and more than 1000 times over baseline CoOp. 

4. Poor Writing:

At times, the paper contains over-convoluted sentences with grammatical and even spelling mistakes, which makes it difficult for readers to ignore.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In general, the main contributions are inspired from Ref [65] [54], i.e., T2I diffusion models. What is the difference between the existing methods and the orthogonal fine-tuning proposed in this paper. It seems that these works are all focused on modifying the FFN layer of the Transformer.

2. The motivation of this paper does not convince me, for example, the authors claim that they aim to make fine-tuning process more stable and efficient using orthogonal fine-tuning. But as shown in Table 5, the complexity of training does not have an advantage. 

3. As for experiments, the proposed method does not have competitiveness compared to recent methods, for example, compared to PromptKD and CasPL, this method is less than 2 percentage points over base2new and domain generalization tasks. 

PromptKD:PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024)

CasPL: Cascade Prompt Learning for Vision-Language Model Adaptation (ECCV 2024)

4. What is the direct connection between cutout and orthogonal fine-tuning methods, and why orthogonal fine-tuning requires data augmentations should be clarified.
soundness: 2
strengths: 1.	This poaper explores the role of orthogonal fine-tuning in downstream classification tasks.
2.	The experimental verification is complete, including tasks such as base2new and domain generalization.
summary: This paper introduces an efficient orthogonal fine-tuning method to adapt the VLMs into task-specific knowledge while maintaining strong generalizability. But there are some concerns about the motivation and the experimental results.
weaknesses: 1.	The experimental performance is not competitive compared to recently published methods.
2.	The motivation for using orthogonal fine-tuning is not clear.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why was orthogonal matrix fine-tuning chosen, and why is it applied solely to the Feed-Forward Network (FFN) layers? Were other weights considered for fine-tuning?
2. Could this approach be combined with Prompt Tuning methods, and if so, might there be potential conflicts?
3. Why cutout can enforce the fine-tuned model that pays more attention to other less-discriminative image regions? It seems that this is the test data augmentation.
soundness: 3
strengths: 1. Proposes the use of orthogonal matrix fine-tuning to enhance VLM robustness.
2. Provides theoretical support for the effectiveness of orthogonal matrix fine-tuning.
summary: This paper introduces a method for fine-tuning Vision-Language Models (VLMs) termed OrthSR, which injects orthogonal matrices into the backbone to enhance the model's generalization capability. To prevent catastrophic forgetting, a regularization loss is incorporated. While the paper claims that the inference phase incurs no additional time cost and that the approach converges relatively quickly, the novelty of the proposed method appears limited. The approach resembles a combination of existing techniques, with regularization methods similar to those introduced in prior work over two years ago. Moreover, the performance improvements presented are marginal.
weaknesses: 1. The method lacks originality, as regularization strategies similar to this have been previously explored in papers such as KgCoOp [1] and PromptSRC [2].
2. The empirical performance gains are minimal; compared to PromptSRC, this method only achieves a 0.05 improvement.
3. Lacks of comparisons with full fine-tuning methods, CLIP-CITE [3], CLIP-ood [4].

[1] Self-regulating prompts: Foundational model adaptation without forgetting. ICCV 2023 (This also is incorrectly cited as CVPR 2023 in 
the manuscript.)

[2] Visual-Language Prompt Tuning with Knowledge-guided Context Optimization. CVPR, 2023.

[3] Fully Fine-tuned CLIP Models are Efficient Few-Shot Learners. arXiv, 2024.

[4] CLIPood: Generalizing CLIP to Out-of-Distributions. ICML, 2023.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My question concerns the motivation and novelty. What is the motivation for using orthogonal fine-tuning to enhance the generalization of VLMs? It seems that if we replace orthogonal fine-tuning with another PEFT method and keep self-regularization, the framework would still work. Additionally, compared to existing prompt-tuning methods, the proposed OrthSR method does not show significant advantages on several downstream tasks.
soundness: 3
strengths: Efficient fine-tuning of vision-language models (VLMs) is an important task. This paper attempts to propose a new fine-tuning method to address this task. The presentation of this paper is easy to follow, with clear explanations in the equations and illustrations in the figures.
summary: This paper introduces a fine-tuning method called Orthogonality Learning with Self-Regularization (OrthSR) for adapting CLIP to task-specific scenarios while preserving generalizability. The main method consists of two operations: (1) injecting orthogonal matrices into both the image encoder and text encoder, and (2) utilizing CutOut to augment images. The final training loss combines two image classification losses (one from the original classification loss and another from the CutOut images) and two distillation losses (both are logit-level regularizations). Experiments on the proposed method show improvements on several downstream image classification tasks.
weaknesses: In the methodology section, the main idea lacks novelty. By comparing Eq. 2-3 in this paper with Eq. 1-2 in paper [1], they are almost the
same. It appears to be only a naive extension of the orthogonal fine-tuning idea from text-to-image model generation to image recognition. What unique advantage does the orthogonal fine-tuning method offer compared to existing prompt-tuning methods?

The self-regularization basically constrains the fine-tuned model weights from deviating from the pretrained model weights, which has already been investigated in paper [2] and paper [3]. Additionally, the CutOut augmentation is a straightforward technique to enhance the diversity of image inputs, with similar ideas appearing in paper [4].

The improvement in experimental results is quite marginal. From Table 1, we see that, compared to the current baseline method, PromptSRC, the base category accuracy of the proposed method actually drops, and the new category accuracy (76.55 vs. 76.10) and HM category accuracy (80.02 vs. 79.97) show less than a 1% improvement. The performance gains marked in the table are misleading, as the comparison with CoOp is not fair. Papers [5] and [6] represent the state-of-the-art in prompt tuning, yet the author did not include them as baselines. If these two papers were included, the author’s results would be even less competitive.

Furthermore, from Table 5, the efficiency comparison shows that the proposed OrthSR method has significantly more parameters than CoOp but the same FPS. This table does not show any advantage of the OrthSR method. Additionally, Table 5 lacks an efficiency comparison with PromptSRC.

[1]Controlling Text-to-Image Diffusion by Orthogonal Finetuning.
[2] CLIPood: Generalizing CLIP to Out-of-Distributions
[3]Self-regulating Prompts: Foundational Model Adaptation without Forgetting
[4] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
[5] Dept: Decoupled prompt tuning, CVPR2024
[6] ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models, CVPR2024