Directional Gradient Projection for Robust Fine-Tuning of Foundation Models

Authors: Chengyue Huang, Junjiao Tian, Brisa Maneechotesuwan, Shivang Chopra, Zsolt Kira
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Fine-tuning, transfer learning, foundation models, robustness, visual question answering

Abstract:
Robust fine-tuning aims to adapt large foundation models to downstream tasks while preserving their robustness to distribution shifts. Existing methods primarily focus on constraining and projecting current model towards the pre-trained initialization based on the magnitudes between fine-tuned and pre-trained weights, which often require extensive hyper-parameter tuning and can sometimes result in underfitting. In this work, we propose $\textbf{Di}$rectional $\textbf{Gra}$dient $\textbf{P}$rojection (DiGraP), a novel layer-wise trainable method that incorporates directional information from gradients to bridge regularization and multi-objective optimization. Besides demonstrating our method on image classification, as another contribution we generalize this area to the multi-modal evaluation settings for robust fine-tuning. Specifically, we first bridge the uni-modal and multi-modal gap by performing analysis on Image Classification reformulated Visual Question Answering (VQA) benchmarks and further categorize ten out-of-distribution (OOD) VQA datasets by distribution shift types and degree (i.e. near versus far OOD). Experimental results show that DiGraP consistently outperforms existing baselines across Image Classfication and VQA tasks with discriminative and generative backbones, improving both in-distribution (ID) generalization and OOD robustness.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730650643155

thread_type:
official_review

<thread_object>
creator: reviewer_d9ep

<content>
summary:
The paper presents the Directional Gradient Projection (DiGraP) method, designed for robust fine-tuning of foundation models, particularly in the face of distribution shifts. DiGraP introduces a novel approach that leverages gradient directionality to resolve conflicts between optimization objectives, aiming to enhance model robustness across in-distribution (ID) and out-of-distribution (OOD) data. Unlike traditional approaches that primarily rely on weight projection and regularization, DiGraP utilizes directional information for gradient adjustments, bridging multi-objective optimization and regularization. The method’s effectiveness is demonstrated across image classification and multi-modal tasks (Visual Question Answering - VQA), outperforming baseline methods in both ID and OOD performance.

strengths:
DiGraP’s gradient-based approach is innovative, incorporating directional information to handle conflicting objectives in a way that traditional regularization does not.

The expansion from uni-modal to multi-modal evaluation settings represents the contribution of robust fine-tuning.

Comprehensive ablation studies and sensitivity analyses of the hyper-parameters (e.g., projection strength) strengthen the validity of the findings, showing DiGraP’s robustness and effectiveness across different configurations.

The paper clearly explains the design choices of DiGraP, with detailed descriptions of its components.

Visualizations and tables effectively illustrate the experimental results and the model’s behavior under varying projection strengths.

weaknesses:
There are many public unimodal and multimodal foundation models, e.g., MAE, CLIP, BEiT3, LLaVa, etc. It is unclear why ResNet50 and PaliGemma are selected as foundation models. The ResNet50 pretrained in a supervised manner on ImageNet can hardly be deemed as foundation models. The PaliGemma is pretrained on a broad mixture of large-scale vision-language tasks. Whether the conclusion of this paper holds across other, more general foundation models, e.g., CLIP or LLaVa, is questionable.

Although DiGraP demonstrates improved performance in near OOD settings, its performance on far OOD tasks remains limited compared to other methods. The authors acknowledge this trade-off between ID/near OOD and far OOD robustness, but a deeper investigation into addressing this limitation would enhance the model’s versatility.

The model’s gradient projection mechanism, while theoretically sound, lacks interpretability in how projection strength decisions impact specific instances.

The figures in Appendix have too small font.

questions:
Could additional techniques be incorporated to enhance DiGraP’s performance on far OOD datasets without sacrificing ID and near OOD performance?

Could the authors provide more insights into how the projection strength parameter adapts dynamically across different layers and tasks, especially in multi-modal settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657668517

thread_type:
official_review

<thread_object>
creator: reviewer_1rwn

<content>
summary:
This paper proposes a novel layer-wise trainable method for robust fine-tuning of foundation models, namely DiGraP, which considers the directional information of the gradients. In addition, the authors generalize the method to multi-modal evaluation settings, filling the gap in experiment. Finally, experiments on several tasks demonstrate the effectiveness and robustness of the proposed method.

strengths:
+This work adopting multi-objective learning to alleviate the previous method's sensitivity to hyper-parameters and underfitting problems, which seems to a relatively novel attempt in the current field.

+The work expands the experimental setting from single-modal to multi-modal, filling the gap in experiments.

weaknesses:
-Although the work attempts to utilize gradient direction information, there are still deficiencies in the innovation of the method. It looks like a combination of previous work, so the innovation needs to be explained more.

-Sec 4.1 lacks qualitative analysis of the experimental results, especially the reasons why the ID performance on the real domain is worse than LP-FT.

-The article seems to have only conducted a quantitative analysis of hyper-parameters for multi-modal tasks, and corresponding ablation experiments are still needed on singal-modal tasks.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731176600298

thread_type:
official_review

<thread_object>
creator: reviewer_fvuq

<content>
summary:
The paper proposes a new method for robust fine-tuning of models for robust fine-tuning that maintains model performance on out-of-distribution datasets while fine-tuning on in-domain data. The authors frame robust finetuning as a multi-objective optimization problem where the two objectives are the original loss function and the distance between the fine-tuned and pretrained weights. They propose an algorithm which projects the gradient of the original loss onto the normal plane of the regularization term’s gradient when the two gradients do not conflict. This approach is evaluated on image classification and VQA tasks, demonstrating improved performance on both in-domain and OOD datasets.

strengths:
1. The proposed approach makes the projection strength $\omega$ as a learnable parameter, and makes it tunable by a learning rate $\mu$. This makes the approach less sensitive to hyperparameters. The authors show good experimental evidence for the claim.
2. The proposed evaluation of reformulating image classification as a VQA task allows for evaluations on foundation VLMs like PaliGemma.
3. The authors show that the proposed approach can be adapted to PEFT approaches like LoRA.

weaknesses:
1. While the paper addresses the problem of robust fine-tuning, the proposed approach Directional Gradient Projection appears similar to PCGrad. The problem of conflicting gradients and the  idea of projecting the gradient to the normal plane have been addressed in PCGrad. I would encourage the authors to highlight the unique contributions more clearly.
2. The paper proposes a general approach for robust fine-tuning of foundation models, but the paper focuses on Image classification and VQA tasks. It would be useful to evaluate the approach by fine-tuning open-source LLMs like LLaMA on a multi-task benchmark like MMLU.

questions:
1. In Table 1, while DiGraP performs well on OOD domains, the performance on the ID domain is poor compared to other approaches like LP-FT and TPGM. By changing the strength of ‘w’, does the approach allow a trade-off to improve performance on ID while compromising OOD performance?
2. In Table 4(b), when the learning rate is increased, which makes the projection strength larger, why does the ID performance improve?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732417902236

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers for their positive feedback on this work's adaptability (fvUQ), performance (fvUQ, D9Ep), and insights (fvUQ, 1rWn, D9Ep). We aim to address your questions with concrete responses and clarify any points of confusion.

The **Additional Experiments for Rebuttal in Appendix (19-20 pages in main PDF)** includes new experiments and studies requested by the reviewers. The new experiments are summarized below:

- **(D9Ep)** We add experiments of fine-tuning on DomainNet with CLIP ViT-Base, DomainNet-oVQA, and VQA with LLaVA-7B. DiGraP consistently achieves the best performance across all experiments.
- **(fvUQ, 1rWn)** We conduct an LP-FT-C experiment (a controlled version of LP-FT) by increasing the regularization strength to ensure that the ID performance matches that of DiGraP. Despite this adjustment, DiGraP still outperforms LP-FT-C on OOD datasets.
- **(1rWn)** We include the hyper-parameter tuning experiment for DomainNet on CLIP ViT-Base. The results demonstrate that DiGraP remains robust to hyperparameter variations on both ID and OOD datasets in the uni-modal image classification task.
- **(D9Ep)** We present a visualization of the variation in regularization strength (λ) across different layers over epochs.

We hope our response addresses your questions and welcome further discussion during the rebuttal period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Directional Gradient Projection (DiGraP), a robust fine-tuning approach that utilizes gradient-based optimization to address conflicting objectives during the fine-tuning of foundation models. The reviewers collectively acknowledged the paper’s strengths and its potential contributions to the field, though they also noted certain weaknesses.

All reviewers commended the novelty of introducing a learnable projection strength parameter, which effectively reduces sensitivity to hyperparameters. Additionally, the expansion of evaluation settings from uni-modal to multi-modal tasks was widely appreciated as a significant step.

On the downside, concerns were raised regarding the novelty of DiGraP against PCGrad, the choice of models such as ResNet50, and the limited performance on far out-of-distribution (OOD) tasks. The authors addressed these concerns in their rebuttal. Reviewer fvUQ, who engaged in the discussion phase, appreciated the authors’ responses and expressed a positive inclination toward accepting the paper. Unfortunately, the other two reviewers did not participate in further discussions post-rebuttal.

The AC panel concurs with the positive sentiment of the reviewers. Despite some limitations, the proposed method is technically sound, well-articulated, and supported by promising experimental results. The strengths of the work, including its theoretical contributions and practical utility, outweigh the identified weaknesses.

The AC panel, therefore, recommends accepting the paper.