Directional Gradient Projection for Robust Fine-Tuning of Foundation Models
Authors: Chengyue Huang
Junjiao Tian
Brisa Maneechotesuwan
Shivang Chopra
Zsolt Kira
Venue: ICLR 2025 Poster (2025)
Keywords: Fine-tuning
transfer learning
foundation models
robustness
visual question answering
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We present a novel regularization technique for robust fine-tuning of large foundation models.

Abstract: Robust fine-tuning aims to adapt large foundation models to downstream tasks while preserving their robustness to distribution shifts. Existing methods primarily focus on constraining and projecting current model towards the pre-trained initialization based on the magnitudes between fine-tuned and pre-trained weights, which often require extensive hyper-parameter tuning and can sometimes result in underfitting. In this work, we propose $\textbf{Di}$rectional $\textbf{Gra}$dient $\textbf{P}$rojection (DiGraP), a novel layer-wise trainable method that incorporates directional information from gradients to bridge regularization and multi-objective optimization. Besides demonstrating our method on image classification, as another contribution we generalize this area to the multi-modal evaluation settings for robust fine-tuning. Specifically, we first bridge the uni-modal and multi-modal gap by performing analysis on Image Classification reformulated Visual Question Answering (VQA) benchmarks and further categorize ten out-of-distribution (OOD) VQA datasets by distribution shift types and degree (i.e. near versus far OOD). Experimental results show that DiGraP consistently outperforms existing baselines across Image Classfication and VQA tasks with discriminative and generative backbones, improving both in-distribution (ID) generalization and OOD robustness.

Published as a conference paper at ICLR 2025
DIRECTIONAL GRADIENT PROJECTION FOR ROBUST TUNNIG OF FOUNDATION MODELS
Chengyue Huang, Junjiao Tian, Brisa Manerchotesuwan, Shivang Chopra, Zsolt Kira
Georgia Institute of Technology
(chuang475, jt an73, bmaneech3, shivangchoprall zkira (aaiiech edu
ABSTRACT
Robust fine-tuning aims to adapt large foundation models to downstream tasks while preserving their robustness to distribution shifts. Existing methods primarily focus on constraining and projecting current model towards the pre-trained ini- tialization based on the magnitudes between fine-tuned and pre-trained weights, which often require extensive hyper-parameter tuning and can sometimes result underfitting. In this work, we propose Directional Gradient Projection (DirraP), a novel layer-wise trainable method that incorporates directional information from gradients to bridge regularization and multi-objective optimization. Besides demon- strating our method on image classification, as another contribution we generalize this area to the multi-modal evaluation settings for robust fine-tuning, Specifically, we first bridge the uni-modal and multi-modal gap by performing analysis on Image Classification reformulated Visual Question Answering (VQA) benchmarks and further categorize ten out-of-distribution (OOD) VQA datasets by distribution shift types and degree (i.e. near versus far OOD). Experimental results show that DiGraP consistently outperforms existing baselines across Image Classfication and VQA tasks with discriminative and generative backbones, improving both in-distribution (ID) generalization and OOD robustness
INTRODUCTION
Robust fine-tuning has become an essential technique in adapting pre-trained models to downstream tasks, particularly in the face of distribution shifts that challenge model generalization. While pre- trained models excel in capturing wide range of features from diverse datasets, fine-tuning them on specific tasks often leads to overfitting, reducing their robustness to duttoo-distiibuttin (OOD) data (Wortsman et al.] 2022) Nguyen et al., 2024). The goal of robust fine-tuning is to strike a balance between task-specific performance and maintaining the generalization abilities of the pre-trained model (Wortsman et al. (022).. This is particularly crucial for real-world applications such as visual question answering (VQA), where models frequently exposed to varying distributions in images, questions, and answers (Agrawal et al. 2018) Shah et al. (019).. Effective robust tinettunig strategies aim mitigate performance degradation by incorporating techniques like regularization (Li et al.) 2018) and bi-level optimization (Tian et al. 2023a b 2024). ensuring that models retain their learned knowledge while adapting new domains.
To tailor the model for downstream tasks while retaining the capabilities of the pre-trained model (c.g. robustness to distribution shifts), L2-SP (Li al. 2018) imposes a regularization term on the distance between the fine-tuned and pre-trained weights. More recently, instead of viewing robust finettunig as a regularization problem, TPGM (Tian et al., 2023a) and FTP (Tian al. 2023b) consider the regularization term as the constraint to reformulate the problem from a bi-level optimization prospective and propose to learn different hard constraints for each layer. However, these methods are computationally heavy and often apply overly strong constraints which results in underfitting (See Sec. 4.2 and Sec. 4.3p. Morcover, these methods perform weight projection to enforce the distance between fine-tuned and pre-trained weights within set of projection radii, which magnitude- wise but does not encode any directional information. This motivates us to think of direction- based methods for this fundamental problem.
'hee code is available at http://////ggggiio//////eaaaaana

---

Published as a conference paper at ICLR 2025
We re-examine the regularization problem from multi- multi-objective optimization perspective and propose Directional Gradient Projection (DiGraP). We consider the regularization term as the second objective besides the first objective, i.c., original loss function, and the goal is to minimize the two simultaneously. This new method involves two aspects: the projecting condition and the directional projection strength. If the projecting condition meets, i.e., the gradient directions of the two objectives are opposite, the decrease of one objective will lead to the increase of the other objective. In this case, we project the opposite gradient to the orthogonal direction of the other to get rid of the conflicting directional information. We also add a directional projection strength w [0, 1] to learn the priority of the two objectives and make it trainable so that can be dynamic throughout the training process.
Regularization (L2-SP): min C()) [() 111 0o11
Bi-level optimization (TPGM, FTP): min C(0) s.t. ||0 0o11 5
Multi-objective optimization (DiGraP): min ((0), 3 10 00
Typical evaluations on robust fine-tuning methods are primarily done in uni uni-modal settings, e.g., image classification or semantic segmentation. However, they have not been analyzed in the context of multiple modalities. We first bridge the gap by comparing our method with baselines on an Image Classification reformulated VQA benchmark. We further propose a new setting for evaluating robust fine-tuning of VQA by leveraging ten VQA datasets and categorizing them into in-distribution (ID), near and far OOD datasets covering uni-modal, multi-mode and adversarial distribution shifts. DiGraP achieves SOTA ID and OOD performance on both image classification and VQA benchmarks. Our contributions are:
We propose layer-wise trainable directional gradient projection method DiGraP for robust fine-tuning of large foundation models with the intuition of bridging regularization and multi-objeccive optimization. This is the first robust fine-tuning methods that considers the directional information of the gradients.
We propose new settings for evaluating robust fine-tuning of VQA. We first conduct ex- periments on Image Classification reformulated VQA benchmarks. We then categorize the existing OOD VQA datasets into different types and degrees of distribution shifts and present a consistent comparative analysis of robust fine-tuning algorithms.
We show that DiGraP consistently outperforms other baselines across uni-modal and multi- modal tasks in both II) generelization and OOD robustness.
RELATED WORKS
Robust Fine-Tume Tane-Tuning of Foundation Models. LP-FT (Kumar et al. 2022) proposes a two-step
strategy of linear probing then full fine-tuning to prevent the feature distortion of the pre-trained layers. WiSE-FT (Wortsman et al., 2022) interpolates the pre-trained and fine-tuned weights to
combine the strengths of the two embedding space. 1.2-SP (Li et al., 2018) explicitly adds I.2 norm penalty on the deviation between the fine-tuned and pre-traimed weights. MARS-SP (Gouk al. 2021) further studies different forms of norms as the penalty. More recently, TPGM (Tian et al. 2023a) approaches the regularization term as constraint, reformulating the problem as bi-level optimization and proposing to learn distinct hard constraints for each layer. FTP (Tian et al. 20236) further improves the efficiency of TPGM (Tian et al. 2023a) by learning the constraint from training set of previous step instead of the current validation set. We argue that these methods are still computationally heavy and requires lots of tuning, whereas DiGraP reformulates the problems as multi-objective optimization, injects directional information and more intuitive to tune.
OOD Robustness in VQA. Previous works have proposed various settings for evaluating robust VQA models. Agrawal et al. 2023) conducts cross-dataset evaluations with four VQA datasets, while Ma et al. (024)) and Li et al. 22021) provides more comprehensive and detailed robustness analysis by incorporating VQAv2 variants and categorizing different types of distribution shifts. We build on these efforts by introducing further granularity with near and far OOD distinctions and measuring the distance between distributions. More importantly, while previous work has primarily focused on testing different backbone models, they have not yet compared different robust fine-tuning methods.
2

---

Published as a conference paper at ICLR 2025
3 DIRECTIONAL GRADIENT PROJECTION FOR ROBUST FINE-TUNING
In this section, we first describe the intuition and the mathematical motivation behind DiGraP. Then, we provide our method's concrete algorithmic design.
Gradients are conflicting: À
9192 [
Gradients are non-conflicting: 0
D1
91 = V (0)
19
91
91 Wy1
92
Ag
82 0g
Figure 1: Directional Gradient Projection. (Left) When the gradients are non-conflicting, we do not perform projection. (Right) When the gradients conflicting, project the gradient for the original loss function according to the gradient for the regularization term. We add w E [0, 1] to control the projection strength, where = 0 is an unconstrained update and w represents full projection to the orthogonal direction of the gradient for the regularization term.
3.1 ROBUST FINE-TUNING AS MULTI-OBJECTIVE OPTIMIZATION PROBLEM
In order adapt the model the downstream tasks but preserve the power of the pre-trained model (e.g. robustness to distribution shifts), methods such as 1.2-SP (Li et al. 2018) impose regularization term on the distance between the fine-tuned and pre- pre-trained weights. Formally,
=())= [(0) + 2 0 00112
(1)
where 9 denotes the fine-tuned weights, 6g the pre-trained weights, ((0) the original loss function, and À the hyper-paramett for regularization strength, i.e., weight decay. In this case, ||e 0u l| serves constraint that the updated model will not deviate from the initialization we can maintain some strengths from the pre-trained model. However, 1.2-SP is not intuitive to tune which often spans over a wide range: a small À may achieve better in-distribution performance but leads to poor OOD robustness, while large À results in underfitting, Besides, L2-SP is also harder to if applied differently across layers.
much, thus
In this work, we instead propose to view robust fine-tuning from multi-objectiiv optimization perspective, leading more explicit method to balance this trade-off. Specifically, there are two objectives that we want to optimize,
Objective, = Objective, = 2 10 00l12
where the first objective represents the original loss function and the second objective represents the distance between the fine-tuned and pre-trained weights. Our goal to minimize both at the same time to achieve ID generalization and OOD robustness.
3.2 PROJECTING CONFLICTING GRADIENTS
Viewed from this perspective, we can leverage prior multi-objective methods towards our problem. PCGrad (Yu et al., 2020) hypothesizes that the key optimization issue multi-objective learning arises from conflicting gradients, where gradients for different objectives point away from each other. Thus, optimizing one them will lead the suboptimality of the others. They propose a form of gradient surgery by projecting task's 's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient, therefore benefiting all objectives.
Inspired by PCGrad, propose the following algorithm for robust finetuning: When the gradients between the two objectives are in conflict, i.c. their cosine similarity negative, we project the gradient for the original loss function the orthogonal direction of the gradient for the regularization
3

---

Published as a conference paper at ICLR 2025
term. Specifically, the gradients for the two objectives and the projection of the first gradient in the direction of the second gradient are respectively:
91 VoL(O), 92 = = 0o: m
9192
02
1 l?
We add hyper-parametes [0, to further control the projection strength. 0 is equivalent to an unconstrained gradient update, while = 1 is the same as full orthogonal projection. The final projected gradient the following:
g wit pooj
0102 00
92,
E [0,
(4)
Note that for L2-SP, the gradient for the regularized loss function is formulated similarly:
g VoL(0) = v vo(( X(e 6o) タ1 Ag2
(5)
Thus, DiGraP is equivalent to L2-SP with different À for every layer. summary, for each layer i:
Gradients are non-conflicting (0i 2 0): X 0
Gradients are conflicting 01
0): X
0igt D
Compared to L2-SP, the hyper-parametts in DiGraP is within the range between 0 and 1, which is more intuitive to tune. Furthermore, even with one fixed w, the regularization strength varies across both layers and iterations, making the fine-tuning process more flexible fit the training data.
3.3 LAYER-WISE TRAINABLE DIRECTIONAL GRADIENT PROJECTION
We emphasize that the regularization problem in Eq. D still not fully equivalent to the multi-objective optimization Eq. Specifically, for a multi-objective optimization problem, we want to optimize all objective functions, i.e., to minimize both (() and 1 | 0p 13 our case. However, for regularization problem. the regularization term does not necessarily decrease. Instead, it acts constraint on the original loss function and the regularization term is smaller compared to the one in model trained without regularization. Projecting the original gradient to the orthogonal direction of the gradient for the regularization term will potentially lead to underfitting. It is especially detrimental at the beginning of the training, where the fine-tuned weights close the pre-traine mre-triieed weights, thus is more benefitial for the model to stick to its original gradient descent direction.
As result, aim for the projection strength to be dynamic throughout the training process. Intuitively, w should start small during the early iterations, allowing the model to prioritize fitting to the downstream task. As training progresses and the fine-tuned model diverges further from its initial state, w should gradually increase to guide the fine-tuned gradient direction towards alignment with the regularization gradient direction. In Sec, 5.2 we will visualize the variation of projection strength throughout training to further validate this motivation.
To achieve this, we make the projection strength trainable, allowing adapt throughout the training process. For the step of unconstrained gradient descent with the learning rate of a, the model weights update as follows,
0. 0,-1 1 o
where 8t, f:-1 and Jt.1 denote the unconstrained model weights current step t, the model updates of previous step - and the gradient for the original loss function at current step l.
For one step of directional gradient descent, the model weights update as follows,
19. Gr. () O, + crwi 0r.2
19,
= = a We 1
where 0,, w, and 9..2 denote the constrained model weights, the projection strength and the gradient for the regularization term current step f. 0, and Øt,1 the the in Eq.
4

---

Published as a conference paper at ICLR 2025
The derivative of the original loss function [(6) w.r.. is follows using the chain rule:
oā(0:-1) Ow
201-1 = agc,119-1.1
V ==
(8)
0
dw
We initialize wi = O for the first iteration and update with learning rate of We also add normaliza- tion on Vw for numerical stability. For DiGraP, instead of tuning the weight decay À in L2-SP, we only tune the learning rate of the projection strength w. We argue that tuning μ is less sensitive and will provide sensitivity analysis in Sec. 5.2 We also compare fixed and trainable projection strength The final algorithm of DiGraP is illustrated in Alg.
in Sec. 5.3
Algorithm 1 Adam with Trainable Directional Gradient Projection
Input: 0g: pre-trained model, learning rate, μa: learning rate for w. (2) <- (0.9, 0.999)
Initialize: my <- 0, Vg +
for 1 to T do
0t,1 +- VaL(0r-)
b Gradients of the Objectives (Eq. 3
_ + 02 0o
G.1 10 9t,2
>> Gradient Projection (Eq.
if 1. then
We +
Initialize w
else
Vwi + Normalization(a--11111
Updating w (Eq. 8)
+ max(0, min(1, Adaml,,AammUie AdamUpdate w1-1, Vwl, p4, t))
if Jt.1 10,2 < 0 then
9t Dt.1
b Unconstrained Gradient Descent (Eq. 6
else
g: 0t,1 Wr.hi11 proj
Directional Gradient Projection (Eq. 4 Eq.
t 8pm-1 + 3.
Ut - 8244-1 (1 32)ge
Bias Correction:
T-31
U.
f- T-
Update: 0g < - vuat
fig
In summary, our unique contribution lies in adapting gradient projection the specific challenges of robust fine-tuning. While PCGrad was designed for multi-task learning, DiGraP extends these principles the fine-tuning of pre-trained models by introducing a hyper-optimizer and tailoring gradient projection to balance both ID and OOD performance. This refinement allows us to address the unique trade-offs in robust fine-tuning, which are distinct from those multi-task optimization.
3.4 COMPATABILITY WITH PARAMETER -EMEETEERREFIIEETT FINE-TUNING (PEFT) METHODS
DiGraP is further compatible with PEFT methods such as LoRA (Hu al. 2021 which is prevalent fine-tuning strategy for large foundation models. PEFT methods generally update new parameters to add to the original weights. this case, instead optimizing the distance between fine-tuned and pre-trained weights 1 Oo 1 we minimize the distance between the updated weight and origin & | in PEFT. Thus, when combined with PEFT, DiGraP does not need to pre-trained copy and requires the same amount of memory as PEFT. In Sec. [.2 and Sec. 4.3 we demonstrate that DiGraP can further improve the results of LoRA on VQA tasks.
additional
EXPERIMENTS
Overview. We test DiGraP on a variety of benchmarks, tasks and architectures to validate its
effectiveness. The experiments split into three sections including image classification (Sec. 4.1).
reformulating image classification as VQA tasks (Sec. 4.2) and fine-tuning on VQA datasets (Sec, 4.3
We emphasize that it is important move robust fine-tuning towards multi-modal models, given their
5

---

Published as a conference paper at ICLR 2025
Table 1: DomainNet Results using MOCO-V3 pre-trained ResNet50 with Real Data. DiGraP outperforms baselines on average OOD. Bold: best. Underline: second best.
ID
OOD
Statistics ID A (%)
Real
Sketch
Painting
Infograph
Clipart
OOD Avg.
00D A (%)
Vanilla FT
81.99
31.52
42.89
18.51
44,98
34.47
0.00
0.00
Lincar Prob.
73.01
24.10
39.56
12.27
30.38
26.58
-10.96
-22.90
Partial Fusion
78.27
27.72
39.74
15.56
38.18
30,30
.4.55
-12.11
L.2-SP
81.51
34,91
45.76
18.97
45.29
36.23
-0.59
5.09
MARS-SP
81.89
34.44
45.05
46.36
36.45
-0.13
5.74
LP-FT
82.92
34,50
45,42
20.12
47.11
36.79
1.13
6.72
TPGM
82.66
35.35
46.20
20.13
45.75
36.86
0.82
6.91
FTP
82.17
36.26
46.58
46.97
37.62
0.22
2.13
DiGraP
82.20
36.43
46.75
21.40
47.46
38.01
0.26
10.27
popularity and richness in terms of different types and strenghts of distribution shfit. For Sec. 4.1 we use discriminative backbone ImageNet pretrained MOCO-V3 ResNet50 (Chen et al. 202 as the pre- pre-trained model. We follow the setting the previous work and further details can be found at Tian al. (2023a). For Sec.
generative backbone Google's recently released PaliGemma (Beyer et al., 2024) pretrained on broad mixture of large-scale vision-language tasks, which is lightweight and achieves state-of-the-art performance VQAv2. In Appendix 7.1 we include additional results with more backbones and a langugae-only experiment.
| and Sec. |.3.
Datasets. For Sec. 4.1 and Sec. 4.2
DomainNet (Peng al. 2019) the benchmark, which consists of six domains (real, sketch, painting, infograph, clipart and quickdraw) with 345
classes. We fine-tune our model on real domain and evaluate on all other domains. For Sec. 4.3
fine-tune on VQAv2 (Goyal et al.) (017) and test on nine OOD datasets using LoRA (Hu et al. 2021 For the near OODs, we evaluate on VQAv2's six variants, namely IV-VQA (Agarwal al. 2020)
CV-VQA (Agarwal et al.] 2020). VOA-Rephrasings (Shah et al. 2019). VQA-CP v2 (Agrawal et al. 2018), VQA-CE (Dancette al. 202T) and AdVQA (Sheng et al. 202T), which cover uni-modal, multi-modal and adversarial distribution shifts from VQAv2. We also include TextVQA (Singh et al. 2019), VizWiz (Bigham et al.) and OK- VQAv2 Reichman et al. 2023), which are constructed from
difterent sources than VQAv2, as the far OOD datasets. Further details can be found in Sec.
Training Details. The hyper-parameter is found through cross-validatio per dataset, and the model with the best ID validation accuracy is taken. We leave all training details to Appendix 7.2
4.1 IMAGE CLASSIFICATION EXPERIMENTS
DiGraP outperforms robust fine-tuning baselines image classification task. We utilize the DomainNet as benchmark and compare DiGraP with several existing methods using ImageNet pre-trained MOCO-V3 V33 ResNet50 as initialization. We follow the training scheme and use the same hyper-paramen parameters of prior work (Tian et al.) (023a). In Tab. we observe that DiGraP achieves the best OOD performance across all OOD domains and competitive ID performance on the real domain. Specifically, DiGraP outperforms L2-SP (Li et al.) 2018)) and magnitude based projection methods (Tian et al. 2223a b) on both ID and OOD results. Note that use the reported baseline results from Tian et al. (2023b) where the Quickdraw results are not presented.
4.2 REFORMULATING IMAGE CLASSIFICATION AS VQA TASKS
Previous work primarily focuses on uni-modal benchmarks but is seldom tested on multi-modal settings. As an additional contribution, we first bridge the gap by using the same benchmark but reformulate it as VQA task and tested several robust fine-tuning methods. Specifically, inspired by Ging et al. (2024), we change DomainNet to DomainNet-o VQA by using the same images but asking questions such as "What is in the image?" with class labels as the ground truth answers. To make the two tasks more comparable, we use the ClipMatch (ClipM) metric from Ging et al.) (2024) by embedding the model prediction and class names with EVA-Clip (Sun et al. 2023) and obtain the most similar one by matching them using cosine similarity. We consider this benchmark as one OOD dataset with distribution shifts only in image modality and will conduct
comprehensive

---

Published as a conference paper at ICLR 2025
DomainNet-oVQA Visual Question Answering
DomainNet- - Image Classification
Apple BCBI
Gon B01
Disctminative
Bigeck
Generative Model
Answer
EVA-Clp
Costro Sariatt Seersh
Whet 3.6
Figure 2: Reformulating Image Classification VQA. (Left) DomainNet as conventional Image Classification tasks. (Right) DomainNet-oVOA as VQA tasks. We use the same images but add questions "What is in the image?" generate answers and use original labels as the ground truth answers. To make the two tasks more comparable, we feed the generated answers and 345 class labels into the EVA-Clip text encoder and use cosine similarity search to obtain the most similar class.
experiments with distribution shifts in other modalities in Sec. 4.3 vision-language model (VLM) PaliGemma (Beyer et al. 2024 as initialization.
We use the pre-trained generative
Table 2: DomainNet-o VQQ VQA Fine-Tuning Results. PaliGemma-3B fine-tuned on DomainNet-o (Real) and evaluated on other domains as a VQA task. We use LoRA for efficiency. Note that L2-SP reduces to Vanilla FT with AdamW under LoRA. DiGraP achieves SOTA results on both ID and OOD performance. Bold: best. Underline: second best.
ID Real
OOD Infograph
Statistics ID A (%)
Sketch
Painting
Clipart
Quickdraw
OOD Avg.
OOD A (%)
Zero-Shot
64.09
39.00
39.80
28.87
57.86
3.27
33.76
Vanilla F
92.57
70.65
71.17
54.75
82.68
18.73
59.60
0.00
0.00
Linear Prob.
91.10
68.52
67.23
49.86
79.78
19.74
57.03
-1.59
-4,31
LP-FT 92.60
71.13
70.64
54.66
81.41
20.94
59.76
0.03
0.27
WT
80.57
64.70
65.08
43.89
72.98
17.80
52.89
-12.96
-11.26
FTP
87.49
71.38
69.33
53.33
79.18
19.74
58.59
-5.49
-1.69
DiGraP (0.5)
92.72
72.56
72.31
57.52
83.32
20.13
61.17
0.16
2.63
Vanilla fine-tuning outperforms zero-shot on both ID and OOD datasets. Surprisingly, vanilla improves the performance every domain in DomainNet- oVQA, while doing the same but as an image classification task degrades the OOD performance compared to zero-shot (Wortsman et al. 2022). Note that during fine-tuning, in the image classification task, we only take the image encoder from the backbone model and add a linear head after it, i.e., we remove the text encoder and uses cross-entropy loss which is different from the pre-trained loss used during pre-training. However, (Goyal er al. 2023) points out that fine-tuning is more robust when trained with the same objective pre-training. This may be potential reason for the robustness of vanilla fine-tume monettunng for VQA tasks, since fine-tuning on the VQAs does not change the pre-training model structure and loss function. We will not focus on this problem in this paper and leave further discussion to future work. Nevertheless, one interesting question remains: when vanilla fine-tuning performs well on datasets with distributon shifts, can previous robust fine-tuning baselines further increase the robustness?
fine-tuning on Semainnet-Reel-oVeC
WiSE worsens both ID generalization and OOD robustness under VQA tasks. Wortsman et al. (2022) reports that ensembling the weights of the zero-shot and fine-tuned models can benefit from both the robustness of the zero-shot and the downstream adaptation of the fine-tuned models for image classification tasks. However, WiSE is highly dependent the performance of the pre-trained model, and only works when vianilla fine-tuning decreases the robustness. In Tab. 2 there is huge gap between the zero-shot and fine-tuned results. In this case, naively combining the pre-trained weights reduces the model's 's robustness under all distribution shifts compared to vanilla fine- tne-tuning.
Directional Gradient Projection benefits the models in general. In Tab. 2 DiGraP outperforms two-stage training (LP-FT)), weight interpolation (WiSE-FT) and bi-level optimization (FTP) on both IDD and average OOD performance. We argue that WiSE-FT only works for the setting where zero-shot performance is strong while DiGraP is beneficial for general cases.
Same as L.2-SP Li ct 2018 under LoRA (uu
2021

---

Published as a conference paper at ICLR 2025
4.3 FINE-TUNING ON VQA DATASETS
We now conduct a comprehensive experiment on various VQA datasets with different distribution shifts. We consider VQAv2 (Goyal al. 2 as the ID dataset. We further evaluate the model on six near OOD datasets which construct different types of distribution shifts from VQAv2 and three far OOD datasets in which the image and text sources are completely different from VQAv2.
Shifts
V+Q+A Shift
000
OCD
platt
Figure 3: ID and OOD VQA datasets with Uni-modal and Multi-modal Distribution Shifts.
ID Dataset. VQAv2 (Goyal al. 2017) builds upon VQAv1 (Agrawal et al. 2017) by balancing question-answer pairs with complementary images to minimize the bias in language priors, thus is more challenging and is widely used benchmark for popular vision-language models.
OOD Datasets. 1) Distribution Shifts to Images. IV-VQA (Agarwal et al. 2020) and CV-VQA (Agar
wal et al., 2020) remove the objects irrelevant to answering the question and generate complementary images with one instance of the object removed respectively. 2) Distribution Shifts to Questions. VQA-Rephrasings (Shah al., 2019) collects three rephrasings of each question. 3) Distribution Shifts to Answers. VQA-CP (Agrawal et al.) 2018) reorganizes the correlation between the question type and correct answer. 4) Distribution Shifts to modlii-modlliiiss. VQA-CE (Dancette al.) 2021) selects subset of VQAv2 that are counterexamples of potential multi-modal shortcuts. 5) Adversarial Distribution Shifts. AdVQA (Sheng et al. (021) provides human-adversarial examples for questions where the model's predicted answer is incorrect. 6) Far OODs. TextVQA (Singh et al., [2019) requires models to answer questions by understanding text embedded in images. VizWiz (Bigham et al.) contains aser-generated images with diverse challenges like poor quality, ambiguity, and irrelevant content for answering visual questions. OK-VQAv2 (Keichman et al. 2023) represents a knowledge- based VQA task where the visual question cannot be answered without external knowledge.
Evaluation and Metrics. We follow the metric from VQAv2 (Goyal et al., 2017) and the evaluation is based on the accuracy of predicted answers compared ground truth human-annotated answers. For each question, the dataset includes 10 human- provided answers. The accuracy is calculated as:
Accuracy = min number of humans who gave the answer
3
Measuring the OOD Distance. We explore shifts on single image modality and joint image question (V+Q) shifts, as well as image question answer (V+Q+A) shifts by computing the test set shift relative to the training domain (i.e. VQAv2 train) using the negative Mahalanobis distance metric. The higher the value, the less the distribution shift. More details are in Appendix 7.3
Experimental Results. We fine-tune the PaliGemma-31 model on the VQAv2 dataset with LoRA and evaluate on the other OOD datasets. The results of DiGraP and other robust fine-tuning methods shown in Tab. We have the following observations.
Smaller distribution shifts correlate with better OOD performance. The analysis of image and joint shifts reveals a high correlation with VQA performance, evidenced by correlation values of 0.83 and 0.80, respectively. This suggests that larger shifts significantly degrade VQA performance. Such trends validate our methodology in quantifying shifts, as far OOD scenarios align with increased shift levels and diminished performance.
Full fine-tuning improves zero-shot performance across ID, near OOD, and far OOD datasets. As shown in Tab, 3 we observe no degradation in OOD performance following vanilla fine-tuning even when PaliGemma is pre-trained on VQA tasks. This may be attributed to reasons similar to
Same as L2-SPP Li
2018 under LoRA Hu al. 2021
8

---

Published as a conference paper at ICLR 2025
Table Visual Question Answering Fine-Tuning Results using PaliGemma-3B. DiGraP outper- forms baselines across ID and near OOD and is competitive on far OOD datasets using LoRA. Note that Vanilla FT with Adam W reduces L2-SP under LoRA. Trainable projection strength Bold: best. Underline: second best.
ID
Near OUD
Fur
Adversaii AdVQA
VQQW2 (val)
VVA
CV VQA
VgJ Rep.
CP
VQJ CE
Avg.
TeMtVQA
OK.
Avg.
Avg.
Shart
54.42
63.95
44.72
$4.29
30.46
45.70
14.36
16.54
28.60
Venilla Fi
22.02
78.24
67.39
42.37
55.00
LP
83.97
15.55
76.49
45.08
15.38
19.01
43.27
WISE-FI1
71.36
61.55
66.42
70.89
08.74
63.55
63.27
36,98
22.41
42.35
53,48
FTP
81:77
67.95
76.06
10.00
72.29
49.12
15.67
$1.07
41.95
62.13
DIGruP (0.5)
$7.40
95.36
68.56
79.29
$7.19
73,63
51.41
75.87
44.32
23.98
19.20
35.77
63.50
Visien Shitt
21.84
27.37
26.67
27.54
27.95
21.91
27.06
27.88
28,95
12.92
27.58
20.94
28.57
Queatin Shin
56.70
19.08
28.10 m0
40.84
16.50
$1.19
40.68
32.92
47.00
46.75
48.70
47.3 18
40.85
Jeiss Shift
34.81
32.91
29.55
33.26
24.81
37.10
34.97
34.10
45.72
45.69
40.91
44.11
37.44
those discussed in Sec. 4.2 or due to differing characteristics of the backbone models and tasks. Once again, WiSE negatively impacts both ID generalization and OOD robustness VQA tasks when interpolating between pre-trained and fine-tuned weights.
DiGraP outperforms baselines on both ID and near- OOD datasets. Beyond improvements on uni-modal tasks, DiGraP enhances enhancess vanilla fine-tuning and consistently outperforms other baselines in multi-modal settings. As shown in Tab. 3 DiGraP achieves the highest ID and average near- near-OO1> results, demonstrating robustness to distribution shifts across various modalities, including vision, question, answer, their combinations, and adversarial shifts.
DiGraP is competitive on far OOD datasets. From Tab. 3. we observe that DiGraP also improves vanilla fine-tuning on the three far OOD benchmarks and is the second best among all robust fine- tuning methods. Notably, while FTP (Tian et al. 2023a) performs well on OOD, it severely underfits the training domain with a substantial lower ID performance compared to vanilla fine-tuning and DiGraP, with the positive gradient annealing factor к O, which indicates weakest regular- ization. However, FTP demonstrates outstanding performance on far OOD tasks with significant higher results on the three far OOD datasets. One potential reason that FTP imposes much stronger regularization since even when k = 0 the projection constraints are non-decreasing during training, which means it still provides regularization. In FTP, the authors also mention that to obtain the best performance if underfitting is observed. However, in DiGraP. = 0 is equivalent to unconstrained fine-tuning with no regularization. Thus, DiGraP enforces weaker regularization than FTP. We emphasize that the significant performance decrease from zero-shot to fine-tuning is mostly observed on rather simple tasks (e.g. image classification), while DiGraP is more general for all cases. Nevertheless, it remains interesting why FTP significantly increases the far OOD performance given the zero-shot is poor in Tab. 3 and we will leave the exploration to future work.
is necessary
HYPER-PARAMETER TUNING AND ABLATION STUDY
5.1 VARIATION OF PROJECTION STRENGTH THROUGHOUT TRAINING
We visualize the variation of the average projection strength w of all layers over iterations for five different hyper-parameters parameters μ E (0.01, 0.1, 1, 100} in Fig. 4 As we increase μ. the projection strength becomes larger. For all cases, the projection strength w starts from zero and converges at the end of the training. This aligns with our intuition that the projection strength should vary over time learn dynamic priority of the objectives during different stage of training.
5.2 IMPACT OF
SENSITIVITY ON ROBUSTNESS
We further perform the sensitivity analysis of the hyper-parameter f on ID and average OOD performance for DomainNet-oVOL, and VQA experiments. Results from Tab. 4 show that both ID and OOD results fluctuate slightly even when spans over a wide range from 0.0T to 100. This again proves that DiGrap is more controllable and less sensitive to hyper-parameter change.

---

Published as a conference paper at ICLR 2025
Domainllet (Real)
VOAu?
Anly mwl MM
0.4
Projecten Streeghh
Stroggh
2.40
Herabiors
Figure 4: Variation of the Average Projection Strength of all Layers over Iterations. We present the results of μa & {0.01, 0.1, 0.5, 100). We use the sliding window with window sizes 50 and 200 visualize the results for DomainNet-o VQA (Real) and VQAv2. The projection strength is dynamic over iterations, growing from small to large and converging in the end.
Table 4: Sensitivity Analysis of Hyper- Parametr i ID and OOD performance. We sweep E (0.01,0.1 0.1, 0.5, 1, 103. For DomainNet-oVOZ and VQA experiments, both ID and average OOD performance fluctuates slightly and robust the change of over wide range.
Saper-Parammter////
100
Hyper- Prrper-Prrmmterr y
0.01
00D Avg.
60.95
59.96
61.17
60.31
60.40
OOD Avg.
63.15
62.78
63.50
63.52
63.44
ID
92.52
92.68
92.72
92.65
92.45
ID
86.91
86.85
87.40
87.08
87.18
(a) DomainNct-oVQA hyper-parameter (μ) sweep.
(b) VQA hyper-parameter (/1) sweep.
Table 5: Comparing Fixed and Trainable Projection Strength on DomainNet-oVOA. Bold: best. Underline: second best. Trainable projection strength outperforms different fixed projection strengths both ID and average OOD,
Real
Sketch
Painting
Infograph
Clipart
Quickdraw
OOD Avg.
DiGraP (w 0.1)
92.49
71.86
71.36
56.34
83.15
19.22
60.39
DiGraP (w 0.5)
92.63
71.24
71.46
56.08
83.
18.61
60.11
DiGraP 0.9)
92.53
73.09
71.73
56.34
82.85
20.11
60.82
DiGraP (trainable)
92.72
72.56
72.31
57.52
83.32
20.13
61.17
5.3 ABLATING FIXED AND TRAINABLE PROJECTION STRENGTH
To validate the effectiveness of trainable projection strength w, we conduct analysis to compare with fixed projection strength with different values w {0.1, 0.5, 0.9}. Tab. Tab. shows that trainable DiGraP outperforms the others and achieves the best ID and average OOD results.
CONCLUSION
We present Directional Gradient Projection (DiGraP), a novel method for robust fine-tuning that leverages gradient- braseent-based directional information to unify regularization and multi-objective opti- mization. DiGraP addresses hyperparameter sensitivity and underfitting issues in existing methods. Experiments image classification and VQA benchmarks show that DiGraP surpasses baselines, improving ID accuracy and OOD robustness, while bridging uni-modal and multi-modal evaluation for robust fine-tuning across domains. However, DiGraP struggles with far OOD datasets due lim- ited regularization, excelling in near OOD scenarios but facing trade-off as stronger regularization may harm ID and near OOD performance (Sec. 4.3 Future work should balance ID, near OOD, and far OOD performance. Additionally, DiGraP is suited for fine-tuning well pre-trained models, with its efficacy in training from scratch or non-robust initialization yet to explored.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9687-9695, Seattle, WA, USA, June 2020. IEEE ISBN 978- 777--1722177-16--1 doi: 10.11000CVP42220000000000 //ieeexplore.c leee. org/ carpuumennt995444/
URL https
Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1.0 dataset, 2017. URL http:::/ttaatttt http::::///xxvvrr/////
fabs/1044 08243
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. June 2018. URthttp:
00377 arXiv:1712.0037 [cs].
Aishwarya Agrawal, Ivana Kajić, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, and Aida Nematzadeh. Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization, April 2023. URL
/bs/
22C5. 12191 arXiv:2205. 12191 [cs].
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby. Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender. Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai PaliGemma: versatile 3B VLM for transfer, July 2024. URL http:
V/arxiv. wrg. abw./4//2220. 07726 arXiv:2407.07726 [cs] version: 1.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: nearly real-time answers to visual questions.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self suff-super vision transformers, 2021. URL URL /tttps:///axiiiiii org/ a2s. 02057
Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond Question- Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering, September 2021. URL
http://aaxxv..rr///b///2220
absgg//bs//100 03149 arXiv:2104.031 [cs].
Simon Ging, María A. Bravo, and Thomas Brox Open-ended vqa benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy, 2024. URL https
//axxiv.org/ abs 2402. 07270
Henry Gouk, Timothy M. Hospedales, and Massimiliano Pontil. Distance-Based Regularisation of Deep Networks for Fine-Tunine Tuning January 2021. URL http://aaaxii..oggaa////////////
08253 arXiv:2002.08253 [cs, stat].
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19347, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, May 2017. URL http:
org/ abs 111..611122 C0837
[cs].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -Zuu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: LawkRank Adaptation of Large Language Models, October 2021. URL. http::/ orgg abs/ 2006. 09685 arXiv:2106.09685 [cs].
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-Tuning can Distort Pretrained Features and Underperform ciitof--iitibuttinn February 2022. URL
http://arxi v. or. wwwg/abb//aaa /2002 10054 arXiv:2202. 10054 [cs].

---

Published as a conference paper at ICLR 2025
Dongxu Li, Jumnan Li, Hung Le, Guangsen Wang. Silvio Savarese, and Steven C. H. Hoi. Lavis:
A library for language- vision intelligence, 2022. URL htttps://atxxiiii wrt. otqg/bb///20/
09019
Linjie Li, Zhe Gan, and Jingjing Liu. Closer Look at the Robustness of Vision-and- Language Pre-trained Models, March 2021. hwl.tttp///raai.o//////// uXXv:2012.08673 [cs].
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit Inductive Bias for Transfer Learning with Convolutional Networks, June 2018. URL htttat/.//araw....////////2222
arXiv:1802.01483 [cs].
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL
attpss://arxuu ww..org///..o.o. bbrg/abs/// / 2304 08485
Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang, Jun Liu, Hongbin Pei, and Junzhou Zhao. Robust Visual Question Answering: Datasets, Methods, and Future Challenges, February 2024. URL arXiv:2307. 1471 [es].
fttp:://arxiv.org/bbbo..
Bac Nguyen, Stefan Uhlich, Fabien Cardinaux, Lukas Mauch, Marzieh Edraki, and Aaron Courville. SAFT: Towards Out-of-Disttrbutiin Generalization in Fine- Tone-Tuning, July 2024. URL http?
/arxiiv. www.v..rc/abs//240000
arXiv:2407.0303. [cs].
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation, 2019. Utttssts: wwwpwxwwwww.ora.ooaaaaeeeeeeee/e wttp.//r/r/bbs/.....111
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL arXiv:2103.. 103.00020 [cs].
http:://arxi org/ abs aat/aaass///1/ 2103, 00020
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https: http://awwx.....aa.../a//
10683,
Benjamin Z. Reichman, Anirudh Sundar, Christopher Richardson, Tamara Zubatiy, Prithwijit Chowd- hury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside Knowledge Visual Question Answering Version 2.0. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, June 2023. URL 10096074]
ieee
Meet Shah, Xinlei Chen, Mareus Rohrbach, and Devi Parikh. Cycle-Consistency for Robust Vi- sual Question Answering, February 2019, URL htt.:tpp///aai...orr//////220000 arXiv:1902.05660 [cs].
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kicla. Sluman-Adversae Visual Question Answering, June 2021. URL
httt:://are wrg.orgg.......... abs/2106. 02280 arXiv:2106.02280 [cs].
Xiangxi Shi and Stefan Lee. forehmmarinnggoto------iiiiiiiii detection visual question answering.
January 2024. URL httpss///pennacesss.heerrr http://seseeeeeeeoooooooooooet/a
wwwp///oooet////////// httll/Sh Bwwchherkinggggccceee.... of ott.://ooo..........o.oootoetooa
Question_Answeringg WWACV_202__P0
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read, May 2019. URL httt::///a/titi org/ abt///bb/19000 08920 arXiv:1904.0892. [cs].
Quan Sun, Yuxin Fang. Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale, 2023. URL ARI.https::0 /ttppp://axivv. wwwpwwwwwwwwoooooooaoo/raaa/aare /2003 15389
12

---

Published as a conference paper at ICLR 2025
Junjiao Tian, Xiaoliang Dai, Chih- Yao Ma. Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable Projected Gradient Method for Robust Fine-tuning, March 2023a. URL 2.03.1102220 1XXv:2303.11070 [es].
Junjiao Tian, Yen-Cheng Liu, James Seale Smith, and Zsolt Kira. Fast Trainable Projection for Robust Fine-Tuning, October 2023b. URL http:://rrxi..orrr/bb//311--. arXiv:2310.19182 [es].
Junjiao Tian, Chengyue Huang, and Zsolt Kira. Rethinking weight decay for robust fine-tuning of foundation models, 2024. URL
01713
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, June 2022. 2109. 01903 arXiv:2109.0190) [es].
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020. URL pttpss//arxx.v.orrr//////0000 06782
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and Ilm evaluations, 2023. URL https: http:/://aaxvvvveaa org. /tt.//3365. 04618
13

---

Published as a conference paper at ICLR 2025
7 APPENDIX
7.1 ADDITIONAL RESULTS
More Backhones: CLIP, LLaVA. We include experiments of fine-tuning DomainNet with CLIP ViT-Base (Radford et al., (021) (Tab. 8) DomainNet-oVQA (Tab. 7 and VQA (Tab. (
with LLaVA-7B (Liu al. 2023). DiGraP consistently achieves the best performance across all experiments.
Table Visual Question Answering Fine-Tuning Results using LLaVA-7B (Liu al., 2023). We sample 10% of the VQAv2 training and validation set. We fine-tune using LoRA with a rank of 4 and target on the w., W.. DiGraP outperforms baselines across and near OOD and competitive on far OOD datasets using LoRA. Note that Vanilla FT with Adam W reduces to L2-SP under LoRA.
Trainable projectionn strength Bold: best. Underline: second best.
0.D
Vainn CV.VQA
Quaziion VQA-Rap
Answerr VQA-CP +2
Multinshil VQA-CE
Advanonial ALVQA
(QA+2
PV-VQA
Avg.
TcuVVQ
VidWia
OK-VQA
Ang.
0OD Avg.
Zao-Sher
3:27
6:34
4.40
3,00
4.26
1.46
1.32
3.46
1.10
0.34
0.71
0.65
2.53
Vanila r
72:49
52.21
58.61
63:95
69.50
$5.04)
40.12
60112
12.15
12.11
16.25
25.02
49.00
LP-I7
51.0
39:26
38.54
37.95
33.14
9.34
23.66
2861
7.50
5.16
9.55
7.64
31.63
WISE-FT
60.47
63.95
46.39
30.26
35.79
20:23
23.35
43.33
10.00
3.15
13.97
9.07
31.98
FTP
67:85
80.65
57.33
61.66
68.05
43.70
39.53
58.49
33.88
12.98
31.77
26.21
47.73
DIGraP (0.00)
72.54
83.64
36.55
64.70
69.71
45.15
43.10
60.45
38.22
12.97
37.43
29.54
50.17
Table 7: DomainNet-oVVA Fine-Tuning Results. LLaVA-7B (Liu et al. DomAinNeetoVQA (Real) and evaluated other domains as VQA task. We use LoRA for efficiency. Note that L2-SP reduces to Vanilla FT with Adam W under LoRA. DiGraP achieves SOTA results on both ID and OOD performance. Bold: best. Underline: second best.
2023) fine-tuned on
ID Real
OOD
Statistics ID A (%)
Sketch
Painting
Infograph
Clipart
Quickdraw
Avg.
OOD (%)
Zero-Shot
67.71
$3.45
56.15
37.90
58.32
15.87
44,34
Vanilla F
86.03
63.93
59.47
45.50
72.85
15.94
51.54
0.00
Lincar Prob.
76.84
51.41
50.53
32.67
60.86
8.72
40.84
-10.68
-20.76
I.P.FT
79.76
$6.12
55.77
37.10
66.03
12.82
45.57
-7.29
-11.58
FTP
77.70
55.96
43.74
39.05
62.61
15.78
43.43
-9.68
-15.74
DiGraP (0.1)
85.68
64.40
60.26
45.94
72.95
16.69
52.05
-0.41
0.98
Table 8: OOD Results on DomainNet. CLIP ViT-Base finetuned on DomainNet (Real) dataset and evaluated on other domains. Bold: best. Underline: second best.
ID Real
OOD
Statistics TD A (%)
Method
Sketch
Painting
Infograph
Clipart
OOD Avg.
OOD A (%)
Vanilla FT
86.5
45.9
58.9
34.8
59.7
49.82
0
83.2
38.6
51.4
27.2
50.7
41.97
-3,82
-15.76
L2-SP
86.4
46.0
58.9
35.0
59.8
49.92
-0.12
0.20
LP-FT
84.4
36.5
50.5
27.2
51.4
41.40
-2.43
-16,90
FTP
86.7
49.1
55.8
32.2
61.4
49.63
0.23
-0.38
DiGraP
86.2
51.3
57.3
34.0
62.4
51.25
-0.35
2.87
2) Language-only Experiments. We also conduct language-only experiment in Tab. where use BOSS (Yuan et al.) (023)). an NLP benchmark suite designed for OOD robustness evaluation. This suite includes both ID and OOD language-only datasets across multiple tasks (e.g., Sentiment Analysis, Toxic Detection).
3) LP-FT Controlled Version. Tab. we conduct L.-FT-C experiment (a controlled version of LP-FT), similar to TPGM-C in Tian et al. (2023a), by increasing the regularization strength to ensure that the ID performance matches that of DiGraP. Despite this adjustment, DiGraP still outperforms LP-FT-C on OOD datasets.
*Same as L2-SP Li et al.
2018
under LoRA Hu et al.
2021
Same as L2-SP
et
2018
under LoRA (uu ot
2021
14

---

Published as a conference paper at ICLR 2025
Table BOSS Yuan al. 2023) Performance Using T5 (Raffel al., 2023) Fine-Tuned with AZ. We leverage BOSS, an NLP benchmark suite for OOD robustness evaluation. We focus on the Sentiment Analysis task which contains Amazon (AZ) as the ID and DynaSent (DS), SemEval and SST as the OOD. Bold: best. Underline: second best.
ID
OOD SE
Statistics ID A (%)
Dataset
AZ
DS
SST
OOD Avg.
OOD A (%)
Vanilla FT L2-SP
85.57
43.63
48.47
67.29
53.13
0.00
0.00
84.87
43.40
48.65
65.07
52.37
-0.82
-1.43
FTP DiGraP (8c-3)
84.73
43.84
48.69
66.73
53.09
-0,98
-0.08
85.60
44.10
49.07
67.39
53.52
0.04
0.73
Table 10: DomainNet Results using MOCO-V3 pre-trained ResNet50 with Real Data. DiGraP outperforms baselines average OOD. Bold: best. Underline: second best.
Statistics ID A (%)
Real
Sketch
Painting
Infograph
Clipart
OOD Avg.
OOD A (%)
Vanilla FT
81.99
31.52
42.89
18.51
44.98
34.47
0.00
0.00
Linear Prob.
73.01
39.56
12.27
26.58
-10.96
Partial Fusion
78.27
27.72
39.74
15.56
38.18
30.30
-455
-12.11
L2-SP
81.51
34.91
45.76
18.97
45.29
36.23
-0.59
MARS-S SP
81.89
34.44
45.05
19.97
46.36
36.45
-0.13
5.74
LP-FT
82,92
34.50
45.42
20.12
47.11
36.79
1.13
6.72
TPGM
82.66
35.35
46.20
20.13
45.75
36,86
0.82
FTP
82,17
36.26
46.58
20.67
46.97
37.62
0.22
9.13
L.P.T.C (reg=5)
82.18
35.13
46.02
20.36
47.61
37.28
0.23
8.15
DiGraP (0.1)
82.20
46.75
21.40
38.01
0.26
10.27
Sensitivity Analysis on Single-Modal Tasks. Tab. 11 we include the hyper-parameter tuning experiment for DomainNet on CLIP ViT-Base. The results demonstrate that DiGraP remains robust hyperparameter variations on both ID and OOD datasets in the uni-modo image classification task.
Table 11: Sensitivity Analysis of Hyper-Paraaettr μ on ID and OOD performance. We sweep (0.01, 0.5, 1, 10). For DomainNet with CLIP ViT-Base experiments, both ID and average OOD performance fluctuates slightly and are robust to the change of over a wide range.
μ
0.01
0.1
0.5
10
OOD Avg.
50.86
51.04
50.79
51.25
51.14
ID
86.00
86.12
86.13
86.14
86.12
(a) DomainNet haperrrrmntee (1) sweep.
5) Visualization of the Variation in Regularization Strength across Layers over Epochs. We present visualization of the variation in regularization strength (A) across different layers over epochs in Fig. The results show that the regularization strength evolves dynamically during training, starting small, increasing over iterations, and eventually converging. In the vision layers (blue), early layers tend to experience weaker regularization compared later layers throughout the training process. Conversely, the language layers (orange) display more uniform regularization strength, with comparable levels observed between early and later layers. The weaker regularization in early vision layers likely allows them to preserve foundational low-level features, while stronger regularization in later layers encourages the model to focus on high-level semantic representations.
7.2 TRAINING DETAILS
Image Classification. For DiGraP, we fine-tune the model using SGD with a learning rate of 1e - and μ 0.1 with batchsize of 256. The regularization sxper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 4 RTX 2080 GPUs for each experiment.
15

---

Published as a conference paper at ICLR 2025
Ray on Stoigha
Voier
5
Mesdel
130
reguiancatinn
(b) Layer- Wise Regularization Strength
Eooohs
(a) 3D Overview of the Variation of Regularization Strength across Layers and Epochs.
Fyodss
(c) Dynamic Regularization Across Epochs
Figure 5: Visualization of the Variation in Regularization Strength (A) across Layers over Epochs. Fig. ia an view of R=regularizztion strength dynamics across layers and epochs. The X-axis represents training epochs, the Y-axis represents model layers (vision in blue, language in orange), and the Z-axis represents the regularization strength À applied to each layer. Fig. [5b projects the data onto the plane formed by layers (Y) and regularization strength (Z). Fig. projects the data onto the plane formed by epochs (X) and regularization strength (Z).
Fine-Tuning DomainNet-oVOO We use the model pretrained with 224 = 224 input images and 128 token input/output text sequences and fine-tune with the precision bfloat16. We use the LAVIS (Li al. 2022) public repository fine-tune all methods. Standard hyper-parameters used for all: learning rate (le 3), weight-decay (le - 4), optimizer (AdamW), scheduler (Linear Warmup With Cosine Annealing), warm-up learning rate (le 4), minimum learning rate (le 4), accumulation steps (2), beam size (5). The model is trained for 10 epochs with a batch size 128 for Tab. 2 For LoRA (Hu al. 2021), limit study to only adapting the attention weights and freeze the MLP modules for parameter-efficiene 0.5 for all DiGraP results in Tab. 2 The regularization hyper-paramete is found through cross-validation, and the model with the best ID validation accuracy is taken. We use A40 GPU for each experiment.
specifically apply LoRA to W,, Ws, W., W, with
8 in Tab. 2
We
Fine-tuning VQA. We use the model pretrained with 224 * 224 input images and 128 token input/output text sequences and fine-tune with the precision bfloat16. We use the LAVIS (Li 2022) public repository to fine-tune all methods. Standard hyper- parameters are used for all: learning rate (le 3), weight-decay (le 4), optimizer (AdamW), scheduler (Linear Warmup With Cosine Annealing), warm-up learning rate (1e - 4), minimum learning rate (1e 4), accumulation steps (2),
beam size (5). The model is trained for 10 epochs with batch size of 128 for Tab. 3 For LoRA A (Hu
al. 2021). we limit our study to only adapting the attention weights and freeze the MLP modules for parameter-efficiency specifically apply LoRA to Wg. We, W., W. with r = 8 in Tab. 3 We
À = 0.5 for all DiGraP results in cross-validation, and the model with the best ID validation accuracy taken. We use A40 GPU for each experiment.
Tab. 3 The regularization hyper-parametee is found through
16

---

Published as a conference paper at ICLR 2025
7.3 MEASURING OOD DISTANCE
We follow procedures similar to typical feature-based OOD detection methods (Shi & Lee 2024). Specifically, given input training split X compute feature representations z of the training samples to estimate the empirical mean μe and covariance matrix Σ. For each test split, we compute the test set shift relative the training domain using the Mahalanobis distance metric defined in Eq.2 The overall shift score for each test dataset, denoted as Smaha. is calculated as the average SMaha across all samples. Let y denote the question, the image (vision input), and the answer. The input features used in measuring shifts include uni-modal embeddings f(v), f()) and joint embeddings f(q. v).
SMaha(2 (Fres)
(2nest E
(2ret (4)
We utilize the vanilla fine-tuned PaliGemma model on the VQAv2 training dataset as our feature encoder. For the image embedding f(v), we obtain it via masking out the question input tokens and mean-pooling the image portion from the final layer of the model before the language model head, Similarly, to get f()). we mask out the image tokens and extract the question portion from the final layer. To obtain f(v, q), we pass in both image and text tokens as input, compute the average embedding for both modalities and then taking the overall mean.
7.3.1 CORRELATION BETWEEN UNI- & MULTI-MODAA SHIFTS PER DATASET
Fig. shows the heatmap of the correlation between uni-modal and multi-modal shifts per dataset. Question-joint shift correlations are higher than image-joint shift correlations across all VQA datasets and fine-tuning methods. However, pre-train model maintains similar correlation between both modalities. Vanilla FT and SPD exhibits the lowest question-joint shift correlation shown by the darkest row color across all fine-tuning methods in 6a Whilst, SPD shows the lowest image-joint shift correlation across the datasets in 6b
7.3.2 HISTUGRAMS FOR EVALUATING DIFFERENT DISTRIBUTION SHIFTS
17

---

Published as a conference paper at ICLR 2025
Pretrain
0.25
0:43
0.6
Vanila FT_L
0.49
0:48
0.31
0.48
0.31
0.48
0.54
0.48
0.53
0.5
0.5
Fra-Tunng Medhoss
LP
0.64
0.66
0.47
0.61
0.06
0.64
0.05
0.51
0.66
0.6
0.4
0.2
FTP
0.58
0.58
0.46
0.57
0.59
0.55
0.66
0.57
0.61
0.6
0.1
SPD
0.52
0.51
0.35
0.51
0.53
0.51
0.58
0.52
0.55
0.54
orep
vat rephresiige
19p sp
vep ce
adkega
Tosvos
Vinet
vit 92.
est Splts
Question-Joint shift correlation heatmap
Varilla LoRA
0.3
0.32
Fin Tunina Methods
0.23
0.29
0.24
0.22
0.28
0.4
LPFT
0.34
0.26
0.48
0.37
0.33
0.32
0.38
0.13
0.35
0.3
0.2
TTP
0.33
0.29
0.42
0.35
0.33
0.33
0.29
0.46
0.25
0.36
0.1
3PD
0.17
0.15
0.16
0.19
0.10
0.15
0.14
0.27
0.0046
0.26
veal
kyea
Crog
vRe reptrasingss
v9e cp
weg ce
ahvna
laubvee
vioee
devga
Iet Splts
(b) Image-Joint shift correlation heatmap
Figure 6: Heatmap of correlation between uni-modal and multi-modal shifts per dataset.
18

---

Published as a conference paper at ICLR 2025
0.34
0.34
0.35
0.32
2.0
0.33
0.325
2.00
0.03
0.300
0.275
0.34
0.59
Notalanblle score
Maha anoble score
Maharanbbis score
(a) VQAv2 Val
(b) IV VQA
(c) CV VQA
2.14
0.24
0.34
332
0.12
0.22
2.30
0.33
0.30
0.32
0.5
Kuhalonobes Score
Mahol anoiii score
Mahalanobis Score
(d) VQA Rephrasings
(e) VQA CP
(f) VQA CE
2.4
0.34
teanhege
0.34
viwi
0.30
0.33
2.00
0.32
0.22
2.00
0.2
30
0.0
40 Rahulanobes
Nabulancbrs score
score
(g) ADVQA
(h) Text VQA
VizWiz
0.38
vou_y2_trv
chigg
0.22
0.34
0.2
0.2
50
(i) OK VQA
Figure 7: Histogram for Vanilla FT Visual Shifts: We depict the SMaha score on the visual modality for each sample in the VQAv2 train split in blue and the corresponding test samples in orange. There's minimal visual shifts for all VQA datasets from the VQAv2 train, except for Figure fi which shows evidence of greater shifts between the orange distribution and the blue distribution.
19

---

Published as a conference paper at ICLR 2025
0.38
0.33
Pecge50
2.04
0.20
2.00
0.02
3.00
0.31
100
conbilanbleecoo score
Matal anoble score
Waharanbiis scone
(a) VQAv2 Val
(b) IV VQA
(c) CV VQA
0.54
0.56
3.5
2.04
0.044
04
0.23
0.32
0.52
2:0
0.31
0.2
3.00
0.32
0.58
Ruhalonobis Score
Mahal amobis score
Mahalanobis Score
(d) VQA Rephrasings
(e) VQA CP
(f) VQA CE
0.09
0.28
Teanheg
viww
2.044
2.004
2.00
2.00
0.37
0.5
2.0
2.00
20
0.32
100
0.20
80
100
Mahulancbis score
score
Mahulanzbaa
(g) ADVQA
(h) Text VQA
VizWiz
0.32
vou_y2_tra
0.35
2..33
0.82
0.22
20
60
100
( OK VQA
Figure 8: Histogram for Vanilla FT Question Shifts: We depict the SMaha score on the question modality for each sample in the VQAv2 train split in blue and the corresponding test samples in orange. Similar to the visual shift histograms, far OODs (Figures h [] also show evidence of greater shifts between the orange distribution and the blue distribution than near OODs.
20

---

Published as a conference paper at ICLR 2025
0.32
0.32
2.00
0.32
20.00
0.06
0.01
2.0
0.32
0.53
Natttanoble score
Mahal bnoble score
Maharanobis scone
(a) VQAv2 Val
(b) IV VQA
(c) CV VQA
0.22
0.38
2.00
0.31
0.28
0.35
0.26
2.04
0.34
0.034
0.32
0.22
2:00
0.32
0.5
Muhalonobss Score
Mahal anobis score
Mahalanobis Score
(d) VQA Rephrasings
(e) VQA CP v2
(f) VQA CE
2.30
0.32
0.30
vibw
3.0
0.32
0.28
2.06
0.02
0.20
2.04
0.244
0.044
3.0
0.37
0.5
0.00
30
40
03
80
0.32
20
40
29
90
0.20
20
50
Nahulancbis score
Mahull anoble score
Kaharanzbes
(g) ADVQA
(h) Text VQA
VizWiz
0.32
vou_y2_tra
0.33
0.36
0.34
0.32
0.02
20
40
60
00
(i) OKVQA
Figure 9: Histogram for Vanilla FT V+Q Shifts: : We depict the SMaha score on the V+Q shift for each sample in the VQA22 train split in blue and the corresponding test samples in orange. For all test splits, V+Q shifts show a greater degree of shift compared to the corresponding visual and question shift.
21


Reviews:
[{"review_id": "CDY6TosXx6", "reviewer": "Reviewer_fvUQ", "summary": "The paper proposes a new method for robust fine-tuning of models for robust fine-tuning that maintains model performance on out-of-distribution datasets while fine-tuning on in-domain data. The authors frame robust finetuning as a multi-objective optimization problem where the two objectives are the original loss function and the distance between the fine-tuned and pretrained weights. They propose an algorithm which projects the gradient of the original loss onto the normal plane of the regularization term’s gradient when the two gradients do not conflict. This approach is evaluated on image classification and VQA tasks, demonstrating improved performance on both in-domain and OOD datasets.", "questions": "1. In Table 1, while DiGraP performs well on OOD domains, the performance on the ID domain is poor compared to other approaches like LP-FT and TPGM. By changing the strength of ‘w’, does the approach allow a trade-off to improve performance on ID while compromising OOD performance?\n2. In Table 4(b), when the learning rate is increased, which makes the projection strength larger, why does the ID performance improve?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed approach makes the projection strength $\\omega$ as a learnable parameter, and makes it tunable by a learning rate $\\mu$. This makes the approach less sensitive to hyperparameters. The authors show good experimental evidence for the claim.\n2. The proposed evaluation of reformulating image classification as a VQA task allows for evaluations on foundation VLMs like PaliGemma.\n3. The authors show that the proposed approach can be adapted to PEFT approaches like LoRA.", "weaknesses": "1. While the paper addresses the problem of robust fine-tuning, the proposed approach Directional Gradient Projection appears similar to PCGrad. The problem of conflicting gradients and the  idea of projecting the gradient to the normal plane have been addressed in PCGrad. I would encourage the authors to highlight the unique contributions more clearly.\n2. The paper proposes a general approach for robust fine-tuning of foundation models, but the paper focuses on Image classification and VQA tasks. It would be useful to evaluate the approach by fine-tuning open-source LLMs like LLaMA on a multi-task benchmark like MMLU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkuH7HZxr2", "reviewer": "Reviewer_1rWn", "summary": "This paper proposes a novel layer-wise trainable method for robust fine-tuning of foundation models, namely DiGraP, which considers the directional information of the gradients. In addition, the authors generalize the method to multi-modal evaluation settings, filling the gap in experiment. Finally, experiments on several tasks demonstrate the effectiveness and robustness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+This work adopting multi-objective learning to alleviate the previous method's sensitivity to hyper-parameters and underfitting problems, which seems to a relatively novel attempt in the current field.\n\n+The work expands the experimental setting from single-modal to multi-modal, filling the gap in experiments.", "weaknesses": "-Although the work attempts to utilize gradient direction information, there are still deficiencies in the innovation of the method. It looks like a combination of previous work, so the innovation needs to be explained more.\n\n-Sec 4.1 lacks qualitative analysis of the experimental results, especially the reasons why the ID performance on the real domain is worse than LP-FT.\n\n-The article seems to have only conducted a quantitative analysis of hyper-parameters for multi-modal tasks, and corresponding ablation experiments are still needed on singal-modal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag0jLupRkd", "reviewer": "Reviewer_D9Ep", "summary": "The paper presents the Directional Gradient Projection (DiGraP) method, designed for robust fine-tuning of foundation models, particularly in the face of distribution shifts. DiGraP introduces a novel approach that leverages gradient directionality to resolve conflicts between optimization objectives, aiming to enhance model robustness across in-distribution (ID) and out-of-distribution (OOD) data. Unlike traditional approaches that primarily rely on weight projection and regularization, DiGraP utilizes directional information for gradient adjustments, bridging multi-objective optimization and regularization. The method’s effectiveness is demonstrated across image classification and multi-modal tasks (Visual Question Answering - VQA), outperforming baseline methods in both ID and OOD performance.", "questions": "Could additional techniques be incorporated to enhance DiGraP’s performance on far OOD datasets without sacrificing ID and near OOD performance?\n\nCould the authors provide more insights into how the projection strength parameter adapts dynamically across different layers and tasks, especially in multi-modal settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "DiGraP’s gradient-based approach is innovative, incorporating directional information to handle conflicting objectives in a way that traditional regularization does not.\n\nThe expansion from uni-modal to multi-modal evaluation settings represents the contribution of robust fine-tuning.\n\nComprehensive ablation studies and sensitivity analyses of the hyper-parameters (e.g., projection strength) strengthen the validity of the findings, showing DiGraP’s robustness and effectiveness across different configurations.\n\nThe paper clearly explains the design choices of DiGraP, with detailed descriptions of its components.\n\nVisualizations and tables effectively illustrate the experimental results and the model’s behavior under varying projection strengths.", "weaknesses": "There are many public unimodal and multimodal foundation models, e.g., MAE, CLIP, BEiT3, LLaVa, etc. It is unclear why ResNet50 and PaliGemma are selected as foundation models. The ResNet50 pretrained in a supervised manner on ImageNet can hardly be deemed as foundation models. The PaliGemma is pretrained on a broad mixture of large-scale vision-language tasks. Whether the conclusion of this paper holds across other, more general foundation models, e.g., CLIP or LLaVa, is questionable.\n\nAlthough DiGraP demonstrates improved performance in near OOD settings, its performance on far OOD tasks remains limited compared to other methods. The authors acknowledge this trade-off between ID/near OOD and far OOD robustness, but a deeper investigation into addressing this limitation would enhance the model’s versatility.\n\nThe model’s gradient projection mechanism, while theoretically sound, lacks interpretability in how projection strength decisions impact specific instances.\n\nThe figures in Appendix have too small font.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)