Robust Fine-tuning of Zero-shot Models via Variance Reduction
Authors: Beier Zhu
Jiequan Cui
Hanwang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: image classification
fine-tuning
zero-shot models
Primary area: machine_vision
TL;DR: We propose a sample-wise ensembling technique for fine-tuning zero-shot models that can simultaneously attain the best ID and OOD accuracy.

Abstract: When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains on (0.9 - 3.1 pp) on other distribution shifts
19 benchmarks. Codes are available in https://github.com/BeierZhu/VRF.

Robust Fine-tuning of Zero-shot Models via Variance Reduction
Beier Zhu
Jiequan Cui
Hanwang Zhang
Nanyang Technological University
beier0020e.. ntu. edu hanvangzhang@ntu.edr edu.
Abstract
When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-baset models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that simultaneously attain the best IID and OOD accuracy without the trade-offs. Specifically, we construct Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains (0.9 3.1 pp) on other distribution shifts benchmarks. Codes are available https:::s:. :/sttps:/giiiie //git com/ BeierZhu/VRR
Introduction
To ensure the reliability of machine learning systems, it is essential develop models that generalize to unseen, out-of-distribution environments. Large pre-trained models such as CLIP [20] and ALIGN [IO] have recently shown remarkable robustness against challenging distribution shifts.
However, is widely acknowledged that these improvements robustness are most pronounced in
the zero-shot setting, while conventional fine-tunine
these models often compromises robustness
when compared to zero-shot performance (28 15] 14]. This phenomenon is known as the ID-OOD trade-offs, i.e., improving performance in-distribution (ID) data sometimes lead to decreased
performance on out-of-distribution (OOD) data 12 25
In recent years, ensemble-based models (ESMs) have demonstrated significant success in addressing 14 31.. Specifically, denote the input as x, the zero-shot model as (ylx: fz)) and the fine-tuned model as P(y|x: 0n), existing ESMs typically employ the output- space ensemble (OSE) 14 B1, which outputs P(y|x; foss) = aP( x: fn) + (1 P)P(s) x; 0zs). and the weight-space ensemble (WSE) [28 17 which outputs (P()) Owse) IP(y|x; aln (1 a))2s); where a E [0, 1]. Compared to fine-tuned models, ESMs offer significant accuracy enhance- ments under distribution shift, while maintaining high ID accuracy.
the ID-OOD dilemma 17
However, ESM cannot fully address the ID-OOD trade-offs. In Figure I (a), by varying the mixing coefficient a, we plot the ID-OOD frontier curves (pink line) for the CLIP ViT-B/16 model on
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Avg. Acc shiftedd datasets (OOD)
62
-3.0% AC
Acta ACCa
19 oot
Fine-Tuned Acc Zero-ZeroSonn Sho-SSttt Acc
59
1.2
Endemblinn varing confricent
Best ID accuracy with amo.5 accuracy
Zero Shot
0.8
Rine Tunce
53
VAF
70
10
1.4
1.6
ImageNet Accuracy (ID)
Distance to ZSF set (d(x)
(a)
(b)
Figure I: (a) ID-OOD frontier curves for the CLIP ViT-B/16 model on the ID (ImageNet) and OOD (V, R. A, Sketch) and ObjectNet) datasets by varying the mixing coefficient The ensemble model achieves its best ID and OOD performance at different a values. Our method VRF simultancously attains the best ID and OOD accuracy, outperforming the ensemble by 3.6% on OOD and 1.6% on ID at its optimal performance points.(b) Relationship between the ratio of fine-tuned accuracy to zero-shot accuracy AAcin and the distance to the zero-shot failure set (d(x)). Acce demonstrates monotonic decrease as d(x) increases.
Acea
ImageNet (ID) and five derived distribution-shifeed datasets (OOD): ImageNet-V2 [21]. ImageNet- R [7]. ImageNet-A [9], ImageNet-Sketch (27) and ObjectNet (I]. We find that the ensemble model achieves its optimal ID and OOD performance different a values: the best ID accuracy achieved at = 0.5 and the best OOD accuracy is obtained at 0.3. When the ensemble model reaches its optimal value for OOD, the performance on ID decreases by 3.6% relative its peak. Similarly, when the ensemble model is optimized for ID, the performance on OOD decreases by 1.6% relative its best value the ID-OOD trade-offs still persist for ESMs. This raises natural question:
Can ensemble-bused models simultaneously altain the best ID and OOD accuracy?
In this paper, we affirmatively answer this question by proposing a sample-wist wise ensembling technique. dubbed variance reduction fine-tuning (VRF). This method is motivated by empirical finding illustrated in Fig [ (b). For each sample in the training dataset, the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect its features representation in the fine- tuned model as the zero-shot failure (ZSF) set. We then measure the distance d(x) )) of each test sample the 7SF set. Based this distance, test samples are grouped into bins, and compute the ratio of fine-tuned accuracy zero-shot accuracy: Acce for each bin (implementation details are in Acea
Section C.7 Interestingly, we observe that the ratio Acea monotonically decreases as d(x) increases. Intuitively, the closer a sample is to the ZSF set, the more likely it that the zero-shot model makes incorrect predictions, whereas the fine-tuned model is more likely be accurate, leading to higher Asea ratio. Therefore, Accy
Ace.
the distance assign weights to the models: smaller d(xx) results in higher weight for the fine-tuned model, and vice versa.
As depicted by the orange diamond in Fig. T (a), by leveraging the sample-wise weights, our VRF simultaneously attains the best ID and OOD accuracy. In Section [] we show that on a variety of different models and tasks, our VRF approach consistently outperforms the existing fine-tuning and ensembling methods, including linear probing, end-to-end fine-tuning, LP-FT [1]. OSE and WSE 28). specific, on ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 2.0 pp over the ensemble baselines while maintaining increasing ID accuracy. Furthermore, Section 4 we justify our approach by demonstrating that it effectively minimizes the variance of the ensemble models, resulting in reduced residual error.

---

2 Related Work
Mitigating ID-OOD trade-offs. Improving performance on in-distribution data can sometimes lead to decrease in performance out-of-disrribttion data, and vice versa. This phenomenon is known as the ID-OOD trade-offs. Xie et al. 22 leverage auxiliary information as outputs of auxiliary tasks to pre-train a model to reduce OOD error. Khani and Liang [1] show that self-training on large amounts of unlabeled data can mitigate such trade-offs by removing spurious features. Tripuraneni et al. [25] tackle this problem by learning representations that robust across diverse tasks. However, these methods usually necessitate additional unlabeled data or auxiliary information. In contrast, our VRF is straightforward variation of fine-tuning that does not require any extra data.
Robust fine-tuning of zero-shot models. Vision-language models like CLIP (0 have demonstrated outstanding improvements in robustness. It is commonly acknowledged that conventional fine- tuning methods often compromise robustness when compared to zero-shot performance. Therefore, enhancing downstream robustness has been the focus of subsequent works [15]
28
5
19 6
30)
Kumar et [55 show that a two-process of linear probing followed by full fine-tuning can alleviate feature distortion, leading to stronger OOD performance without sacrificing ID accuracy. Wortsman et al. 28 propose a method of weight interpolation between the zero-shot and the fine-tuned models to improve both ID and OOD accuracy. Goyal et al. [5] demonstrate that mimicking the contrastive pre-training objectives to fine-tune the zero-shot models outperforms tuning via the traditional supervised cross-entropy loss. However, the ID-OOD trade-offs are still observed with these methods. In contrast, our method VRF can simultaneously achieve the best ID and OOD accuracy.
3 Methods
3.1 Set Up
Task:
Consider classification setting where the goal is to map instance E X to label y Y = [K] We are provided with a zero-shot model (:; f2s), a downstream dataset D = fx,, Vi LV Ji=+ and fine-tuned model f:; 0%) which is trained D. Below, we outline the implementation of the zero-shot and fine-tuned models:
Zero-shot models (ZS): We investigate CLIP models 20 as our zero-shot models. CLIP models are pre-trained using image-text pairs {(11, t1 (x tn)) from the Internet. The objective of the CLJP models to train a visual encoder t and a text encoder o such that the cosine similarity (x,), (t,tt) > is maximized relative unmatched pairs. CLIP models perform zero-shot inference for K classes by matching x with potential class names {ci,...,ck' Concretely, by extending the class name {c&} prompt "t, =a photo of a {cx)", the zero-shot model outputs the score (logit) for class k as f(x; 0zs): =< Pv(x), (t(t)) >, The predicted probabilities can be calculated using the softmax function, i.e. P(y|x: Oz)) voftmax(f(x: )))) The model outputs the label as pred (fx; 0zs)) = argmax, f(x; 025)s
Linear classifiers (L.C): We learn a linear classifier on top of the visual embedding (x) while freezing the visual encoder Φy,, The parameters of the linear classifier are optimized to minimize the cross-entropy loss on D.
End-to-end fine-tuning (E2E-FT): We update both the linear classifier and the visual encoder by minimizing the cross-entropy loss on D.
Linear probing then full fine-tuning [5] (LP-FT):: We employ two-phase fine-tuning approach: initially training a linear classifier, followed by full fine-tuning starting from the solution derived from training the linear classifier.
Output-space ensemble (OSE): We perform linear interpolation of the outputs between zero-shot model and fine-tuned model (e.g., E2E-FT or LP-FT):
( Ooze) = of xx; 0R) + (1 ())x fz), where a E [0. 1]
(1)
Weight-space ensemble 28 (WSE). We combine the weights through linear interpolation
between a zero-shot model and fine-tuned model:
P(y|x (wee) P(y|x; cbtr + (1 a))ss), where e [0,1]
(2)
3

---

Algorithm 1 Variation Reduction Fine-tuning
1: Given: Training dataset DD, zero-shot model fzs and a fine-tuned model fn.
2: Build zero-shot failure set V using Eq.
Step 1: Identification
3: Inference Stage:
4: Given test sample compute its feature representation zero-shot prediction (ulxx) and fine-tuned model prediction (r)(y/)).
5: Compute the k-NN distance to as d(x) using Eq. 4
>> Step 2: Distance Calculation
6: Compute the weight (xx) using Eq. (6).
7: Return P (y/(() using Eq.
Step 3: Sample-Wise Ensembling
3.2 Variance Reduction Fine-tuning
We now present proposed method, VRF, which consists of three steps. First, before the inference stage, we collect the Zero-Shot Failure (ZSF) set. Second, for given test sample, we calculate its distance the ZSF set. Third, we assign weights combine predictions from the zero-shot and fine-tuned models based on this distance.
Step (Identification). For each x; in the training dataset D, if the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect feature representation v; = (x(x, 0+) from the fine-tuned model to form the zero-shot failure set V. Specifically, V is defined as:
V {v; Yt pred(fn (x:)) and yı * (xx)(f2((x.))-
Here, f2s()) and fit(-) are used to denote f(=:02)) and (t). respectively, for simplicity.
Step (Distance Calculation). The key empirical observation underpinning VRF is that in the vicinity of the ZSF set, test sample typically exhibits lower zero-shot accuracy (Acez) and higher fine-tuned accuracy (Accæ). Consequently, the Ace.,
Acce ratio demonstrates a monotonic decrease as the distance from the sample to the 7SF set increases. In this paper, we adopt non-parametric density estimation using nearest neighbors [24] to measure the distance of a test sample to the dataset V. Specifically, during inference, we derive the feature representation of a test sample x, and compute the f2 distances ||v vill2 w.r.t. v, V. We reorder V according to the increasing E distance and denote the ordered set in sequence as V (V(1). V(2),.... V(vD). The distance of x to V is defined as the le distance to the k-tth nearest neighbor (K-NN), i.e.,
d(x; V, k)
||v Va2yl2..
If there is no ambiguity, we use d(x) denote d( x: V, k)) for readability. Since the features in CL.IP models are f2 normalized, d(xx) are bounded between [0, 2].
Step 3 (Sample-Wis Wise Ensembling). We implement
sample-wise sptce ensembling in the form:
Accuarcy
(u)) w(x ( - (1-w(x)) - (y)x), (5)
0.70
where (xx) (). We use the distance ZSF set d(x) to determine the weight w. As shown by the blue line in
0.2-
2.0
Ace. Fig 2 smaller value of d(x) corresponds to larger Ace. ratio, and vice versa. Therefore, we set the weight w to be inversely proportional d(x). Given that is bounded between 0 and 1, we employ a sigmoid function (-) as:
6.55
0.8-
0.8
0.9
1.1
1.3
1.5
Dieance to 257
est
08.11
Figure 2: Relationship between
Asig and Acea
((x) = (d(x) - a)/b),
(6)
the weight (xx).
where b >> 0 are two hyper-parametts parameters sweeped using accuracy on ID validation set. We visualize the weight curve in green on Fig 2 setting a = 1.5 and b = 0.6. We summarize the whole process in Algorithm
[
4 Justiffcation
We now prove that our VRF can effectively reduce the variance of the combined model, resulting in lower errors compared ensembling using constant mixing coefficient.

---

4.1 Background
The outputs of well trained classifier are expected to approximate the a posterior class distribution. Apart from the irreducible error (Bayes error), the residual error of classifier can be broken down into bias and variance components. In specific. for a test sample x, the probability output of classifier parameterized by can be expressed as:
P(y 0) P(y + By 7
residual
where P(y|x) denotes the true posterior distribution, Bg is the label bias of P(y) x; 0) which is independent to the input x, and 77y (>)) is related to the given input x. In this study, we primarily attribute the residual error the variance term (i.e., By 0), the label bias problem in foundation
models has been effectively addressed by Zhu et al. 31 Tumer et al. [26] have proven that the expected residual error E is given by:
(2)
E
(8)
where is a constant factor related to the derivative of the true posterior distribution and is independent of the trained model, and Vnn (x) the variance.
4.2 Variance Reduction Fine-tuning Leads to Lower Residual Error
Let us shift our focus to the effects of combining the zero-shot and fine-tuned models. Let 92s(.) and gte ()) be two functions that produce weights for ensembling the models. Subject the constraint that Ozs (x) + Ott (x) 1, the residual error of the combined classifier obtained by:
P (vlx)) Izs (x (v)))) + gh (x)))
P(y]x) + (x)
(x)(x) + 9t(()) (x)))),
(9)
Tee (x))
where we omit the subscript of for readability. The variance of Muv (x) can be expressed as:
V(nvrt (x)² V[)z((x) + 9n (x)² " ([n(x)).
(10)
Here, we assume the residual errors are independent following the assumption of the previous studies of CLIP fine-tuning [14 31. We further explore the of correlated residual errors in Section B According to Eq. (8). the reduction in variance can readily translated into a reduction in error rates. To obtain the smallest variance V(pwr(
minimize Eq. (IO) using Lagrange multiplier to enforce the constraint that (x)(x) + (x) = 1, and obtain the optimal weight function as:
V (x))
E
Eitt Ezz
AcC₁
ght (x(
(1
(11)
V V/ (x)) + V (x)
Fezs
Fxt
Acezs
Since Areg we design the weighting function (x) = (xx) x d(x)- -1 in Eq. (6).
Acca x d(x)- (a smaller distance dxxx) is associated with larger
Acca as shown in Fig. Acea
2b.
5 Experiments
5.1 Experimental Setup
Datasets with distribution shifts. We provide the results for ImageNet 33 and its five derived
distribution shifts: (1) ImageNet-V2 V2 (IN-V2) (21): Test images sampled after decade of the original
ImageNet. (2) ImageNet-R (IN-R) 77 Contains renditions (e.g., art, cartoons, graffiti).
(3) ImageNet Sketch (IN-Sketch) 277 Consists of sketches rather than natural photos. (4) ImageNet-A (IN-A) :
Collects real-world images that are misclassified by ResNet models. (5) ObjectNet (I) test set featuring objects with diverse backgrounds, rotations, and imaging viewpoints. We extend our
analysis to include a standard distribution shift benchmark [5 14 4]: CIFAR-I0 - STL-10, where the ID is CIFAR-10 10 13) and the OOD is STL-10 2 We removed the "monkey" class from STL-10, as it does not exist in CIFAR-10. In addition, we also consider subpopulation shifts, where the ID data contains a few sub-categories, and the OOD data comprises different sub-categories within the
5

---

Table Accuracy various methods on ImageNet and derived distribution shifts for CLIP VIT-
Distribution shifts
Avg shifts
Method
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
Zero-shot 20
63.3
55.9
42.3
31.5
69.3
43.5
48.5
Linear classifier
20
75.4
63.4
38.8
26.1
58.7
41.5
45.7
E2E-FT
28
76.2
64.2
38.7
21.0
57.1
40.1
44.2
Weight-space ensemble
77.9
67.2
45.1
66.4
50.5
+ Output-space ensemble
77.3
66.0
44.2
27.1
68.4
44.4
50.0
+ VRF (ours)
77.6
66.7
47.0
29.2
70.9
46.3
52.0
+0.3
+0.7
+2.8
+2.1
+2.5
+1.9
+2.0
LP-FT [15]
76.9
64.8
39.9
25.7
69.9
42.6
48.6
Weight-space Ensemble
28
78.0
67.0
44.8
31.2
65.8
46.1
51.0
Output-space Ensemble
77.8
66.3
44.0
29.5
66.2
VRF (ours)
77.8
66.7
46.1
31.0
70.0
46.3
51.8
+0.0
+0,4
+2.1
+1.5
+3.8
+0.8
+1.5
Table 2: Accuracy various methods on ImageNet and derived distribution shifts for CLIP VIT-B/16.
Distribution shifts IN-A
Avg shifts
Method
IN-V2
IN-Sketch
IN-R
ObjectNet
Zero-shot 20
68.3
61.9
48.3
50.1
77.6
54.2
58.4
Linear classifier 20
79.3
69.1
44.8
44.3
66.7
51.1
55.2
E2F-FT 28
81.3
70.6
45.1
36.6
65.6
50.5
53.7
+ Weight-space ensemble
82.5
51.6
47.6
75.1
55.7
60.6
+ Output-space ensemble
82.2
72.0
50.6
46.8
76.7
54.9
60.2
+ VRF (ours)
82.3
72.1
52.9
48.4
78.7
56.4
61.8
+0.1
+0.1
+2.3
+1.6
+2.0
+1.5
+1.6
LP-FT 15
81.5
70.7
46.7
41.4
66.4
52.4
55.5
Weight-space ensemble
28
82.4
73.0
51.5
50.6
74.2
56.6
61.2
+ Output-space ensemble
82.1
50.9
50.9
74.9
60.9
+ VRF (ours)
82.1
72.3
52.9
51.2
78.8
57.2
62.4
+0.0
+0.0
+2.0
+0.3
+3.9
+1.5
+1.5
same parent category. Following [5 [4],
we adopt Eo dataset (23), which aims to categorize images into of 30 entity categories, such as "vehicle" and "insect".
Baselines. We adopt two models: CLIP VIT-B/32 and larger ViT-B/16 from OpenAl 20 The default model used in ablation studies is the CLIP ViT-B/16. In addition to the zero-shot models, we compare our approach against five standard methods for adapting pre-trained models: (1) linear classifier [20). (2) E2E-FT, (3) LP-FT [5 (4) OSE, and (5) WSE (28). The descriptions of these
methods have been included in Section
Implementation details. When fine-tuning E2F-FT models, we adhere to Wortsman et al. (28). employing the default PyTorch AdamW optimizer for 10 epochs with weight decay of 0.1 and cosine-annealing learning rate schedule with 500 warm-up steps. Unless specified, we use learning rate of 10 gradient clipping at norm 1. When fine-tuning LP-FT, we first adopt the settings
of Wortsman et al. (28 to train the linear classifier, then full fine-tune the models at learning
rate of x 10-5 For efficiently performing k-NN search, we use Faiss library [1I]. Denote the
size of the ZSF set is [V], we scale the &o according to percentage p% of the sample set, where =ffoorr(pp (p% [V]). )). In this paper, is set to 0.1%, a value consistent with the default setting proposed by Sun et al. 24. Note that all the hyperparameters, egg., a, are searched using the accuracy on the in-distribution (ID) validation set. Derived distribution shift datasets are only for evaluation and not for hyperparameter sweeps. See Appendix C.I for the details of experimental details.

---

Table 3: Accuracy various methods on CIFAR-I0 STL-10 and Entity-30.
CIFAR
STL
Entity-30
CIFAR
STL
Entity-30
Method
Method
ID
OOD
ID
OOD
ID
OOD
ID
OOD
Zero-shot 20
88.3
97.1
65.2
66.5
Zero-shot 20
90.1
98.4
68.3
68.2
Linear classifier
95.0
96.6
93.3
68.1
Linear classifier
95.8
97.7
95.3
69.6
E2E-FT (28
97.9
93.5
94.4
65.1
E2E-FT (28
98.6
96.1
96.9
68,2
WSE 28
95.7
94.6
68.8
WSE 28
97.8
97.2
71.9
+ OSE
97.9
95.9
94.4
66.4
+ OSE
98.6
96.6
97.0
71.5
+ VRF (ours)
97.8
97.3
94.5
69.5
+ VRF (ours)
98.6
98.4
97.0
72.7
A
-0.1
+1.4
+0.1
+3.1
A
+0.0
+1.8
+0.0
+1.2
LP-FT [5
97.9
95.0
94.6
67.7
LP-FT [SS
98.5
96.3
96.9
68.8
WSE 28
98.1
96.4
94.8
68.8
WSE 28
98.7
97.9
97.3
72.1
OSE
98.1
94.7
97.7
97.2
VRF (ours)
98.1
97.5
94.8
70.1
+ VRF (ours)
98.6
98.6
97.4
72.9
+0.0
+1.1
+0.1
+1.6
+0.0
+0,9
+0.2
+1.1
(a) CLIP VIT-B/32
(b) CLIP ViT-B/16
CIFAR- 10 -STL-10
Entiy-50
(000)
D000
Romrs alvurre Ios accamo oVC)
0:40
Bor 0:0 anc aie
0.85
Rec
20
52
26
92
6.3 0.6
0.5
83 72
77
02
07
05 0.8
CIFRR-10 (ID)
Dolanne 1000
(1.1)
(2.2)
(b.1)
(b.2)
Figure 3: ID-OOD frontier curves by varying the mixing coefficient a and Aces curves for the CLIP Accas ViT-B/16, (a) CIFAR-10 (ID) and STL-10 (OOD) results. (b) Entity-30 results.
5.2 Results
ImageNet and its five shifted distribution results. In Table | and we report the ID-OOD accuracies of fine-tuning baselines for CLIP ViT-32 and CLIP ViT-16 models, respectively. OSE and WSE, we choose the mixing coefficient with the highest ID validation accuracy. To enhance clarity in the results, we denote the improvement over OSE as in Tables and We observe that our VRF boosts the accuracy of fine-tuned models, including ensembling baseline models, across five ImageNet distribution shifted datasets, while maintaining or improving the ImageNet in-distribution performance. For instance, in Table when ensembling with the E2E-FT model, our VRF outperforms the OSE model by 2.0% on distribution shifts while increasing the ID accuracy by 0.3%. Compared WSE models, VRF achieves delta of 1.2% distribution shifts, while maintaining ID performance within 0.2%. as shown in E2E-FT part Table
CIFAR-10 - STL-10 and Entity-30 results. We report the accuracy of various methods in Table (a.b). We note that fine-tuning baselines can enhance the accuracy on CIFAR-10 compared to zero-shot models. However, this improvement comes at the expense of reduced accuracy on 100 For instance, E2E-FT Ieads to a decrease of approximately 3.6% in STL-10 10 accuracy, as shown in Table a). Previous ensemble methods performance still lags behind the zero-shot performance, e.g., In Table 3(b, the accuracy of E2E-FT + WSE is 97.8% whereas the zero-shot performance is 98.4%. In contrast, our VRF simultaneously improves accuracy on both CIFAR-10 and STL-10. Similarly, for Entity-30, our VRF can further improvement the OOD performance when compared to WSE and OSE methods.
mitigate the degradation to some extent, but the STL-10
In addition, we plot the ID-OOD frontier curves in Figure (a. &&b1), respectively. Similar to results on ImageNet (Figure aa)), the ensemble model achieves its best ID and OOD performances different values. For instance, on the CIFAR-10 benchmark, when the ensemble model attains its optimal ID value at 0.7, the OOD performance decreases by 2.0% relative to its peak.
7

---

Table 4: Results of VRF for linear-probed models using CLIP ViT-B/16 models.
ImageNet
CIFAR-10
Entity-30
Method
ID
OOD
ID
0OD
ID
OOD
Zero-shot classifier 20
68.3
58.4
90.1
98.4
68.3
68.2
I Linear classifier
79.3
55.2
95.8
97.7
95.3
69.6
WSE/OSE
79.9
57.8
95.8
97.7
95.5
70.5
VRF (ours)
79.8
58.5
95.8
98.4
95.4
71.4
Conversely, when the optimal OOD value is reached at a 0.3, the performance on ID diminishes by 2.7% from its best. In contrast, our VRF simultaneously attains the ID and OOD performance.
We also analyze the relation between the ratio Acea and d(x) in Figure (a.2&b.2). Consistent with the findings from ImageNet (Figure (b)), we observe that the ratio decreases as d(xx) increases, which further supports our design of assigning higher weight to fine-tuned models if d(x) is smaller.
Aces
5.3 Further Analysis and Ablation Studies
VRF for linear-probed models. A drawback of the proposed method is its doubled inference and storage cost compared to WSE and other single-model robust fine-tuning methods. To address concerns regarding space-time complexity, present the results in Table 4 We also compare with output-space ensembling, since the model is linear, it is equivalent weight-space ensembling. We also compare it with output-space ensembling, which, given the linear nature of the model, is equivalent to weight-space ensembling. Consistent with the conclusions drawn from fully fine-tuned models, our VRF method further improves OOD performance while maintaining comparable ID performance to OSE/WSE ensembling.
apply our VRF method to linear-probed models and
Using ZSF set \/ or entire training set D? In Step 1 of our VRF, we define the zero-shot failure set V and use it to compute distances. We aim to find out whether using the entire training set D offers comparable performance. In Figure we plot the Acce curves and report both ID
Accuracy Accuracy
ZSF set V
FT data
Ace.
and OOD accuracy using the two sets. We observe that the ratio curve using D does not exhibit a monotonic trend with d(x): it initially increases and then decreases as d(x) increases. Furthermore, the ratio Acin using D is less Aces informative when d(xx) is smaller than 1.2, as the curve relatively remains flat. As the zero-shot models rately predict a large proportion of the II) data (recall that the sero accuracy is 68.3%), smaller distance entire training set DD does not reliably indicate whether the fine-tuned model can make more accurate predictions. In comparison, our ZSF set contains only the samples where zero-shot models fail but fine-tuned models succeed. When a sample is close to V, it is more likely that the accuracy ratio will be high. Consequently, the performances using D clearly outperformed by those using V.
Apc.
Al data
82.2
61.0
06-
2SF
82.3
61.8
0.8
1.0
1.2
1.4
1.5
Distance to different datasets d(x).
accu-
Figure 4: ZSF set V vs. all data D
Comparison with selective prediction using OOD detector. sim- ple approach to address the ID-OOD trade-offs is to use an OOD de- tector for selective prediction. The OOD detector is a binary classifier Gx(-) to decide whether a sample is ID or OOD based on threshold X. For a test sample, predictions are made with the fine-tuned model if classified as ID, and with the zero-shot model otherwise:
Table 5: Selective prediction using OOD detector.
Method
ID
OOD
MSP X
81.5
57.3
Energy 18
81.0
57.6
MD 16
81.0
57.7
(x) = [x(())
ix) = ID
KNN 24
80.8
58.4
(12)
(xx(x)
Gx OOD,
RMD
22
81.1
58.4
ID
if) 2
VRF (ours)
82.3
61.8
Gx(x)
(13)
(OOD
S()) < X,
8

---

0.21
0.7
Accurscy
0.63
Imagekat P Imageher DCB
Aog 56-
0.3
Treuhh
108 bure
0.1
70
5,001%
InageNet Accuracy
(a)
(b)
(c)
Figure 5: (a) Averaged weight [x)] on different datasets. (b) VRF based on logit-space
ensembling. (c) Comparison with the effect of different in the k-NN distance.
Imugstut Accuracy (84
Imagelte 0OD Rctur
0:75
ami
01.7
00.5
dml
19.
Breny, 0+3.3
Liser: p=1.5 bed
00.5
Sgnsid, a-1.6. budLE
(a)
(b)
(c)
Figure 6: (a) Effect of and b on ImageNet ID accuracy. (b) Effect of a and b on ImageNet OOD accuracy. (c) Other designs of (x)), hyper-parameters are searched on validation set.
where instances with higher scores S(x) are classified as ID and vice versa. À is typically chosen to achieve achieve a 95% true positive rate for samples. We report the results of several im- plementations of 5(x) in Table 5 (Details are in Section C.6). We note that selective prediction achieves comparable ID performance to E2E-FT models and similar OOD performance to zero-shot models. However, its accuracy still falls significantly short of VRF. This is because traditional OOD detectors are designed for scenarios where the OOD data have a completely disjoint label space from the ID data, i.e, VOOD Vio D. However, in setup, the zero-shot models show predictive power on ID data, and the fine-tuned models are effective on OOD data. Making binary selections may overlook the complementary knowledge from the other model. Instead, weight function (xx) intelligently selects the contribution of each model based on the distance to the ZSF Another reason why our method outperforms selective prediction is the effective use of the ZSF set, as illustrated in Figure Directly using all ID data as traditional OOD detectors (e.g.. kNN and MD) leads to weak correlation between the accuracy ratio and the distance d(x) (or score S(x))
Examination of the averaged weight for ID and OOD test sets. Figure 5 a) shows the average weight (Ex (Ex[w(()) (x))) of the E2E-FT model in ensembling for both ID and OOD test sets. As expected, higher average weights are observed the ID test set, as the fine-tuned models excel in such domain.
Logits-based ensembling. In this paper, we implement OSE by linearly interpolating the probabilities of the two models. Another common strategy for ensembling, known Logits-Space Ensembling (LSE), involves interpolating in the logits space: [x; 0ss)) af (x: 0n) + (1 /x)(xx Ozs). We aim to investigate whether our VRF can enhance the robustness LSE without compromising the accuracy. The results depicted in Figure ()) confirm that our VRF can indeed generalize to LSE
Effect of & in k-NN distance. In Figure cc) compute & floor(p [ vary p across the range (0.0001%) %, 0.01%, 0.05%, 0.1%, 10%, 50%). We note two observations: (1) Varying A slightly less than 0.1 %. (2) The OOD accuracy declines as
affect the ID performance: the fluctuations increases, but the degradation is very slight for relative small values of (e.g., when
0.01%. the decline smaller than 0.2%). In our implementation, we use the k-th nearest sample instead of the nearest one to reduce the potential impact of label noise. If the nearest sample is mislabeled, the distance may be unreliable. The k-th sample, being higher-density region, offers more stable distance estimates with lower variance, as it lies between the (A- 1)-th and (k: + 1)-th samples. This makes the measure more robust to outliers. Additionally, prior research 24 shows that using the k-th nearest distance improves density estimation, which we adopt here.

---

Table 6: Accuracy of designs of w ImageNet.
Smallest d(x)
Largest d(x)
Design of w
ID
OOD
Binary
81.3
58.4
Linear
82.3
61.7
Sigmoid 82.3
61.8
Figure 7: Visualization the samples with the smallest/largest d(x).
Inference speed of computing k-nearest neighbor distance. Thanks to the Faiss library [II] the k-NN search can be ef- ficiently implemented. When evaluated on ImageNet bench- marks using CLIP ViT/B-16 features, the inference speed is approximately 1.8 milliseconds per-image which does not sig- nificantly improve the inference time. In Figure we further present the per-image inference speed of the k-nearest neighbor distance computation for various values. The inference speed less than 2 ms when k < 512.
Inference Time (ms)
1.9
logz(k)
Figure 8: Inference speed (per- image) using different k.
Effect of and b in W. We demonstrate the effect of and in Figure (a&b). We highlight three trends: (1) ID performance
peaks at b 2 0.6 across different values of a. (2) OOD performance often improves as b increases across different values a. (3) When sufficiently large, e.g., > 10 for and b > 2 for OOD, has marginal effect on the performance of ID and OOD. In Appendix C.2
further plot the
trade-offs when tuning a and
Other designs of (xx). We further explore alternative designs of (x) beyond the sigmoid format in Eq. (6):
Binary weight: Wbinary (x) 11 [d(x) < a). where E [0, 2] and [[] is the indicator function.
Linear weight: Wiinear (x) = clamp 0.1 -b- (d(x) - a)), where E [0, 2]. b > 0 and
clamp (-) rectifies the weight within [0. 1].
We report the results ImageNet in Table 6 and plot the weight curves with the value of hyper- parameters in Figure (cc. We find that the Linear and the Sigmoid weights show comparable performance and assign similar values of w around d = 1.5.
Visualization of samples according d(x).. In Figure we randomly sample testing images with the top-100 100 smallest d(x) values in the range [0.40, 0.62 and the top-100 largest d(x) values in the range [1.59, 1.63). Interesting, we observe that: (1) Samples with the smallest d(x) predominantly consist of fine-grad fine-grained species, e.g., "Triturus vulgaris". "eft" and "lycaenid", where the fine-tuned models possess domain-specifie knowledge, which is often lacking in the zero-shot models. (2) Images with the largest d(x) exhibit styles different from those of the fine-tuning samples, including tattoos, cartoons, and sketches, contrasting with the photos typically seen fine- fin-tuning. Zero-shot models are more skilled in non-real photo styles compared to fine-tuned models.
6 Impact, limitations and conclusion
Impact. Zero-shot models inherit the weaknesses from pre-training data to downstream tasks, such noisy and malicious samples. Our VRF might propagate the negative impact.
Limitations. Our approach is built on the premise that zero-shot models posses predictive capabilities for downstream tasks. However, if the pre-training knowledge significantly differs from the down- stream tasks, our algorithm might fail, which is also an open problem in transfer learning. addition, the proposed method doubles inference cost compared to WSE and other fine-tuning methods, runs both the zero-shot and fine-tuned models. However, this overhead can be mitigated by parallel execution.
Conclusion. Inspired by the ID-OOD trade-offs in ensemble-be based fine-tuning, we propose VRF to simultancously optimize the best ID and OOD accuracy. By leveraging the distance to the ZSF set. assign sample-wie wise weights the two models. Despite its simplicity, our VRF demonstrates strong empirical performance, offering a promising technique for solving ID-OOD trade-offs.
10

---

Acknowledgments
This research supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISGG-PPDD-2221001-002
References
[1] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectee: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurlPS, 2019.
[2] Adam Coates, Andrew Ng. and Honglak Lec. An analysis of single-layer networks in unsuper- vised feature learning. In AISTATS, 2011.
[3] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[4] Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In ICLR, 2018.
[5] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, 2023.
[6] Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. Anchor-base -based robust finetuning vision-language models. arXiv preprint arXiv:2404.06244, 2024.
[7] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR, 2021.
[8] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -uitt--disttibution examples in neural networks. In ICLR, 2016.
[9] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[10] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML., 2021.
[11] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactonns on Big Data, 7(3):535-547, 2019.
[12] Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 196-205, 2021.
[13] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
[14] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI. PMLR, 2022.
[15] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022.
[16] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018
[17] Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out----diittiiutiio generalization. In ICLR, 2023.
11

---

[18] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of- detection. In NeurIPS, 2020.
[19] Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning zero-shot models using random text guidance. arXiv preprint arXiv:2404.00886 2024.
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[21] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[22] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Laksh- minarayanan. simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2 wrxiv:2/06.0902. /06.09022, 2021.
[23] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopu- lation shift. ArXiv, abs/2008 abs/2008.04859, 2020.
[24] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
[25] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In NeurIPS, 2020.
[26] Kagan Tumer and Joydeep Ghosh. Analysis of decision boundaries in linearly combined neural classifiers. Pattern recognition, 29(2):341-348, 1996.
[27] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. NeurIPS, 2019.
[28] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022.
[29] Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang In-n-out: Pre-training and self-training using auxiliary information for disti--iist robustness. In ICLR, 2021.
[30] Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang. Debiased fine-tuning for vision-language models by prompt regularization. In AAAI, 2023.
[31] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS, 2023.
12

---

Contents
Introduction
2
Related Work
3
Methods
3.1
Set Up
3.2
Variance Reduction Fine-tuning
4
Jusiiication
Background
4.2
Variance Reduction fine-tuning Leads to Lower Residual Error
5
Experiments
5.1
Experimental Setup
5.2
Results
5.3
Further Analysis and Ablation Studies
6
Impact, limitations and conclusion
10
Licenses
13
Analysis in the Presence of Correlated Errors
14
Additional Experimental Details and Results
14
Additional Experimental Details
14
C.2
Plotting ID-OOD Trade-Offs of VRF
15
Additional Results for OSE, WSE VRF with Varying
15
Optimal Performance Searched on Test Sets
15
C.S
Combining VRF with Other Robust Fine- Tuning Methods
15
C.6
Additional Results for Selective Prediction Using OOD Detectors
15
C.7 Curves of Acca for ImageNet and its Five Distribution Shifted Datasets
16
Ace.
A
Licenses
All the datasets we considered are publicly available, we list their licences and URLs as follows:
CIFAR-10 13:
MIT License,
html
STL-10 2 Non-commercial, http://://////////oo///////////a
Entity-30 23
Non-commercial, http://////gggoooo/o///a/aa/aaaa Banaa//
ImageNet 3 Non-commercial, htcommmrril.ttt:///igggeee..rggo
IN-V2 21: MIT Mit.e///..ttrc////ccooooo////otr
IN-R MIT Lec.n./.hhgeeehhhhhocco////e/eer
IN-Sketch [27 MIT http://..wwgo..g........o/oaaaaa
13

---

Table 7: Syyper-parmmetee a and b for different backbones and datasets.
ImageNet
CIFAR-10
Entity-30
Backbone
b
b
CLIP B/32
1.5
0.6
0.3
0.3
1.1
0.6
CLIP ViT-B/16
1.5
0.5
0.3
0.3
0.6
IN-A 99: MIT License,
ObjectNet []]
Creative Commons Attribution 4.0, n.0.itttss//bbeeetteee.
B
Analysis in the Presence of Correlated Errors
Our assumption of independent residual errors based on the previous studies (28] (Section 5), where an empirical phenomena is observed: the zero-shot and the fine-tuned models produce diverse predictions. In general (i.e., the fine-tuned models are initialized from the zero-shot models), we cannot assume that the errors in the zero-shot and fine-tuned models are totally uncorrelated. (25(x) (x) and (x),,, the variance of ()v((x) can be
Let Cines (x)) be the covariance between Yas expressed as:
(x)] (x)² (x)= + grt (x)² (x)r + 2. (xs(x) (x) ()) (14)
Maintaining that 92s (x) + (x) 1, the optimal weight gic (x) to minimize Eq. 11 becomes:
V/n (xx) C( (x), Mh (x)]
gf (1
(15)
V[nza( C 175 (x), (x)]
Recall that we are interested using the dis- tance to ZSF set, i.e., d(x), to surrogate of (x). To understand the relationship between d(x)) and Ih (x), we first group test samples in ImageNet and its five distribution shifted datasets into bins based the value of d(x). We then compute the averaged 9h (x) for each bin and plot the use temperature scaling [14] to calibrate the zero- shot and fine-tuned models over the validation set. Afterwards, we define the true distribution P(yl)) xx) one-hot vector, where the value of 1 corresponds to the true label for a given input
0.525
9t
0.500
0.475
0.425
relationship in Figure 9 In specific,
0.400
0.375
0.6
0.8
1.0
1.2
1.4
Distance ZSF set (dix))
Figure 9: Relationship between gft (x) and d(x)).
x. We then calculate 7/. (x.x)) (x) = ((yy|x x; 0n.) [Pyy|x) and Tes x
P(y]| 0zx) PP(y)x). Finally, we compute the average o (x) for each bin as shown in Figure Interestingly, we observe the similar
trend in Figure (b): the weight for fine-tuned models decreases as d(x) increases. This phenomena
indicates that our weighting function wxx derived under the assumption of independent errors is
also valid in the presence of correlated errors.
C Additional Experimental Details and Results
C.1 Additional Experimental Details
For CLIP VIT-32 based E2E-FT and LP-FT models, we use a batch size of 512. For CLIP ViT-16 based E2E-FT, directly download the fine-tuned models from Wortsman et al. [28 The batch size for training CLIP VIT-16 based LP-FT models is set to 384, which is the largest batch size that fits into 2 A6000 GPUs. When performing linear probing, learning rate of 0.1 for all experiments. The mixing coefficient a for OSE and WSE are searched over [0, 0.1, 0.2, ... 0.9, 1.0). The values of and for VRF reported in Table
batch of 512 and the initial
bittps://dring
14

---

STL-10 (000)
000
A00 carig corticers
Rea Bend
12
imageNett Accuracy (I0)
CIFAR-10 (I))
10
Figure 10: ID-OOD scatters for VRF ImageNet and its vari- ants.
Figure 11: ID-OOD frontier Figure 12: ID-OOD frontier curves for OSE and WSE for curves for OSE and WSE for Entity-30.
CIFAR-10 - STL10.
Table 8: Accuracy of E2E-FT based OSE on ImageNet and derived distribution shifts for various values of the mixing coefficient a. Results shown for CLIP ViT-B/16.
Distribution shifts
Avg shifts
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
0.0
68.3
61.9
48.3
50.1
77.6
54.2
58.4 4
0.1
71.6
64,8
49.9
50.7
78.5
55.5
59.9
0.2
75.4
67.5
51.6
51.2
79.2
56.3
61.1
0.3
78.6
69.7
52.3
50.9
79.3
56.8
61.8
81.0
71.3
52.0
49.6
78.6
56.6
61.6
0.5
82.2
72.0
50.6
46.8
76.7
54.9
60.2
0.6
82.1
71.6
48.7
42.9
73.8
53.3
58.1
0.7
81.8
71.2
47.3
40.5
71.1
52.2
56.4
0.8
81.6
70.9
46.3
38.5
68.3
51.4
55.1
0.9
81.5
70.7
45.6
37.4
66.6
50.8
54.2
1.0
81.3
70.6
45.1
36.6
65.6
50.5
53.7
C.2 Plotting ID-OOD Trade-Offs of VRF
In Figure 10 we present ID-OOD scatter plot over wide range of a and b values on ImageNet and its five variants using CLIP ViT-B/16. Specifically, varies from 1.2 1.8, while ranges from 0.5 to 1.0. Our method consistently achieves better ID-OOD trade-offs, as indicated by its points lying outside the OSE curve (represented by the magenta curve) across different config configurations.
C.3 Additional Results for OSE, WSE and VRF with Varying Hlper-Parameters
Results for all mixing coefficient for OSE and WSE are available in Table Table respectively, Results for values of and are available Table I0 In addition, we plot the ID-OOD trade-off' curves for OSE and WSE the CIFAR-10 and Entity-30 datasets in Figures and 12 respectively.
C.4 Optimal Performance Searched Test Sets
We have conducted additional experiments where we optimized the hyperparameters for each test set of the ImageNet benckmarks. The results summarized in Table II
C.5 Combining VRF with Other Robust Fine- Sine-Tuning Methods
Our VRF framework is orthogonal and complementary to existing fine-tuned models. To demonstrate this, we integrated FLYP (5) into our VRF framework. The results in Table [2] show that VRF improves OSE's performance under distribution shift by 1.1% without compromising in-distribution (ID) performance.
C.6 Additional Results for Selective Prediction Using 0OD Detectors
We provide the breakdown performance for selective prediction using OOD detectors in Table 13
15

---

Table Accuracy of E2E-FT based WSE on ImageNet and derived distribution shifts for various values of the mixing coefficient a. Results shown for CLIP ViT-B/16.
Distribution shifts
Avg shifts
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
0.0
68.3
61.9
48.3
50.1
77.6
54.2
58
0.1
72.9
65.7
50.8
52.5
79.4
55.7
60.8
0.2
76.4
68.7
52.5
54.2
80.1
57.1
62.5
0.3
78.9
70.6
53.6
54.6
80.1
57.5
63.3
80.5
72.1
79.6
57.7
63.5
0.5
81.7
72.8
53.9
52.2
78.7
57.3
63.0
0.6
82.4
72.9
53.4
50.0
77.2
56.2
61.9
0.7
82.5
73.2
52.4
47.4
75.2
55.0
60.6
0.8
82.5
72.8
51.0
44.6
72.7
53.5
58.9
0,9
82.1
72.0
48.9
40.9
69.5
51.7
56.6
1.0
81.3
70.6
45.1
36.6
65.6
50.5
53.7
Table Accuracy of E2E-FT based VRF on ImageNet and derived distribution shifts for various values of and b. Results shown for CLIP ViT-B/16.
Distribution shifts
Avg shifts
b
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
1.4
0.5
82.2
72.2
52.7
49.6
79.4
56.7
62.1
1.4
0.6
82.2
72.2
52.7
49.6
79.4
56.7
62.1
1,4
0.7
82.2
72.2
52.7
49.7
79.4
56.8
62.2
1.4
0.8
82.1
72.2
52.7
49.7
79.4
56.8
62.2
1.4
0.9
82.1
72.1
52.7
49.7
79.4
56.8
62.2
1.4
1.0
82.1
72.1
52.7
49.7
79.4
56.8
62.2
1.5
0.5
82.3
72.1
52.3
48.7
79.0
56.2
61.7
1.5
0.6
82.3
72.1
52.4
48.9
79.1
56.4
61.8
1.5
0.7
82.2
72.2
52.4
49.1
79.2
56.5
61.9
1.5
0.8
82.2
72.2
52.5
49.2
79.2
$6.6
61.9
0.9
82.2
52.5
49.3
79.2
$6.6
62.0
1.5
1.0
82.2
72.2
52.6
49.4
79.2
56.6
62.0
1.6
0.5
82.3
71.9
51.9
48.0
78.5
55.7
61.2
1,6
0.6
82.3
72.1
52.1
48.3
78.6
55.9
61.4
1,6
0.7
82.3
72.1
52.2
48.5
78.8
56.0
61.5
1.6
0.8
82.3
72.2
52.3
48.6
78.9
56.1
61.6
1.6
0.9
82.3
72.2
52.3
48.7
79.0
61.7
1.6
1.0
82.3
72.1
52.4
48.8
79.0
56.3
61.7
C.7 Curves of for ImageNet and its Five Distribution Shifted Datasets
Ace
Figure
13
Actn and d(x) for ImageNet and its five derived we examine the relationship between Acey
distribution shifted datasets. Based on the value of d(x), test samples are grouped into bins, and we compute the ratio of fine-tuned accuracy zero-shot accuracy for each bin. For example, compute the value of Acsa, at d(x) 0.8, we first identify the samples with d(x) [0.7, 0.9), then compute Accy the averaged accuracy for these samples using zero-shot models and fine-tuned models, and finally
Table 11: Optimal Results search test set ImageNet and its five variants for CLIP ViT-B/16.
Distribution shifts IN-Sketch IN-A
Avg shifts
Method
IN
IN-V2
IN-R
ObjectNet
ETE-FT
81.3
70.6
45.1
36.6
65.6
50.5
53.7
VRF (ours)
82.3
72.1
52.9
48.4
78.7
56.4
61.8
VRF (oracle)
82.3
72.2
53.0
51.4
79.7
57.9
62.9
16

---

Table 12: Applying VRF to other robust fine-tuning methods.
Distribution shifts
Avg
Method
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
shifts
FLYP 5
82.6
73.0
71.4
48.1
49.6
58.7
60.2
+ WSE
82.9
73.5
76.0
53.0
52.3
60.8
63.1
+ OSE
73.6
77.0
52.5
51.9
59.9
62.8
+ VRF 82.8
73.6
78.6
52.9
53.0
61.2
64.0
Table 13: Breakdown performance for selective prediction using OOD detector.
Distribution shifts
Avg shifts
OOD Detector
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
MSP g
81.5
71.1
48.6
42.1
71.6
52.9
57.3
Energy [8]
81.0
70.5
48.1
42.3
73.9
53.0
57.6
MD
[6
81.0
70.4
49.7
41.7
74.1
$2.6
57.7
kNN
80.8
49.5
43.6
74.8
53.6
58.4
RMD
81.1
70.6
49.6
44.4
74.4
53.1
58.4
compute the ratio. Note that the averaged ratio Ace. ImageNet-(A.R) and ObjectNet is undefined for d(x) 0.6. This is because in these datasets, the zero-shot accuracy around d(x) 0.6 is O. We observe that the trend of the ratio Acee decreasing d(x) increasing is stable for all ImageNet related datasets.
Acen
Acea
2.50
ImageNet
Fine-Tuned Acc Zero-SSr Shot Acc
ImageNetV2
2.25
ImageNetSketch
2.00
ImageNetA
ImageNetR
1.75
ObjectNet
1.50
1.25
1.00
0.75
0.6
0.8
1.0
1.2
1.4
1.6
Distance to ZSF set (d(x))
Acca and d(x) on ImageNet benchmarks. Figure 13: Relationship between Acc,,
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We point out that the ID-OOD trade-off's still exist in existing fine-tuning meth- ods, and propose VRF to simultaneously attain the best ID and OOD accuracy. Experiments variety of different models and tasks validate the effectiveness of our proposed method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: Section we prove that our VRF can effectively reduce the variance of the ensemble model and thus achieve lower errors.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the VRF algorithm Algorithm with descriptions in Section and included the implementation details in Section and Section C.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We have uploaded the codes in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praip..cc/pbllicguuudee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the training and testing details in Section 5 and Sec-
tion CI
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given the zero-shot and the fine-tuned models, the process our post-hoc method is deterministic. Run multiple times will not introduce randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We have provided the GPU type and number to reproduce the results in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///wwwpppccoooo/o//ccccccta
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impacts in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method aims to improve the robustness when fine-tuning models, which poses no such risks to the best of our knowledge.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have provided the licences of each dataset in Section A
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "frYusJByOE", "reviewer": "Reviewer_vpPs", "summary": "This paper addresses the issue of ID-OOD (In-Distribution vs. Out-of-Distribution) in the context of robust fine-tuning techniques commonly used in ensemble methods. The authors propose a sample-wise mixing approach to resolve this problem. The method involves creating a Zero-Shot Failure set, which contains samples that fail in zero-shot models but succeed in fine-tuned models, and mixing the predictions of the zero-shot and fine-tuned models based on the distance between test samples and the Zero-Shot Failure set. This approach demonstrates superior performance over traditional weight-ensemble and prediction ensemble methods on the ImageNet variants benchmark.", "questions": "I am curious about the performance when finding hyperparameters for each test set as an oracle result.", "limitations": "The authors have adequately addressed some limitations of their work. However, they have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no negative societal impacts.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, clearly explaining the proposed method and its motivation. The intuitive explanation, especially the use of Figure 2 to show the monotonic relationship between fine-tuned/zero-shot accuracy ratio and distance, strongly supports the proposed approach.\n- The paper makes a novel contribution by addressing the ID-OOD problem, which is not fully addressed by the existing ensemble-based robust fine-tuning.\n- The proposed method is simple and easy to implement, yet it achieves better performance compared to existing ensemble methods.", "weaknesses": "- The main drawback of the proposed method is the reliance on multiple hyperparameters (a, b, p), making tuning more complex compared to traditional ensemble methods that typically involve only one hyperparameter. Finding optimal values for these hyperparameters is challenging as they can vary significantly across different datasets, potentially affecting performance.\n- To demonstrate the robustness of the proposed method, evaluations should be extended to more datasets beyond the ImageNet variants, similar to the experiments in the WiSE-FT paper, which include datasets like iWILDCam and FMoW. Differences in hyperparameter sweep ranges across datasets could highlight potential issues with the method.\n- There is a lack of performance comparison with other robust fine-tuning methods, such as FLYP[1] and MaskFill[2], which limits the assessment of the proposed method’s effectiveness. In the ImageNet classificaiton experiments, the proposed method is inferior to [1,2] in performance.\n- The paper need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity, providing deeper insights into the robustness of the proposed method.\n\n[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et. al., CVPR 23\n[2] Masked images are counterfactual samples for robust fine-tuning, Xiao et. al., CVPR 23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  The main drawback of the proposed method is the reliance on multiple hyperparameters.\n\n**A1**: We understand and acknowledge the reviewer’s concern regarding the complexity of tuning multiple hyperparameters. However, we would like to provide further clarification and context to address this issue:\n* No Need to Optimize p: First, we emphasize that the hyperparameter $p$ does not require optimization in our approach. We adopt the default value from [24], and as demonstrated in Figure 5(c), the performance is not sensitive to changes in $p$. To further validate this, we conducted experiments on CIFAR and Entity-30 datasets, where we varied $p$ from 0.0002% to 50%. The ID and OOD performance fluctuated by less than 0.1% and 0.3%, respectively, confirming that $p$ does not significantly impact the overall performance.\n* Simplified Search for a and b: Regarding the hyperparameters $a$ and $b$, the optimization process is straightforward and can be managed through grid search on the ID validation set. Additionally, we observed that the performance tends to peak at consistent values of $b$ even as $a$ varies. This pattern significantly reduces the search space for $b$, making the tuning process more efficient and less complex than it might initially appear.\n\nIn conclusion, while our method involves multiple hyperparameters, we have shown that the tuning process is manageable and not as complex as it may seem. The robustness of $p$ and the structured tuning of $a$ and $b$ make our approach both practical and efficient across different datasets.\n\n\n**Q2**: Need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity.\n\n**A2**: We appreciate the reviewer’s insightful suggestion. We have reported the searched hyperparameters across different datasets and architectures below and will include them in the revised version of the paper. \n\nOur findings indicate that for the same dataset, the hyperparameters remain consistent across different architectures, suggesting that our method's performance is robust to architectural changes. However, the hyperparameters do vary between datasets, which is expected due to differences in data distribution. For example, the k-NN distance ranges from 0.4 to 1.6 for ImageNet and from 0.4 to 1.2 for Entity30, reflecting the underlying data characteristics. Correspondingly, the searched values for $a$ are 1.5 for ImageNet and 1.1 for Entity30. This variability in  $a$ highlights the importance of adapting hyperparameters to specific datasets to achieve optimal performance.\n\nViT/32\n\n| Dataset | ImageNet | CIFAR | Entity30 |\n|---------|----------|-------|----------|\n| a       | 1.5      | 0.3   | 1.1      | \n| b       | 0.6      | 0.3   | 0.6      | \n\nViT/16\n\n| Dataset | ImageNet | CIFAR | Entity30 | \n|---------|----------|-------|----------|\n| a       | 1.5      | 0.3   | 1.1      | \n| b       | 0.5      | 0.3   | 0.6      |\n\n\n**Q3**:  The performance when finding hyperparameters for each test set as an oracle result.\n\nWe have conducted additional experiments where we optimized the hyperparameters for each test set of the ImageNet benckmarks. The results are summarized in the following table.\n\n| ViT/16     | IN   | IN-V2 | IN-S | IN-A | IN-R | ObjectNet |\n|------------|------|-------|------|------|------|-----------|\n| VRF        | 82.3 | 72.1  | 52.9 | 48.4 | 78.7 | 56.4      |\n| VRF oracle | 82.3 | 72.2  | 53.0 | 49.7 | 79.4 | 56.8      |\n\n\n**Q4**: iWILDCam and FMoW results.\n\n**Q4**: We have conducted experiments (E2E-FT model as the fine-tuned models) on fmow and iwildcam and report the performance with the hyper-parameters below. We will add the results of iWILDCam and FMoW in the revised version. \n\n| iWILDCam            | ID   | OOD  |\n|---------------------|------|------|\n| zero-shot           | 8.7  | 11.0 |\n| E2E-FT              | 48.0 | 34.7 |\n| +WSE                | 48.1 | 35.3 |\n| +OSE                | 48.0 | 35.0 |\n| +VRF (a=1.2, b=0.6) | 48.1 | 36.1 |\n\n| FMoW                | ID   | OOD  |\n|---------------------|------|------|\n| zero-shot           | 20.4 | 18.7 |\n| E2E-FT              | 68.5 | 39.2 |\n| +WSE                | 68.5 | 39.2 |\n| +OSE                | 68.5 | 39.2 |\n| +VRF (a=1.4, b=0.5) | 68.6 | 41.0 |\n\n**Q5**: Comparison with other robust fine-tuning methods.\n\n**A5**:  Our VRF framework is designed to be orthogonal and complementary to existing fine-tuned models. To demonstrate this, we conducted additional experiments using FLYP as the fine-tuned model within our VRF framework. The results show that our VRF framework enhances the performance of OSE by improving the distribution shift performance by 1.1% while maintaining the in-distribution (ID) performance.\n\n\n|      | IN   | V2   | R    | A    | S    | ObjectNet | OOD  |\n|------|------|------|------|------|------|-----------|------|\n| FLYP | 82.6 | 73.0 | 71.4 | 48.1 | 49.6 | 58.7      | 60.2 |\n| +WSE | 82.9 | 73.5 | 76.0 | 53.0 | 52.3 | 60.8      | 63.1 |\n| +OSE | 82.8 | 73.6 | 77.0 | 52.5 | 51.9 | 59.9      | 62.8 |\n| +VRF | 82.8 | 73.6 | 78.6 | 52.9 | 53.0 | 61.2      | 64.0 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OT1WqpNKxl", "reviewer": "Reviewer_e2mj", "summary": "This paper aims to tackle the ID-OOD trade-off in the fine-tuning of pre-trained models with zero-shot abilities like CLIP. The proposed Variance Reduction Fine-tuning (VRF) is a sample-wise ensembling method concerning the zero-shot and fine-tuned models. The ensemble weights are determined by the distance from the test sample to the training samples that are incorrectly predicted by the zero-shot model. Theoretical analysis and experimental results justify the effectiveness of VRF.", "questions": "1. To what extent is VRF less efficient than existing ensembling methods?\n2. Why is k-th nearest neighbor distance selected for the distance calculation step of VRF?\n3. What is the meaning of solving the ID-OOD trade-offs?", "limitations": "The authors have addressed some limitations and potential impacts of the work. I suggest additional discussions on the efficiency of the proposed method in the limitation part.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The proposed VRF can achieve better performance on both ID and OOD data compared with existing ensemble-based methods.\n2. Unlike previous methods, the ensemble weights in VRF does not require tuning.\n3. The experiments are thorough, and the results are clearly presented in the tables and figures.", "weaknesses": "1. A major downside of VRF compared to previous ensemble methods is that the determination of the ensemble weights requires the storage of ZSF features at test time and computation of the distance from the test samples to each sample in ZSF set. This may bring a significant budget when the ZSF set is large. There should be a discussion on the theoretical and practical computational complexity regarding space and time.\n2. The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF.\n3. It is claimed that the proposed method “can simultaneously attain the best ID and OOD accuracy without the trade-offs” (Line 9). However, this is not sufficiently justified. Specifically, it is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  There should be a discussion on the theoretical and practical computational complexity regarding space and time.\n\n**A1**: To address the efficiency of k-NN search, we leverage the Faiss library [11], which is optimized for large-scale similarity searches using GPUs. In our ImageNet experiments, the inference speed of the k-NN search for a single image is approximately 3.2 milliseconds, demonstrating that our approach can be executed efficiently even with large datasets.\n\nFor our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.\n\nWe will include the discussion of the computational complexities in the revised version.\n\n**Q2**: The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF.\n\n**A2**: We appreciate the reviewer’s observation and would like to clarify our rationale for choosing k-NN distance as the metric for density estimation in our VRF framework:\n\n* Training-Free and Easy Implementation: The k-NN method is training-free, making it simple to implement without requiring additional trained models. This aligns with our goal of maintaining a lightweight and efficient approach.\n* Non-Parametric Nature: k-NN is a non-parametric method, meaning it does not make any assumptions about the underlying distribution of the data. This flexibility contrasts with methods like those in [16, 22], which assume the feature space follows a Gaussian distribution, making k-NN a more robust choice across different datasets.\n* Strong ID and OOD Performance: We selected k-NN based on its demonstrated ability to effectively capture the relationship between distance measures and performance metrics (Acc_ft/Acc_zs) in both ID and OOD settings. This consistent performance across different scenarios reinforces our confidence in k-NN as a reliable metric.\n\nWhile we acknowledge that the averaged k-NN distance could also be a viable alternative, our experiments show that it yields similar results (82.3 ID and 61.8 OOD on ImageNet benchmarks) to the k-NN distance. However, the averaging step introduces additional computational complexity, albeit slight. To keep our method succinct and efficient, we chose to use k-NN distance.\nWe will include the discussion in the revised version.\n\n**Q3**: It is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.\n\n**A3**: The statement “best ID and OOD accuracy” refers to the comparison with other existing methods. Our framework is designed to be orthogonal to existing ensembling techniques, meaning it can be applied in conjunction with them to enhance performance. Traditional ensembling methods often face trade-offs where optimizing for ID performance can degrade OOD performance, and vice versa. However, our VRF framework is intended to mitigate these trade-offs by simultaneously optimizing for both ID and OOD accuracy.\n\nIn Figure 3 (a.1), the best OOD performance of conventional ensembling is 98.5%, while VRF achieves 98.4%. Although VRF is marginally lower by 0.1%, it is important to note that this difference is within a very small margin, particularly as both approaches are nearing the upper performance limit (close to 100%). Thus, we consider VRF's performance to be effectively on par with the best OOD accuracy, while also offering strong ID performance without requiring a trade-off.\n\n**Q4**: To what extent is VRF less efficient than existing ensembling methods?\n\n**A4**: The key distinction between VRF and conventional ensembling methods lies in their objectives. While traditional ensembling methods primarily focus on enhancing in-distribution (ID) performance, our VRF framework is designed to simultaneously improve both ID and out-of-distribution (OOD) performance. This broader objective naturally introduces additional computational overhead. However, as mentioned earlier, we utilize the Faiss library for efficient k-NN search, significantly reducing the time complexity. For example, in our ImageNet experiments, the k-NN search for a single image takes approximately 3.2 milliseconds, which we consider manageable given the overall performance benefits.\n\n**Q5**: why is k-th nearest neighbor distance selected.\n\n**A5**: We avoid using the nearest sample to reduce the impact of label noise in the training set. If the nearest sample is mislabeled, the distance calculation could be unreliable. By selecting the $k$-th nearest sample, we mitigate this risk, as the likelihood of all $k$ nearest samples being mislabeled is low.\n\n\n**Q6**: What is the meaning of solving the ID-OOD trade-offs?\n\n**A6**: Solving the ID-OOD trade-offs means developing models that can maintain high accuracy on in-distribution (ID) data while also performing well on out-of-distribution (OOD) data. Typically, improving OOD performance can reduce ID accuracy, but addressing these trade-offs aims to balance both, ensuring models are reliable in both familiar and new environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqOVtw8nAK", "reviewer": "Reviewer_8xce", "summary": "This paper studies the trade-off between in-distribution (ID) and out-of-distribution (OOD) performance of pre-trained models before and after fine-tuning. The authors observed that the sample distance is inversely proportional to $\\frac{Acc_{ft}}{Acc_{zs}}$. After modeling the residual error of the model, they found that for a training sample $x$, the optimal ensemble weight is proportional to the sample's accuracy. Therefore, the authors set different ensemble weights for different samples based on the distance between the samples and the erroneous samples in the training set. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper effectively improves the model's ID-OOD performance by setting different ensemble weights for different sample.\n2. Experiments show that this method can be effectively applied to different fine-tuning techniques and can significantly enhance performance.", "weaknesses": "1. VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations.\n2. The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in $V$, or is it clustering the representations in $V$ first and then calculating the distance to the $k$-th cluster center?\n3. Is the representation $ v $ of $ x $ calculated by ft or zs?\n4. For each sample, recalculating and sorting the distance to $V$ incurs additional inference costs. How much extra computational cost does this introduce?\n5. Since line 278 indicates that different values of $k$ have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the $k$-th sample for distance calculation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations.\n\n**A1**: For our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.\n\n**Q2**: The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in , or is it clustering the representations in  first and then calculating the distance to the -th cluster center?\n\n**A2**: We apologize for the unclear description. In Step 2, we calculate the distance to the k-th nearest sample in the ZSF set. We do not perform any clustering of the representations. We will clarify this in the revised version of the paper.\n\n**Q3**: Is the representation v of x calculated by ft or zs?\n\n**A3**: We use the representation from the fine-tuned model, as mentioned in Line 120: \"we collect its feature representation from the fine-tuned model.\"’\n\n**Q4**: For each sample, recalculating and sorting the distance to incurs additional inference costs. How much extra computational cost does this introduce?\n\n**A4**: To address the efficiency of k-NN search, we leverage the Faiss library [11], which is optimized for large-scale similarity searches using GPUs. In our ImageNet experiments, the inference speed of the k-NN search for a single image is approximately 3.2 milliseconds, demonstrating that our approach can be executed efficiently even with large datasets.\n\n**Q5**: Since line 278 indicates that different values of  have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the -th sample for distance calculation?\n\n**A5**: We avoid using the nearest sample to reduce the impact of label noise in the training set. If the nearest sample is mislabeled, the distance calculation could be unreliable. By selecting the $k$-th nearest sample, we mitigate this risk, as the likelihood of all $k$ nearest samples being mislabeled is low.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYkkCaDUKT", "reviewer": "Reviewer_52xu", "summary": "This study examines the robust fine-tuning of CLIP models. The proposed method, Variance Reduction Fine-tuning (VRF), employs a sample-wise ensembling technique to enhance both ID and OOD accuracy, reducing trade-offs between them. Experimental results on ImageNet and associated distribution shifts empirically validate the effectiveness of this approach.", "questions": "1. In combining zero-shot and fine-tuned models, WSE offers the advantage of requiring only half the inference cost compared to VRF by maintaining inference costs equivalent to that of a single model through weight averaging. When combining zero-shot and linear-probed models, since the image encoder part, which carries most of the inference cost, only needs to be processed once, it appears that VRF's concerns regarding cost could be mitigated. In this context, could the proposed VRF method also extend to \"Linear classifier\" models, alongside \"E2E-FT\" and \"LP-FT\"? It is worth noting that Wortsman et al. (2021) explores ensembling not only between zero-shot models and end-to-end fine-tuned models (\"E2E-FT\" in this work) but also between zero-shot models and linear classifier fine-tuned models (referred to as \"Linear classifier\" here).\n\n2. In Figures 3, does \"Ensembling with varying coefficient $\\alpha$\" refer to OSE? It would be helpful to also provide the curve for WSE.\n\n3. Table 7 shows the results of exploring $\\alpha$ values for OSE. It would be beneficial to provide the same results for WSE as well.\n\n4. The proposed VRF approach also produces multiple points on trade-off plots (e.g., Figure 3(a.1)) depending on the values of $a$ and $b$ (especially $a$). It would be advantageous to visualize a scatter plot across a broad range of $a$ and $b$ values, allowing us to observe the trade-off curve similar to OSE and WSE, which interpolate between zero-shot and fine-tuned models.", "limitations": "The authors addressed the limitations and potential negative societal impact in Section 6.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed approach is straightforward, combining outputs from zero-shot and fine-tuned models through output-space ensembling. Unlike conventional methods that use uniform or predefined weighting coefficients, the notable aspect here is the use of different weighting coefficients for each individual instance.\n\n- It is clear that adjusting these weighting coefficients according to whether zero-shot or fine-tuned models are suitable for specific data can improve the final performance. Consequently, the primary challenge lies in devising an effective method to determine such instance-specific weighting coefficients. The authors propose a method that makes use of the Zero-Shot Failure (ZSF) set, consisting of training examples where the zero-shot model fails to predict accurately but the fine-tuned model succeeds. Experimental validation confirms the efficacy of the proposed approach.\n\n- I appreciate Section 5.3 for further presenting an empirical analysis of the proposed method. Essentially, the approach is a variation of OSE, determining how much the zero-shot model and the fine-tuned model are used on a per-instance basis, and VRF is not the only strategy to achieve this. It is gratifying to see that VRF outperforms other alternatives and that the analysis examines how different VRF designs produce varying outcomes.", "weaknesses": "- Although there has been considerable research on robust fine-tuning for CLIP (as noted by the authors referencing various studies), the baselines analyzed in Tables 1 and 2 are quite limited. A key advantage of the VRF method is its applicability to any (zero-shot, fine-tuned) pair, both retrospectively and prospectively. Therefore, applying VRF to other robust fine-tuning methods beyond FT and LP-FT would further demonstrate the effectiveness of the proposed VRF approach.\n\n- The proposed method has a drawback of needing more than double the inference cost compared to WSE and other robust fine-tuning methods that yield a single fine-tuned model, since it requires performing a forward pass through the entire network twice (once for zero-shot and once for fine-tuned models) and calculating the ZSF-based weighting coefficient. However, an analysis of such inference costs has not yet been performed. It is important to note that Wortsman et al. (2022) favored WSE over OSE for combining zero-shot and fine-tuned models, highlighting this choice not only for its improved performance but also for its lack of additional costs.\n\n- In the context of robust fine-tuning, we can only observe performance on in-distribution data. Therefore, the parameters \n$a$ and $b$ in the proposed Eq. (6) for sample-wise mixing coefficients should be determined based on the in-distribution validation set, as mentioned by the authors in lines 141 and 198-200. From Appendix C.1, it is understood that $\\\\alpha \\\\in \\\\{1.4, 1.5, 1.6\\\\}$ and $b \\\\in \\\\{0.5, 0.6, ..., 1.0\\\\}$ were considered. However, it seems that the range of values for $a$ and $b$ is too narrow. Based on the in-distribution results in Table 8, larger values for $a$ should be explored, but the authors did not do so. What happens if the value of $a$ is increased further? Would it still outperform OSE and WSE baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Applying VRF to other robust fine-tuning methods.\n\n**A1**: To further demonstrate the versatility and effectiveness of our VRF method, we applied it to another robust fine-tuning method, FTYP, and conducted experiments on ImageNet and its variants. As expected, our VRF framework further improves OOD performance without sacrificing ID accuracy, reinforcing its applicability across different fine-tuning strategies. We will include these additional results in the revised version.\n\n|      | IN   | V2   | R    | A    | S    | ObjectNet | OOD  |\n|------|------|------|------|------|------|-----------|------|\n| FLYP | 82.6 | 73.0 | 71.4 | 48.1 | 49.6 | 58.7      | 60.2 |\n| +WSE | 82.9 | 73.5 | 76.0 | 53.0 | 52.3 | 60.8      | 63.1 |\n| +OSE | 82.8 | 73.6 | 77.0 | 52.5 | 51.9 | 59.9      | 62.8 |\n| +VRF | 82.8 | 73.6 | 78.6 | 52.9 | 53.0 | 61.2      | 64.0 |\n\n**Q2**: What happens if the value of  is increased further? Would it still outperform OSE and WSE baselines?\n\n**A2**: For the hyperparameter $a$, we chose a range less than 1.6 based on the nature of the ZSF set of ImageNet, where nearly 99% of validation sample distances were less than 1.6. Exploring larger values of $a$ (i.e., $a>1,6$) is not necessary because it results in almost all predictions being assigned to the fine-tuned models, reducing the contribution of the zero-shot models. Since the features are L2-normalized, the distance d(x) is bounded between [0, 2], we further tested values $a = 1.7, 1.8, 1.9 $ and observed that OOD performance decreased as expected, with the fine-tuned model dominating the ensemble contribution.\n\nFor the hyperparameter $b$, as shown in Figure 6(a), we noticed that increasing $b$ beyond 1 leads to a decrease in ID performance. Therefore, we limited our exploration to $b \\leq 1 $ to maintain a balance between ID and OOD performance.\n| a   | b   | ID   | OOD  |\n|-----|-----|------|------|\n| 1.5 | 0.5 | 82.3 | 61.8 |\n| 1.7 | 0.5 | 82.2 | 60.6 |\n| 1.7 | 0.7 | 82.2 | 61.1 |\n| 1.7 | 0.9 | 82.3 | 61.3 |\n| 1.7 | 1.0 | 82.2 | 61.3 |\n| 1.8 | 0.5 | 82.2 | 60.0 |\n| 1.8 | 0.7 | 82.3 | 60.7 |\n| 1.8 | 0.9 | 82.2 | 61.1 |\n| 1.8 | 1.0 | 82.2 | 61.2 |\n| 1.9 | 0.5 | 82.1 | 59.4 |\n| 1.9 | 0.7 | 82.2 | 60.3 |\n| 1.9 | 0.9 | 82.2 | 60.8 |\n| 1.9 | 1.0 | 82.2 | 60.9 |\n\n**Q3**: Could the proposed VRF method also extend to \"Linear classifier\" models?\n\n**A3**: Yes, using linear classifiers can indeed reduce the inference costs, as the image encoder only needs to be processed once, effectively halving the costs compared to VRF's standard approach. To explore this, we trained a linear classifier based on CLIP-ViT/16 models and compared the performance with WSE and our VRF. The results confirmed that our VRF framework effectively addresses the ID-OOD trade-offs, achieving higher OOD performance without compromising ID accuracy. This demonstrates that VRF can extend to linear classifier models, providing a versatile and effective solution for robust fine-tuning.\n\n|                  | ID   | OOD  |\n|------------------|------|------|\n| Linear Classifer | 79.3 | 55.2 |\n| +WSE/OSE         | 79.9 | 57.8 |\n| +VRF             | 80.0 | 61.0 |\n\n**Q4**: In Figures 3, does \"Ensembling with varying coefficient \" refer to OSE? It would be helpful to also provide the curve for WSE.\n\n**A4**:  Yes, the \"Ensembling with varying coefficient\" refers to OSE. We have also evaluated the ID and OOD performance with varying $\\alpha$ for WSE. The curves are plotted in the attached PDF.\n\n**Q5**:  Table 7 shows the results of exploring  values for OSE. It would be beneficial to provide the same results for WSE as well.\n\n**A5**: Thanks for your suggestions. We provide the WSE results with varying $\\alpha$ values as below and we will add them in the revised version. \n\n| $\\alpha$ | ID   | OOD  |\n|----------|------|------|\n| 0.0      | 68.3 | 58.1 |\n| 0.1      | 72.9 | 60.8 |\n| 0.2      | 76.4 | 62.5 |\n| 0.3      | 78.9 | 63.3 |\n| 0.4      | 80.5 | 63.4 |\n| 0.5      | 81.7 | 63.0 |\n| 0.6      | 82.4 | 61.9 |\n| 0.7      | 82.5 | 60.6 |\n| 0.8      | 82.5 | 58.9 |\n| 0.9      | 82.1 | 56.6 |\n| 1.0      | 81.3 | 53.8 |\n\n**Q6**: It would be advantageous to visualize a scatter plot across a broad range of  a and b values.\n\n**A6**: We appreciate the suggestions. We have visualized the trade-off curves in the attached PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2udNmun9W", "reviewer": "Reviewer_ydTd", "summary": "This paper addresses the ID-OOD trade-off of the ensemble-based method under the pretrain-finetune regime. The authors propose a simple yet effective adaptive ensemble method that determines the ensemble coefficient based on the feature distance between a test sample and the zero-shot failure set. The proposed method has a theoretical foundation, and it consistently improves the OOD generalization performance compared to the vanilla ensemble baseline in diverse situations.", "questions": "* Could the proposed method also be applied to the weight-space ensemble rather than the output-space ensemble?\n* How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?", "limitations": "The authors adequately stated the limitation in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written overall and easy to follow.\n* The research problem this work focuses on is important in the era of foundation models, and the proposed method to address this is very straightforward and reasonable.\n* The proposed method shows consistent performance improvement in numerous distribution shift setups, and the author provides an intuitive theoretical justification for the proposed method.\n* Besides the empirical success in terms of improved ID-OOD performance trade-off, the authors present extensive qualitative and quantitative analysis results that give numerous insights into this field.", "weaknesses": "* The method requires 1) accessibility to the entire training dataset and 2) distance computation between each test sample and the entire train sample during inference time. These limit its application to resource-constrained (in terms of accessibility, memory storage, and runtime) situations.\n* Limited implication of the theoretical result\n  * While the current theoretical analysis provides a justification for the optimal strategy for determining the ensemble coefficients, which is realized by the authors, it does not say anything about the relative generalization error between the proposed method and other non-ensemble-based fine-tuning methods. \n* Lack of possible comparison with more advanced fine-tuning [4,5,6,7] or ensemble [1,2,3] methods.\n  * While the proposed approach has its unique implication compared to the vanilla ensemble, missing comparison with advanced baselines raises concern about the practical utility of the proposal.\n\n---\n> Reference\n* [1] SIMPLE: Specialized Model-Sample Matching for Domain Generalization, Li et al. 2023\n* [2] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models, Zhu et al. 2023\n* [3] Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization, Mavromatis et al. 2024\n* [4] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, Kumar et al. 2022\n* [5] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et al. 2023\n* [6] Trainable Projected Gradient Method for Robust Fine-tuning, Tian et al. 2023\n* [7] Towards Calibrated Robust Fine-Tuning of Vision-Language Models, Oh et al. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  Weaknesses on computation costs.\n\n**A1**: For efficiently performing k-NN search, we use Faiss library [11], which can perform billion-scale similarity search with GPUs. For our ImageNet experiments, the inference speed of kNN search for a single image is averaged 3.2 ms. For our ImageNet experiments, the storage size for the ZSF set features is 289 MB. While this does require additional storage compared to some traditional methods, we believe it is a manageable overhead given the benefits in robustness and performance that our VRF framework provides.\n\n**Q2**: Limited implication of the theoretical result.\n\n**A2**: In non-ensemble-based fine-tuning methods, the model can be seen as using fixed coefficients, essentially $g_{ft}(x) = 1$. Since this is not the optimal strategy for balancing ID and OOD performance, the generalization error for non-ensemble-based fine-tuning is expected to be relatively larger compared to our VRF method, which optimizes these coefficients.\n\n**Q3**: Lack of possible comparison with more advanced fine-tuning.\n\n**A3**: Since our VRF framework is orthogonal to the fine-tuned models, we use the [4][5] as the fine-tuned models and report results below.\n|      | IN    | V2    | R     | A     | S     | ObjectNet | OOD  |\n|------|-------|-------|-------|-------|-------|-----------|------|\n| [4]  | 81.5  | 70.7  | 46.7  | 41.4  | 66.4  | 52.4      | 55.5 |\n| +WSE | 82.4  | 73.0  | 51.5  | 50.6  | 74.2  | 56.6      | 61.2 |\n| +OSE | 82.1  | 72.3  | 50.9  | 50.9  | 74.9  | 55.7      | 60.9 |\n| +VRF | 82.1  | 72.3  | 52.9  | 51.2  | 78.8  | 57.2      | 62.4 |\n\n|      | IN    | V2    | R     | A     | S     | ObjectNet | OOD  |\n|------|-------|-------|-------|-------|-------|-----------|------|\n| [5]  | 82.6  | 73.0  | 71.4  | 48.1  | 49.6  | 58.7      | 60.2 |\n| +WSE | 82.9  | 73.5  | 76.0  | 53.0  | 52.3  | 60.8      | 63.1 |\n| +OSE | 82.8  | 73.6  | 77.0  | 52.5  | 51.9  | 59.9      | 62.8 |\n| +VRF | 82.8  | 73.6  | 78.6  | 52.9  | 53.0  | 61.2      | 64.0 |\n\n**Q4**: Could the proposed method also be applied to the weight-space ensemble?\n\n**A4**: Yes, our method can be applied to weight-space ensemble models. To do this efficiently, we first generate a set of weight-ensembled models with varying coefficients $\\alpha=[0.1,0.2,..,0.9]$. For each sample, we calculate the weight function and select the cached WSE model with the closest $\\alpha$. The results for CIFAR10 -> STL10 validate that our VRF approach is effective for weight-space ensemble methods as well.\n\n|             | ID   | OOD  |\n|-------------|------|------|\n| VRF for OSE | 98.6 | 98.4 |\n| VRF for WSE | 98.5 | 98.7 |\n\n**Q5**: How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?\n\n**A5**: Our proposed method is robust to variations in the number of training samples in the zero-shot failure set. To validate this, we randomly downsampled the zero-shot failure set by 10%, 20%, and 50%. The results showed only minor fluctuations of ID and OOD performances. This demonstrates that our method maintains its effectiveness even with a reduced zero-shot failure set.\n\n|     | 10%  | 20%  | 50%  | 100% |\n|-----|------|------|------|------|\n| ID  | 82.2 | 82.4 | 82.5 | 82.3 |\n| OOD | 61.8 | 62.0 | 61.7 | 61.8 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have uploaded ID-OOD frontier curves for WSE in the attachment.

Meta-review: This paper presents an output-space ensemble approach to robust finetuning of zero-shot model (i.e., weighted sum of outputs from zero-shot and finetuned models). The reviewers appreciate the novel, simple, and effective method and its strong performance, and the clarity of the manuscript. They however at the same time raised multiple nontrivial concerns. Most reviewers were concerned about the high space-time complexity due to the double forward propagations and the k-th nearest neighbor distance computation (e2mj, 8xce, 52xu, ydTd). Other concerns include additional hyperparameters that may have to be carefully tuned per dataset (vpPs, 52xu), lack of comparisons with latest techniques (vpPs, ydTd), lack of empirical evidences supporting the versatility of the method (52xu), experiments on limited datasets (vpPs), design choices that are not well justified (e.g., the use of k-th nearest neighbor distance) (e2mj, 8xce), and some presentation issues (e2mj). The authors' rebuttal and subsequent responses in the discussion period address most of these concerns; a borderline reviewer was not fully satisfied by the response and still concerned in particular about the lack of sensitivity analysis, but mentioned that it is not a ground for rejection. In the end, all the reviewers were supportive of the paper. 

The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers and to add new results and discussions brought up in the rebuttal and discussions, in particular a more thorough analysis on the sensitivity to hyperparameters and more experiments with a linear probed model; the latter is essential to fully resolve the concerns with the space-time complexity.
Decision: Accept (poster)