Title: Robust Fine-tuning of Zero-shot Models via Variance Reduction
Venue: NeurIPS 2024
Accepted: yes
Keywords: image classification

fine-tuning

zero-shot models
Abstract: When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains on (0.9 - 3.1 pp) on other distribution shifts
19 benchmarks. Codes are available in https://github.com/BeierZhu/VRF.

Robust Fine-tuning of Zero-shot Models via Variance Reduction

Abstract

When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains (0.9 - 3.1 pp) on other distribution shifts benchmarks. Codes are available in https://github.com/BeierZhu/VRF .

1 Introduction

To ensure the reliability of machine learning systems, it is essential to develop models that can generalize to unseen, out-of-distribution environments. Large pre-trained models such as CLIP [20] and ALIGN [10] have recently shown remarkable robustness against challenging distribution shifts. However, it is widely acknowledged that these improvements in robustness are most pronounced in the zero-shot setting, while conventional fine-tuning on these models often compromises robustness when compared to zero-shot performance [28, 15, 14]. This phenomenon is known as the ID-OOD trade-offs, i.e. , improving performance on in-distribution (ID) data can sometimes lead to decreased performance on out-of-distribution (OOD) data [12, 25].

In recent years, ensemble-based models (ESMs) have demonstrated significant success in addressing the ID-OOD dilemma [17, 28, 14, 31]. Specifically, denote the input as x , the zero-shot model as ˆ P ( y | x ; θ zs ) and the fine-tuned model as ˆ P ( y | x ; θ ft ) , existing ESMs typically employ the outputspace ensemble (OSE) [14, 31], which outputs ˆ P ( y | x ; θ ose ) = α ˆ P ( y | x ; θ ft ) + (1 -α ) ˆ P ( y | x ; θ zs ) , and the weight-space ensemble (WSE) [28, 17], which outputs ˆ P ( y | x ; θ wse ) = ˆ P ( y | x ; αθ ft +(1 -α ) θ zs ) , where α ∈ [0 , 1] . Compared to fine-tuned models, ESMs offer significant accuracy enhancements under distribution shift, while maintaining high ID accuracy.

However, ESM cannot fully address the ID-OOD trade-offs. In Figure 1 (a), by varying the mixing coefficient α , we plot the ID-OOD frontier curves (pink line) for the CLIP ViT-B/16 model on

(a)

(b)

ImageNet [3] (ID) and five derived distribution-shifted datasets (OOD): ImageNet-V2 [21], ImageNetR [7], ImageNet-A [9], ImageNet-Sketch [27] and ObjectNet [1]. We find that the ensemble model achieves its optimal ID and OOD performance at different α values: the best ID accuracy is achieved at α = 0 . 5 and the best OOD accuracy is obtained at α = 0 . 3 . When the ensemble model reaches its optimal value for OOD, the performance on ID decreases by 3 . 6% relative to its peak. Similarly, when the ensemble model is optimized for ID, the performance on OOD decreases by 1 . 6% relative to its best value - the ID-OOD trade-offs still persist for ESMs. This raises a natural question:

Can ensemble-based models simultaneously attain the best ID and OOD accuracy?

In this paper, we affirmatively answer this question by proposing a sample-wise ensembling technique, dubbed variance reduction fine-tuning (VRF). This method is motivated by an empirical finding illustrated in Fig 1 (b). For each sample in the training dataset, if the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect its features representation in the finetuned model as the zero-shot failure (ZSF) set. We then measure the distance d ( x ) of each test sample x to the ZSF set. Based on this distance, test samples are grouped into bins, and we compute the ratio of fine-tuned accuracy to zero-shot accuracy: Acc ft Acc zs for each bin (implementation details are in Section C.7). Interestingly, we observe that the ratio Acc ft Acc zs monotonically decreases as d ( x ) increases. Intuitively, the closer a sample is to the ZSF set, the more likely it is that the zero-shot model makes incorrect predictions, whereas the fine-tuned model is more likely to be accurate, leading to a higher Acc ft Acc zs ratio. Therefore, we use the distance to assign weights to the models: a smaller d ( x ) results in a higher weight for the fine-tuned model, and vice versa.

As depicted by the orange diamond in Fig. 1 (a), by leveraging the sample-wise weights, our VRF simultaneously attains the best ID and OOD accuracy. In Section 5, we show that on a variety of different models and tasks, our VRF approach consistently outperforms the existing fine-tuning and ensembling methods, including linear probing, end-to-end fine-tuning, LP-FT [15], OSE and WSE [28]. In specific, on ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. Furthermore, in Section 4, we justify our approach by demonstrating that it effectively minimizes the variance of the ensemble models, resulting in reduced residual error.

2 Related Work

Mitigating ID-OOD trade-offs. Improving performance on in-distribution data can sometimes lead to a decrease in performance on out-of-distribution data, and vice versa. This phenomenon is known as the ID-OOD trade-offs. Xie et al. [29] leverage auxiliary information as outputs of auxiliary tasks to pre-train a model to reduce OOD error. Khani and Liang [12] show that self-training on large amounts of unlabeled data can mitigate such trade-offs by removing spurious features. Tripuraneni et al. [25] tackle this problem by learning representations that are robust across diverse tasks. However, these methods usually necessitate additional unlabeled data or auxiliary information. In contrast, our VRF is a straightforward variation of fine-tuning that does not require any extra data.

Robust fine-tuning of zero-shot models. Vision-language models like CLIP [20] have demonstrated outstanding improvements in robustness. It is commonly acknowledged that conventional finetuning methods often compromise robustness when compared to zero-shot performance. Therefore, enhancing downstream robustness has been the focus of subsequent works [15, 28, 5, 19, 6, 30]. Kumar et al. [15] show that a two-process of linear probing followed by full fine-tuning can alleviate feature distortion, leading to stronger OOD performance without sacrificing ID accuracy. Wortsman et al. [28] propose a method of weight interpolation between the zero-shot and the fine-tuned models to improve both ID and OOD accuracy. Goyal et al. [5] demonstrate that mimicking the contrastive pre-training objectives to fine-tune the zero-shot models outperforms tuning via the traditional supervised cross-entropy loss. However, the ID-OOD trade-offs are still observed with these methods. In contrast, our method VRF can simultaneously achieve the best ID and OOD accuracy.

3 Methods

3.1 Set Up

Task: Consider a classification setting where the goal is to map an instance x ∈ X to a label y ∈ Y = [ K ] . We are provided with a zero-shot model f ( · ; θ zs ) , a downstream dataset D = { x i , y i } N i =1 , and a fine-tuned model f ( · ; θ ft ) which is trained on D . Below, we outline the implementation of the zero-shot and fine-tuned models:

Zero-shot models (ZS): We investigate CLIP models [20] as our zero-shot models. CLIP models are pre-trained using image-text pairs { ( x 1 , t 1 ) , ..., ( x B , t B ) } from the Internet. The objective of the CLIP models is to train a visual encoder Φ v and a text encoder Φ t such that the cosine similarity < Φ v ( x i ) , Φ t ( t i ) > is maximized relative to unmatched pairs. CLIP models perform zero-shot inference for K classes by matching x with potential class names { c 1 , ..., c K } . Concretely, by extending the class name { c k } to a prompt ' t k = a photo of a { c k } ', the zero-shot model outputs the score (logit) for class k as f ( x ; θ zs ) k = < Φ v ( x ) , Φ t ( t k ) > . The predicted probabilities can be calculated using the softmax function, i.e. , ˆ P ( y | x ; θ zs ) = softmax ( f ( x ; θ zs )) y . The model outputs the label as pred ( f ( x ; θ zs )) = argmax i f ( x ; θ zs ) i

Linear classifiers (LC): We learn a linear classifier on top of the visual embedding Φ v ( x ) while freezing the visual encoder Φ v . The parameters of the linear classifier are optimized to minimize the cross-entropy loss on D .

End-to-end fine-tuning (E2E-FT): We update both the linear classifier and the visual encoder by minimizing the cross-entropy loss on D .

Linear probing then full fine-tuning [15] (LP-FT): We employ a two-phase fine-tuning approach: initially training a linear classifier, followed by full fine-tuning starting from the solution derived from training the linear classifier.

Output-space ensemble (OSE): We perform linear interpolation of the outputs between a zero-shot model and a fine-tuned model ( e.g. , E2E-FT or LP-FT):

Weight-space ensemble [28] (WSE). We combine the weights through linear interpolation between a zero-shot model and a fine-tuned model:

Algorithm 1 Variation Reduction Fine-tuning

1: Given : Training dataset D , a zero-shot model f zs and a fine-tuned model f ft .

2: Build zero-shot failure set V using Eq. (3).

3: Inference Stage:

4: Given a test sample x , compute its feature representation v , zero-shot prediction ˆ P zs ( y | x ) and fine-tuned model prediction ˆ P ft ( y | x ) .

5: Compute the k -NN distance to V as d ( x ) using Eq. (4).

6: Compute the weight ω ( x ) using Eq. (6).

7: Return ˆ P vrf ( y | x ) using Eq. (5)

3.2 Variance Reduction Fine-tuning

We now present our proposed method, VRF, which consists of three steps. First, before the inference stage, we collect the Zero-Shot Failure (ZSF) set. Second, for a given test sample, we calculate its distance to the ZSF set. Third, we assign weights to combine predictions from the zero-shot and fine-tuned models based on this distance.

Step 1 (Identification). For each x i in the training dataset D , if the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect its feature representation v i = Φ v ( x i ; θ ft ) from the fine-tuned model to form the zero-shot failure set V . Specifically, V is defined as:

̸

Here, f zs ( · ) and f ft ( · ) are used to denote f ( · ; θ zs ) and f ( · ; θ ft ) , respectively, for simplicity.

Step 2 (Distance Calculation). The key empirical observation underpinning VRF is that in the vicinity of the ZSF set, a test sample typically exhibits lower zero-shot accuracy (Acc zs ) and higher fine-tuned accuracy (Acc ft ). Consequently, the Acc ft Acc zs ratio demonstrates a monotonic decrease as the distance from the sample to the ZSF set increases. In this paper, we adopt non-parametric density estimation using nearest neighbors [24] to measure the distance of a test sample to the dataset V . Specifically, during inference, we derive the feature representation v of a test sample x , and compute the ℓ 2 distances ∥ v -v i ∥ 2 w.r.t. v i ∈ V . We reorder V according to the increasing ℓ 2 distance and denote the ordered set in sequence as V ′ = ( v (1) , v (2) , ..., v ( |V| ) ) . The distance of x to V is defined as the ℓ 2 distance to the k -th nearest neighbor ( k -NN), i.e. ,

If there is no ambiguity, we use d ( x ) to denote d ( x ; V , k ) for readability. Since the features in CLIP models are ℓ 2 normalized, d ( x ) are bounded between [0 , 2] .

Step 3 (Sample-Wise Ensembling). We implement sample-wise out-space ensembling in the form:

where ω ( x ) ∈ (0 , 1) . We use the distance to ZSF set d ( x ) to determine the weight ω . As shown by the blue line in Fig 2, a smaller value of d ( x ) corresponds to a larger Acc ft Acc zs ratio, and vice versa. Therefore, we set the weight ω to be inversely proportional to d ( x ) . Given that ω is bounded between 0 and 1 , we employ a sigmoid function σ ( · ) as:

where a, b > 0 are two hyper-parameters sweeped using accuracy on ID validation set. We visualize the weight curve in green on Fig 2, setting a = 1 . 5 and b = 0 . 6 . We summarize the whole process in Algorithm 1.

4 Justification

We now prove that our VRF can effectively reduce the variance of the combined model, resulting in lower errors compared to ensembling using a constant mixing coefficient.

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

Distance to ZSF set (

d

(

x

))

0.8

1.0

1.2

1.4

Fine-Tuned Acc / Zero-Shot Acc

(

x

),

a

=1.5,

b

=0.6

0.45

0.50

0.55

0.60

0.65

0.70

0.75

weight of FT model

Figure 2: Relationship between Acc ft Acc zs and the weight ω ( x ) .

▷ Step 2: Distance Calculation

▷ Step 3: Sample-Wise Ensembling

▷ Step 1: Identification

4.1 Background

The outputs of a well trained classifier are expected to approximate the a posterior class distribution. Apart from the irreducible error (Bayes error), the residual error of a classifier can be broken down into bias and variance components. In specific, for a test sample x , the probability output of a classifier parameterized by θ can be expressed as:

where P ( y | x ) denotes the true a posterior distribution, β y is the label bias of ˆ P ( y | x ; θ ) which is independent to the input x , and η y ( x ) is related to the given input x . In this study, we primarily attribute the residual error to the variance term ( i.e. , β y = 0 ), as the label bias problem in foundation models has been effectively addressed by Zhu et al. [31]. Tumer et al. [26] have proven that the expected residual error E is given by:

where s is a constant factor related to the derivative of the true a posterior distribution and is independent of the trained model, and V [ η y ( x )] is the variance.

4.2 Variance Reduction Fine-tuning Leads to Lower Residual Error

Let us shift our focus to the effects of combining the zero-shot and fine-tuned models. Let g zs ( · ) and g ft ( · ) be two functions that produce weights for ensembling the models. Subject to the constraint that g zs ( x ) + g ft ( x ) = 1 , the residual error of the combined classifier is obtained by:

where we omit the subscript y of η for readability. The variance of η vrf ( x ) can be expressed as:

Here, we assume the residual errors are independent following the assumption of the previous studies of CLIP fine-tuning [14, 31]. We further explore the case of correlated residual errors in Section B. According to Eq. (8), the reduction in variance can be readily translated into a reduction in error rates. To obtain the smallest variance V [ η vrf ( x )] , we minimize Eq. (10) using Lagrange multiplier to enforce the constraint that g zs ( x ) + g ft ( x ) = 1 , and obtain the optimal weight function g ft as:

Since Acc ft Acc zs ∝ d ( x ) -1 (a smaller distance d ( x ) is associated with a larger Acc ft Acc zs as shown in Fig. 2), we design the weighting function g ft ( x ) = ω ( x ) ∝ d ( x ) -1 as in Eq. (6).

5 Experiments

5.1 Experimental Setup

Datasets with distribution shifts. We provide the results for ImageNet [3] and its five derived distribution shifts: (1) ImageNet-V2 (IN-V2) [21]: Test images sampled after a decade of the original ImageNet. (2) ImageNet-R (IN-R) [7]: Contains renditions ( e.g. , art, cartoons, graffiti). (3) ImageNet Sketch (IN-Sketch) [27]: Consists of sketches rather than natural photos. (4) ImageNet-A (IN-A) [9]: Collects real-world images that are misclassified by ResNet models. (5) ObjectNet [1], a test set featuring objects with diverse backgrounds, rotations, and imaging viewpoints. We extend our analysis to include a standard distribution shift benchmark [15, 14, 4]: CIFAR-10 → STL-10, where the ID is CIFAR-10 [13], and the OOD is STL-10 [2]. We removed the 'monkey' class from STL-10, as it does not exist in CIFAR-10. In addition, we also consider subpopulation shifts, where the ID data contains a few sub-categories, and the OOD data comprises different sub-categories within the

same parent category. Following [15, 14], we adopt Entity30 dataset [23], which aims to categorize images into one of 30 entity categories, such as 'vehicle' and 'insect'.

Baselines. We adopt two models: CLIP ViT-B/32 and a larger ViT-B/16 from OpenAI [20]. The default model used in ablation studies is the CLIP ViT-B/16. In addition to the zero-shot models, we compare our approach against five standard methods for adapting pre-trained models: (1) linear classifier [20], (2) E2E-FT, (3) LP-FT [15], (4) OSE, and (5) WSE [28]. The descriptions of these methods have been included in Section 3.1.

Implementation details. When fine-tuning E2E-FT models, we adhere to Wortsman et al. [28], employing the default PyTorch AdamW optimizer for 10 epochs with weight decay of 0.1 and a cosine-annealing learning rate schedule with 500 warm-up steps. Unless specified, we use a learning rate of 3 × 10 -5 , gradient clipping at norm 1. When fine-tuning LP-FT, we first adopt the settings of Wortsman et al. [28] to train the linear classifier, then full fine-tune the models at a learning rate of 1 × 10 -5 . For efficiently performing k -NN search, we use Faiss library [11]. Denote the size of the ZSF set is |V| , we scale the k according to a percentage p % of the sample set, where k = floor ( p % · |V| ) . In this paper, p is set to 0 . 1% , a value consistent with the default setting proposed by Sun et al. [24]. Note that all the hyperparameters, e.g. , α, a, b , are searched using the accuracy on the in-distribution (ID) validation set. Derived distribution shift datasets are only for evaluation and not for hyperparameter sweeps. See Appendix C.1 for the details of experimental details.

(a) CLIP ViT-B/32

(b) CLIP ViT-B/16

CIFAR-10

STL-10

→

(b.1)

(a.1)

Entity-30

(a.2)

(b.2)

5.2 Results

ImageNet and its five shifted distribution results. In Table 1 and 2, we report the ID-OOD accuracies of fine-tuning baselines for CLIP ViT-32 and CLIP ViT-16 models, respectively. For OSE and WSE, we choose the mixing coefficient α with the highest ID validation accuracy. To enhance clarity in the results, we denote the improvement over OSE as ∆ in Tables 1 and 2. We observe that our VRF boosts the accuracy of fine-tuned models, including ensembling baseline models, across five ImageNet distribution shifted datasets, while maintaining or improving the ImageNet in-distribution performance. For instance, in Table 1, when ensembling with the E2E-FT model, our VRF outperforms the OSE model by 2 . 0% on distribution shifts while increasing the ID accuracy by 0 . 3% . Compared to WSE models, our VRF achieves a delta of 1 . 2% on distribution shifts, while maintaining ID performance within 0 . 2% , as shown in E2E-FT part of Table 2.

CIFAR-10 → STL-10 and Entity-30 results. We report the accuracy of various methods in Table 3 (a,b). We note that fine-tuning baselines can enhance the accuracy on CIFAR-10 compared to the zero-shot models. However, this improvement comes at the expense of reduced accuracy on STL-10. For instance, E2E-FT leads to a decrease of approximately 3 . 6% in STL-10 accuracy, as shown in Table 3(a). Previous ensemble methods can mitigate the degradation to some extent, but the STL-10 performance still lags behind the zero-shot performance, e.g. , In Table 3(b), the accuracy of E2E-FT + WSE is 97 . 8% whereas the zero-shot performance is 98 . 4% . In contrast, our VRF simultaneously improves accuracy on both CIFAR-10 and STL-10. Similarly, for Entity-30, our VRF can further improvement the OOD performance when compared to WSE and OSE methods.

In addition, we plot the ID-OOD frontier curves in Figure 3 (a.1&b.1), respectively. Similar to the results on ImageNet (Figure 1(a)), the ensemble model achieves its best ID and OOD performances at different α values. For instance, on the CIFAR-10 benchmark, when the ensemble model attains its optimal ID value at α = 0 . 7 , the OOD performance decreases by 2 . 0% relative to its peak.

Conversely, when the optimal OOD value is reached at α = 0 . 3 , the performance on ID diminishes by 2 . 7% from its best. In contrast, our VRF simultaneously attains the ID and OOD performance.

We also analyze the relation between the ratio Acc ft Acc zs and d ( x ) in Figure 3 (a.2&b.2). Consistent with the findings from ImageNet (Figure 1 (b)), we observe that the ratio decreases as d ( x ) increases, which further supports our design of assigning a higher weight to fine-tuned models if d ( x ) is smaller.

5.3 Further Analysis and Ablation Studies

VRF for linear-probed models. A drawback of the proposed method is its doubled inference and storage cost compared to WSE and other single-model robust fine-tuning methods. To address concerns regarding space-time complexity, we apply our VRF method to linear-probed models and present the results in Table 4. We also compare with output-space ensembling, since the model is linear, it is equivalent to weight-space ensembling. We also compare it with output-space ensembling, which, given the linear nature of the model, is equivalent to weight-space ensembling. Consistent with the conclusions drawn from fully fine-tuned models, our VRF method further improves OOD performance while maintaining comparable ID performance to OSE/WSE ensembling.

Using ZSF set V or entire training set D ? In Step 1 of our VRF, we define the zero-shot failure set V and use it to compute distances. We aim to find out whether using the entire training set D offers comparable performance. In Figure 4, we plot the Acc ft Acc zs curves and report both ID and OOD accuracy using the two sets. We observe that the ratio curve using D does not exhibit a monotonic trend with d ( x ) : it initially increases and then decreases as d ( x ) increases. Furthermore, the ratio Acc ft Acc zs using D is less informative when d ( x ) is smaller than 1.2, as the curve relatively remains flat. As the zero-shot models can accurately predict a large proportion of the ID data (recall that

Acc.

ID

OOD

All data

82.2

61.0

ZSF

82.3

61.8

Distance to di

ff

erent datasets

.

d

FT Accuracy / ZS Accuracy

(

x

)

the zero-shot accuracy is 68 . 3% ), a smaller distance to entire training set D does not reliably indicate whether the fine-tuned model can make more accurate predictions. In comparison, our ZSF set contains only the samples where zero-shot models fail but fine-tuned models succeed. When a sample is close to V , it is more likely that the accuracy ratio will be high. Consequently, the performances using D are clearly outperformed by those using V .

Comparison with selective prediction using OOD detector. A simple approach to address the ID-OOD trade-offs is to use an OOD detector for selective prediction. The OOD detector is a binary classifier G λ ( · ) to decide whether a sample is ID or OOD based on a threshold λ . For a test sample, predictions are made with the fine-tuned model if classified as ID, and with the zero-shot model otherwise:

(b)

(a)

(c)

p

(a)

(b)

(c)

where instances with higher scores S ( x ) are classified as ID and vice versa. λ is typically chosen to achieve achieve a 95% true positive rate for ID samples. We report the results of several implementations of S ( x ) in Table 5 (Details are in Section C.6). We note that selective prediction achieves comparable ID performance to E2E-FT models and similar OOD performance to zero-shot models. However, its accuracy still falls significantly short of our VRF. This is because traditional OOD detectors are designed for scenarios where the OOD data have a completely disjoint label space from the ID data, i.e. , Y OOD ∩ Y ID = ∅ . However, in our setup, the zero-shot models show predictive power on ID data, and the fine-tuned models are effective on OOD data. Making binary selections may overlook the complementary knowledge from the other model. Instead, our weight function ω ( x ) intelligently selects the contribution of each model based on the distance to the ZSF set. Another reason why our method outperforms selective prediction is the effective use of the ZSF set, as illustrated in Figure 4. Directly using all ID data as traditional OOD detectors (e.g., kNN and MD) leads to a weak correlation between the accuracy ratio and the distance d ( x ) (or score S ( x ) )

Examination of the averaged weight for ID and OOD test sets. Figure 5(a) shows the average weight ( E x [ ω ( x )] ) of the E2E-FT model in ensembling for both ID and OOD test sets. As expected, higher average weights are observed in the ID test set, as the fine-tuned models excel in such domain.

Logits-based ensembling. In this paper, we implement OSE by linearly interpolating the probabilities of the two models. Another common strategy for ensembling, known as Logits-Space Ensembling (LSE), involves interpolating in the logits space: f ( x ; θ lse ) = αf ( x ; θ ft ) + (1 -α ) f ( x ; θ zs ) . We aim to investigate whether our VRF can enhance the robustness of LSE without compromising the ID accuracy. The results depicted in Figure 5(b) confirm that our VRF can indeed generalize to LSE.

Effect of k in k -NN distance. In Figure 5(c), to compute k = floor ( p · |V| ) , we vary p across the range { 0 . 0001% , 0 . 01% , 0 . 05% , 0 . 1% , 10% , 50% } . We note two observations: (1) Varying k slightly affect the ID performance: the fluctuations are less than 0 . 1% . (2) The OOD accuracy declines as p increases, but the degradation is very slight for relative small values of p ( e.g. , when p < 0 . 01% , the decline is smaller than 0 . 2% ). In our implementation, we use the k -th nearest sample instead of the nearest one to reduce the potential impact of label noise. If the nearest sample is mislabeled, the distance may be unreliable. The k -th sample, being in a higher-density region, offers more stable distance estimates with lower variance, as it lies between the ( k -1) -th and ( k +1) -th samples. This makes the measure more robust to outliers. Additionally, prior research [24] shows that using the k -th nearest distance improves density estimation, which we adopt here.

Smallest d ( x )

Largest d ( x )

Inference speed of computing k -nearest neighbor distance. Thanks to the Faiss library [11], the k -NN search can be efficiently implemented. When evaluated on ImageNet benchmarks using CLIP ViT/B-16 features, the inference speed is approximately 1 . 8 milliseconds per-image, which does not significantly improve the inference time. In Figure 8, we further present the per-image inference speed of the k -nearest neighbor distance computation for various k values. The inference speed is less than 2 ms when k < 512 .

Effect of a and b in ω . We demonstrate the effect of a and b in Figure 6 (a&b). We highlight three trends: (1) ID performance

0

2

4

6

8

log2(k)

1.3

1.4

1.5

1.6

1.7

1.8

1.9

Inference Time (ms)

peaks at b ≈ 0 . 6 across different values of a . (2) OOD performance often improves as b increases across different values of a . (3) When b is sufficiently large, e.g. , b > 10 for ID and b > 2 for OOD, a has marginal effect on the performance of ID and OOD. In Appendix C.2, we further plot the trade-offs when tuning a and b .

Other designs of ω ( x ) . We further explore alternative designs of ω ( x ) beyond the sigmoid format in Eq. (6):

Binary weight: ω binary ( x ) = 1 [ d ( x ) < a ] , where a ∈ [0 , 2] and 1 [ · ] is the indicator function.

Linear weight: ω linear ( x ) = clamp [0 , 1] ( -b · ( d ( x ) -a )) , where a ∈ [0 , 2] , b > 0 and clamp [0 , 1] ( · ) rectifies the weight within [0 , 1] .

We report the results on ImageNet in Table 6 and plot the weight curves with the value of hyperparameters in Figure 6(c). We find that the Linear and the Sigmoid weights show comparable performance and assign similar values of ω around d = 1 . 5 .

Visualization of samples x according to d ( x ) . In Figure 7, we randomly sample testing images with the top-100 smallest d ( x ) values in the range [0 . 40 , 0 . 62] and the top-100 largest d ( x ) values in the range ∈ [1 . 59 , 1 . 63] . Interesting, we observe that: (1) Samples with the smallest d ( x ) predominantly consist of fine-grained species, e.g. , 'Triturus vulgaris', 'eft' and 'lycaenid', where the fine-tuned models possess domain-specific knowledge, which is often lacking in the zero-shot models. (2) Images with the largest d ( x ) exhibit styles different from those of the fine-tuning samples, including tattoos, cartoons, and sketches, contrasting with the photos typically seen in fine-tuning. Zero-shot models are more skilled in non-real photo styles compared to fine-tuned models.

6 Impact, limitations and conclusion

Impact. Zero-shot models inherit the weaknesses from pre-training data to downstream tasks, such as noisy and malicious samples. Our VRF might propagate the negative impact.

Limitations. Our approach is built on the premise that zero-shot models posses predictive capabilities for downstream tasks. However, if the pre-training knowledge significantly differs from the downstream tasks, our algorithm might fail, which is also an open problem in transfer learning. In addition, the proposed method doubles inference cost compared to WSE and other fine-tuning methods, as it runs both the zero-shot and fine-tuned models. However, this overhead can be mitigated by parallel execution.

Conclusion. Inspired by the ID-OOD trade-offs in ensemble-based fine-tuning, we propose VRF to simultaneously optimize the best ID and OOD accuracy. By leveraging the distance to the ZSF set, we assign sample-wise weights to the two models. Despite its simplicity, our VRF demonstrates strong empirical performance, offering a promising technique for solving ID-OOD trade-offs.

Acknowledgments

This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-01-002).

References

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS , 2019.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS , 2011.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In ICLR , 2018.

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR , 2023.

Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. Anchor-based robust finetuning of vision-language models. arXiv preprint arXiv:2404.06244 , 2024.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR , 2021.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2016.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data , 7(3):535-547, 2019.

Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 196-205, 2021.

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical Report , 2009.

Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI . PMLR, 2022.

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR , 2022.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018.

Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. In ICLR , 2023.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. arXiv preprint arXiv:2404.00860 , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML , 2019.

Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021.

Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. ArXiv , abs/2008.04859, 2020.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In NeurIPS , 2020.

Kagan Tumer and Joydeep Ghosh. Analysis of decision boundaries in linearly combined neural classifiers. Pattern recognition , 29(2):341-348, 1996.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurIPS , 2019.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR , 2022.

Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. In ICLR , 2021.

Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang. Debiased fine-tuning for vision-language models by prompt regularization. In AAAI , 2023.

Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS , 2023.

Contents

A Licenses

All the datasets we considered are publicly available, we list their licences and URLs as follows:

CIFAR-10 [13]: MIT License, https://www.cs.toronto.edu/~kriz/cifar.html .

STL-10 [2]: Non-commercial, https://cs.stanford.edu/~acoates/stl10/ .

Entity-30 [23]: Non-commercial, https://github.com/MadryLab/BREEDS-Benchmarks .

ImageNet [3]: Non-commercial, http://image-net.org .

IN-V2 [21]: MIT License, https://github.com/modestyachts/ImageNetV2 .

IN-R [7]: MIT License, https://github.com/hendrycks/imagenet-r .

IN-Sketch [27]: MIT License, https://github.com/HaohanWang/ImageNet-Sketch .

zs

IN-A [9]: MIT License, https://github.com/hendrycks/natural-adv-examples .

ObjectNet [1]: Creative Commons Attribution 4.0, https://objectnet.dev .

B Analysis in the Presence of Correlated Errors

Our assumption of independent residual errors is based on the previous studies [28] (Section 5), where an empirical phenomena is observed: the zero-shot and the fine-tuned models produce diverse predictions. In general ( i.e. , the fine-tuned models are initialized from the zero-shot models), we cannot assume that the errors in the zero-shot and fine-tuned models are totally uncorrelated. Let C [ η zs ( x ) , η ft ( x )] be the covariance between η zs ( x ) and η ft ( x ) , the variance of η vrf ( x ) can be expressed as:

Maintaining that g zs ( x ) + g ft ( x ) = 1 , the optimal weight g ∗ ft ( x ) to minimize Eq. (14) becomes:

Recall that we are interested in using the distance to ZSF set, i.e. , d ( x ) , to surrogate g ∗ ft ( x ) . To understand the relationship between d ( x ) and g ∗ ft ( x ) , we first group test samples in ImageNet and its five distribution shifted datasets into bins based on the value of d ( x ) . We then compute the averaged g ∗ ft ( x ) for each bin and plot the relationship in Figure 9. In specific, we use temperature scaling [14] to calibrate the zeroshot and fine-tuned models over the validation set. Afterwards, we define the true distribution P ( y | x ) as a one-hot vector, where the value of 1 corresponds to the true label for a given input

0.6

0.8

1.0

1.2

1.4

Distance to ZSF set (

d

(

x

))

0.375

0.400

0.425

0.450

0.475

0.500

0.525

gft

gft

x . We then calculate η ft ( x ) = ˆ P ( y | x ; θ ft ) -P ( y | x ) and η zs ( x ) = ˆ P ( y | x ; θ zs ) -P ( y | x ) . Finally, we compute the average g ∗ ft ( x ) for each bin as shown in Figure 9. Interestingly, we observe the similar trend in Figure 1 (b): the weight for fine-tuned models decreases as d ( x ) increases. This phenomena indicates that our weighting function ω ( x ) derived under the assumption of independent errors is also valid in the presence of correlated errors.

C Additional Experimental Details and Results

C.1 Additional Experimental Details

For CLIP ViT-32 based E2E-FT and LP-FT models, we use a batch size of 512. For CLIP ViT-16 based E2E-FT, we directly download the fine-tuned models from Wortsman et al. [28] 1 . The batch size for training CLIP ViT-16 based LP-FT models is set to 384, which is the largest batch size that fits into 2 A6000 GPUs. When performing linear probing, we use a batch of 512 and the initial learning rate of 0.1 for all experiments. The mixing coefficient α for OSE and WSE are searched over [0 , 0 . 1 , 0 . 2 , ..., 0 . 9 , 1 . 0] . The values of a and b for our VRF are reported in Table 7.

67

72

77

82

ImageNet Accuracy (ID)

53

58

63

Avg. Acc on 5 OOD datasets

OSE

VRF

a

=1.2

VRF

a

=1.3

VRF

a

=1.4

VRF

a

=1.5

VRF

a

=1.6

VRF

a

=1.7

VRF

a

=1.8

90

92

94

96

98

CIFAR-10 (ID)

96

98

STL-10 (OOD)

OSE with varing coefficient

WSE with varing coefficient

Zero-Shot

Fine-Tuned

VRF (ours)

67 72 77

82

87

92

97

ID

67

72

OOD

OSE with varing coefficient

WSE with varing coefficient

Zero-Shot

Fine-Tuned

VRF (ours)

C.2 Plotting ID-OOD Trade-Offs of VRF

In Figure 10, we present an ID-OOD scatter plot over a wide range of a and b values on ImageNet and its five variants using CLIP ViT-B/16. Specifically, a varies from 1.2 to 1.8, while b ranges from 0.5 to 1.0. Our method consistently achieves better ID-OOD trade-offs, as indicated by its points lying outside the OSE curve (represented by the magenta curve) across different configurations.

C.3 Additional Results for OSE, WSE and VRF with Varying Hyper-Parameters

Results for all mixing coefficient α for OSE and WSE are available in Table 8 and Table 9, respectively. Results for values of a and b are available in Table 10. In addition, we plot the ID-OOD trade-off curves for OSE and WSE on the CIFAR-10 and Entity-30 datasets in Figures 11 and 12, respectively.

C.4 Optimal Performance Searched on Test Sets

We have conducted additional experiments where we optimized the hyperparameters for each test set of the ImageNet benckmarks. The results are summarized in Table 11.

C.5 Combining VRF with Other Robust Fine-Tuning Methods

Our VRF framework is orthogonal and complementary to existing fine-tuned models. To demonstrate this, we integrated FLYP [5] into our VRF framework. The results in Table 12 show that VRF improves OSE's performance under distribution shift by 1.1% without compromising in-distribution (ID) performance.

C.6 Additional Results for Selective Prediction Using OOD Detectors

We provide the breakdown performance for selective prediction using OOD detectors in Table 13.

C.7 Curves of Acc ft Acc zs for ImageNet and its Five Distribution Shifted Datasets

In Figure 13, we examine the relationship between Acc ft Acc zs and d ( x ) for ImageNet and its five derived distribution shifted datasets. Based on the value of d ( x ) , test samples are grouped into bins, and we compute the ratio of fine-tuned accuracy to zero-shot accuracy for each bin. For example, to compute the value of Acc ft Acc zs at d ( x ) = 0 . 8 , we first identify the samples with d ( x ) ∈ [0 . 7 , 0 . 9] , then compute the averaged accuracy for these samples using zero-shot models and fine-tuned models, and finally

compute the ratio. Note that the averaged ratio Acc ft Acc zs on ImageNet-{A,R} and ObjectNet is undefined for d ( x ) = 0 . 6 . This is because in these datasets, the zero-shot accuracy around d ( x ) = 0 . 6 is 0. We observe that the trend of the ratio Acc ft Acc zs decreasing as d ( x ) increasing is stable for all ImageNet related datasets.

0.6

0.8

1.0

1.2

1.4

1.6

Distance to ZSF set (

d

(

x

))

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Fine-Tuned Acc / Zero-Shot Acc

ImageNet

ImageNetV2

ImageNetSketch

ImageNetA

ImageNetR

ObjectNet

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We point out that the ID-OOD trade-offs still exist in existing fine-tuning methods, and propose VRF to simultaneously attain the best ID and OOD accuracy. Experiments on a variety of different models and tasks validate the effectiveness of our proposed method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations have been discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 4, we prove that our VRF can effectively reduce the variance of the ensemble model and thus achieve lower errors.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the VRF algorithm in Algorithm 1 with descriptions in Section 3, and included the implementation details in Section 5 and Section C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded the codes in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:We have provided all the training and testing details in Section 5 and Section C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Given the zero-shot and the fine-tuned models, the process of our post-hoc method is deterministic. Run multiple times will not introduce randomness.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the GPU type and number to reproduce the results in Section C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the societal impacts in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method aims to improve the robustness when fine-tuning models, which poses no such risks to the best of our knowledge.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided the licences of each dataset in Section A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the ID-OOD trade-off of the ensemble-based method under the pretrain-finetune regime. The authors propose a simple yet effective adaptive ensemble method that determines the ensemble coefficient based on the feature distance between a test sample and the zero-shot failure set. The proposed method has a theoretical foundation, and it consistently improves the OOD generalization performance compared to the vanilla ensemble baseline in diverse situations.\n\nSTRENGTHS:\n* The paper is well-written overall and easy to follow.\n* The research problem this work focuses on is important in the era of foundation models, and the proposed method to address this is very straightforward and reasonable.\n* The proposed method shows consistent performance improvement in numerous distribution shift setups, and the author provides an intuitive theoretical justification for the proposed method.\n* Besides the empirical success in terms of improved ID-OOD performance trade-off, the authors present extensive qualitative and quantitative analysis results that give numerous insights into this field.\n\nWEAKNESSES:\n* The method requires 1) accessibility to the entire training dataset and 2) distance computation between each test sample and the entire train sample during inference time. These limit its application to resource-constrained (in terms of accessibility, memory storage, and runtime) situations.\n* Limited implication of the theoretical result\n  * While the current theoretical analysis provides a justification for the optimal strategy for determining the ensemble coefficients, which is realized by the authors, it does not say anything about the relative generalization error between the proposed method and other non-ensemble-based fine-tuning methods. \n* Lack of possible comparison with more advanced fine-tuning [4,5,6,7] or ensemble [1,2,3] methods.\n  * While the proposed approach has its unique implication compared to the vanilla ensemble, missing comparison with advanced baselines raises concern about the practical utility of the proposal.\n\n---\n> Reference\n* [1] SIMPLE: Specialized Model-Sample Matching for Domain Generalization, Li et al. 2023\n* [2] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models, Zhu et al. 2023\n* [3] Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization, Mavromatis et al. 2024\n* [4] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, Kumar et al. 2022\n* [5] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et al. 2023\n* [6] Trainable Projected Gradient Method for Robust Fine-tuning, Tian et al. 2023\n* [7] Towards Calibrated Robust Fine-Tuning of Vision-Language Models, Oh et al. 2024\n\nQUESTIONS:\n* Could the proposed method also be applied to the weight-space ensemble rather than the output-space ensemble?\n* How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?\n\nLIMITATIONS:\nThe authors adequately stated the limitation in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study examines the robust fine-tuning of CLIP models. The proposed method, Variance Reduction Fine-tuning (VRF), employs a sample-wise ensembling technique to enhance both ID and OOD accuracy, reducing trade-offs between them. Experimental results on ImageNet and associated distribution shifts empirically validate the effectiveness of this approach.\n\nSTRENGTHS:\n- The proposed approach is straightforward, combining outputs from zero-shot and fine-tuned models through output-space ensembling. Unlike conventional methods that use uniform or predefined weighting coefficients, the notable aspect here is the use of different weighting coefficients for each individual instance.\n\n- It is clear that adjusting these weighting coefficients according to whether zero-shot or fine-tuned models are suitable for specific data can improve the final performance. Consequently, the primary challenge lies in devising an effective method to determine such instance-specific weighting coefficients. The authors propose a method that makes use of the Zero-Shot Failure (ZSF) set, consisting of training examples where the zero-shot model fails to predict accurately but the fine-tuned model succeeds. Experimental validation confirms the efficacy of the proposed approach.\n\n- I appreciate Section 5.3 for further presenting an empirical analysis of the proposed method. Essentially, the approach is a variation of OSE, determining how much the zero-shot model and the fine-tuned model are used on a per-instance basis, and VRF is not the only strategy to achieve this. It is gratifying to see that VRF outperforms other alternatives and that the analysis examines how different VRF designs produce varying outcomes.\n\nWEAKNESSES:\n- Although there has been considerable research on robust fine-tuning for CLIP (as noted by the authors referencing various studies), the baselines analyzed in Tables 1 and 2 are quite limited. A key advantage of the VRF method is its applicability to any (zero-shot, fine-tuned) pair, both retrospectively and prospectively. Therefore, applying VRF to other robust fine-tuning methods beyond FT and LP-FT would further demonstrate the effectiveness of the proposed VRF approach.\n\n- The proposed method has a drawback of needing more than double the inference cost compared to WSE and other robust fine-tuning methods that yield a single fine-tuned model, since it requires performing a forward pass through the entire network twice (once for zero-shot and once for fine-tuned models) and calculating the ZSF-based weighting coefficient. However, an analysis of such inference costs has not yet been performed. It is important to note that Wortsman et al. (2022) favored WSE over OSE for combining zero-shot and fine-tuned models, highlighting this choice not only for its improved performance but also for its lack of additional costs.\n\n- In the context of robust fine-tuning, we can only observe performance on in-distribution data. Therefore, the parameters \n$a$ and $b$ in the proposed Eq. (6) for sample-wise mixing coefficients should be determined based on the in-distribution validation set, as mentioned by the authors in lines 141 and 198-200. From Appendix C.1, it is understood that $\\\\alpha \\\\in \\\\{1.4, 1.5, 1.6\\\\}$ and $b \\\\in \\\\{0.5, 0.6, ..., 1.0\\\\}$ were considered. However, it seems that the range of values for $a$ and $b$ is too narrow. Based on the in-distribution results in Table 8, larger values for $a$ should be explored, but the authors did not do so. What happens if the value of $a$ is increased further? Would it still outperform OSE and WSE baselines?\n\nQUESTIONS:\n1. In combining zero-shot and fine-tuned models, WSE offers the advantage of requiring only half the inference cost compared to VRF by maintaining inference costs equivalent to that of a single model through weight averaging. When combining zero-shot and linear-probed models, since the image encoder part, which carries most of the inference cost, only needs to be processed once, it appears that VRF's concerns regarding cost could be mitigated. In this context, could the proposed VRF method also extend to \"Linear classifier\" models, alongside \"E2E-FT\" and \"LP-FT\"? It is worth noting that Wortsman et al. (2021) explores ensembling not only between zero-shot models and end-to-end fine-tuned models (\"E2E-FT\" in this work) but also between zero-shot models and linear classifier fine-tuned models (referred to as \"Linear classifier\" here).\n\n2. In Figures 3, does \"Ensembling with varying coefficient $\\alpha$\" refer to OSE? It would be helpful to also provide the curve for WSE.\n\n3. Table 7 shows the results of exploring $\\alpha$ values for OSE. It would be beneficial to provide the same results for WSE as well.\n\n4. The proposed VRF approach also produces multiple points on trade-off plots (e.g., Figure 3(a.1)) depending on the values of $a$ and $b$ (especially $a$). It would be advantageous to visualize a scatter plot across a broad range of $a$ and $b$ values, allowing us to observe the trade-off curve similar to OSE and WSE, which interpolate between zero-shot and fine-tuned models.\n\nLIMITATIONS:\nThe authors addressed the limitations and potential negative societal impact in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper studies the trade-off between in-distribution (ID) and out-of-distribution (OOD) performance of pre-trained models before and after fine-tuning. The authors observed that the sample distance is inversely proportional to $\frac{Acc_{ft}}{Acc_{zs}}$. After modeling the residual error of the model, they found that for a training sample $x$, the optimal ensemble weight is proportional to the sample's accuracy. Therefore, the authors set different ensemble weights for different samples based on the distance between the samples and the erroneous samples in the training set. Experimental results demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. This paper effectively improves the model's ID-OOD performance by setting different ensemble weights for different sample.
2. Experiments show that this method can be effectively applied to different fine-tuning techniques and can significantly enhance performance.

WEAKNESSES:
1. VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations.
2. The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in $V$, or is it clustering the representations in $V$ first and then calculating the distance to the $k$-th cluster center?
3. Is the representation $ v $ of $ x $ calculated by ft or zs?
4. For each sample, recalculating and sorting the distance to $V$ incurs additional inference costs. How much extra computational cost does this introduce?
5. Since line 278 indicates that different values of $k$ have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the $k$-th sample for distance calculation?

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to tackle the ID-OOD trade-off in the fine-tuning of pre-trained models with zero-shot abilities like CLIP. The proposed Variance Reduction Fine-tuning (VRF) is a sample-wise ensembling method concerning the zero-shot and fine-tuned models. The ensemble weights are determined by the distance from the test sample to the training samples that are incorrectly predicted by the zero-shot model. Theoretical analysis and experimental results justify the effectiveness of VRF.\n\nSTRENGTHS:\n1. The proposed VRF can achieve better performance on both ID and OOD data compared with existing ensemble-based methods.\n2. Unlike previous methods, the ensemble weights in VRF does not require tuning.\n3. The experiments are thorough, and the results are clearly presented in the tables and figures.\n\nWEAKNESSES:\n1. A major downside of VRF compared to previous ensemble methods is that the determination of the ensemble weights requires the storage of ZSF features at test time and computation of the distance from the test samples to each sample in ZSF set. This may bring a significant budget when the ZSF set is large. There should be a discussion on the theoretical and practical computational complexity regarding space and time.\n2. The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF.\n3. It is claimed that the proposed method “can simultaneously attain the best ID and OOD accuracy without the trade-offs” (Line 9). However, this is not sufficiently justified. Specifically, it is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.\n\nQUESTIONS:\n1. To what extent is VRF less efficient than existing ensembling methods?\n2. Why is k-th nearest neighbor distance selected for the distance calculation step of VRF?\n3. What is the meaning of solving the ID-OOD trade-offs?\n\nLIMITATIONS:\nThe authors have addressed some limitations and potential impacts of the work. I suggest additional discussions on the efficiency of the proposed method in the limitation part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of ID-OOD (In-Distribution vs. Out-of-Distribution) in the context of robust fine-tuning techniques commonly used in ensemble methods. The authors propose a sample-wise mixing approach to resolve this problem. The method involves creating a Zero-Shot Failure set, which contains samples that fail in zero-shot models but succeed in fine-tuned models, and mixing the predictions of the zero-shot and fine-tuned models based on the distance between test samples and the Zero-Shot Failure set. This approach demonstrates superior performance over traditional weight-ensemble and prediction ensemble methods on the ImageNet variants benchmark.\n\nSTRENGTHS:\n- The paper is well-written, clearly explaining the proposed method and its motivation. The intuitive explanation, especially the use of Figure 2 to show the monotonic relationship between fine-tuned/zero-shot accuracy ratio and distance, strongly supports the proposed approach.\n- The paper makes a novel contribution by addressing the ID-OOD problem, which is not fully addressed by the existing ensemble-based robust fine-tuning.\n- The proposed method is simple and easy to implement, yet it achieves better performance compared to existing ensemble methods.\n\nWEAKNESSES:\n- The main drawback of the proposed method is the reliance on multiple hyperparameters (a, b, p), making tuning more complex compared to traditional ensemble methods that typically involve only one hyperparameter. Finding optimal values for these hyperparameters is challenging as they can vary significantly across different datasets, potentially affecting performance.\n- To demonstrate the robustness of the proposed method, evaluations should be extended to more datasets beyond the ImageNet variants, similar to the experiments in the WiSE-FT paper, which include datasets like iWILDCam and FMoW. Differences in hyperparameter sweep ranges across datasets could highlight potential issues with the method.\n- There is a lack of performance comparison with other robust fine-tuning methods, such as FLYP[1] and MaskFill[2], which limits the assessment of the proposed method’s effectiveness. In the ImageNet classificaiton experiments, the proposed method is inferior to [1,2] in performance.\n- The paper need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity, providing deeper insights into the robustness of the proposed method.\n\n[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et. al., CVPR 23\n[2] Masked images are counterfactual samples for robust fine-tuning, Xiao et. al., CVPR 23\n\nQUESTIONS:\nI am curious about the performance when finding hyperparameters for each test set as an oracle result.\n\nLIMITATIONS:\nThe authors have adequately addressed some limitations of their work. However, they have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}