Robust Fine-tuning of Zero-shot Models via Variance Reduction

Authors: beier  zhu
jiequan  cui
hanwang  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 10:36:54

Abstract: When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains on (0.9 - 3.1 pp) on other distribution shifts
19 benchmarks. Codes are available in https://github.com/BeierZhu/VRF.

Main paper: 
 introduction: 
 To ensure the reliability of machine learning systems, it is essential to develop models that can generalize to unseen, out-of-distribution environments. Large pre-trained models such as CLIP [20] and ALIGN [10] have recently shown remarkable robustness against challenging distribution shifts. However, it is widely acknowledged that these improvements in robustness are most pronounced in the zero-shot setting, while conventional fine-tuning on these models often compromises robustness when compared to zero-shot performance [28, 15, 14] . This phenomenon is known as the ID-OOD trade-offs, i.e., improving performance on in-distribution (ID) data can sometimes lead to decreased performance on out-of-distribution (OOD) data [12, 25] .In recent years, ensemble-based models (ESMs) have demonstrated significant success in addressing the ID-OOD dilemma [17, 28, 14, 31] . Specifically, denote the input as x, the zero-shot model as P(y|x; θ zs ) and the fine-tuned model as P(y|x; θ ft ), existing ESMs typically employ the outputspace ensemble (OSE) [14, 31] , which outputs P(y|x; θ ose ) = α P(y|x; θ ft ) + (1 -α) P(y|x; θ zs ), and the weight-space ensemble (WSE) [28, 17] , which outputs P(y|x; θ wse ) = P(y|x; αθ ft + (1α)θ zs ), where α ∈ [0, 1]. Compared to fine-tuned models, ESMs offer significant accuracy enhancements under distribution shift, while maintaining high ID accuracy.However, ESM cannot fully address the ID-OOD trade-offs. In Figure 1 The ensemble model achieves its best ID and OOD performance at different α values. Our method VRF simultaneously attains the best ID and OOD accuracy, outperforming the ensemble by 3.6% on OOD and 1.6% on ID at its optimal performance points.(b) Relationship between the ratio of fine-tuned accuracy to zero-shot accuracy ( Acc ft Acczs ) and the distance to the zero-shot failure set (d(x)).Acczs demonstrates a monotonic decrease as d(x) increases.ImageNet [3] (ID) and five derived distribution-shifted datasets (OOD): ImageNet-V2 [21] , ImageNet-R [7] , ImageNet-A [9] , ImageNet-Sketch [27] and ObjectNet [1] . We find that the ensemble model achieves its optimal ID and OOD performance at different α values: the best ID accuracy is achieved at α = 0.5 and the best OOD accuracy is obtained at α = 0.3. When the ensemble model reaches its optimal value for OOD, the performance on ID decreases by 3.6% relative to its peak. Similarly, when the ensemble model is optimized for ID, the performance on OOD decreases by 1.6% relative to its best value -the ID-OOD trade-offs still persist for ESMs. This raises a natural question:Can ensemble-based models simultaneously attain the best ID and OOD accuracy?In this paper, we affirmatively answer this question by proposing a sample-wise ensembling technique, dubbed variance reduction fine-tuning (VRF). This method is motivated by an empirical finding illustrated in Fig 1 (b) . For each sample in the training dataset, if the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect its features representation in the finetuned model as the zero-shot failure (ZSF) set. We then measure the distance d(x) of each test sample x to the ZSF set. Based on this distance, test samples are grouped into bins, and we compute the ratio of fine-tuned accuracy to zero-shot accuracy: Acc ft Acczs for each bin (implementation details are in Section C.7). Interestingly, we observe that the ratio Acc ft Acczs monotonically decreases as d(x) increases. Intuitively, the closer a sample is to the ZSF set, the more likely it is that the zero-shot model makes incorrect predictions, whereas the fine-tuned model is more likely to be accurate, leading to a higher Acc ft Acczs ratio. Therefore, we use the distance to assign weights to the models: a smaller d(x) results in a higher weight for the fine-tuned model, and vice versa.As depicted by the orange diamond in Fig. 1 (a) , by leveraging the sample-wise weights, our VRF simultaneously attains the best ID and OOD accuracy. In Section 5, we show that on a variety of different models and tasks, our VRF approach consistently outperforms the existing fine-tuning and ensembling methods, including linear probing, end-to-end fine-tuning, LP-FT [15] , OSE and WSE [28] . In specific, on ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 -2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. Furthermore, in Section 4, we justify our approach by demonstrating that it effectively minimizes the variance of the ensemble models, resulting in reduced residual error. 
related work: 
 Mitigating ID-OOD trade-offs. Improving performance on in-distribution data can sometimes lead to a decrease in performance on out-of-distribution data, and vice versa. This phenomenon is known as the ID-OOD trade-offs. Xie et al. [29] leverage auxiliary information as outputs of auxiliary tasks to pre-train a model to reduce OOD error. Khani and Liang [12] show that self-training on large amounts of unlabeled data can mitigate such trade-offs by removing spurious features. Tripuraneni et al. [25] tackle this problem by learning representations that are robust across diverse tasks. However, these methods usually necessitate additional unlabeled data or auxiliary information. In contrast, our VRF is a straightforward variation of fine-tuning that does not require any extra data.Robust fine-tuning of zero-shot models. Vision-language models like CLIP [20] have demonstrated outstanding improvements in robustness. It is commonly acknowledged that conventional finetuning methods often compromise robustness when compared to zero-shot performance. Therefore, enhancing downstream robustness has been the focus of subsequent works [15, 28, 5, 19, 6, 30] . Kumar et al. [15] show that a two-process of linear probing followed by full fine-tuning can alleviate feature distortion, leading to stronger OOD performance without sacrificing ID accuracy. Wortsman et al. [28] propose a method of weight interpolation between the zero-shot and the fine-tuned models to improve both ID and OOD accuracy. Goyal et al. [5] demonstrate that mimicking the contrastive pre-training objectives to fine-tune the zero-shot models outperforms tuning via the traditional supervised cross-entropy loss. However, the ID-OOD trade-offs are still observed with these methods. In contrast, our method VRF can simultaneously achieve the best ID and OOD accuracy.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations have been discussed in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
set up: 
 Task: Consider a classification setting where the goal is to map an instance x ∈ X to a label y ∈ Y = [K]. We are provided with a zero-shot model f (•; θ zs ), a downstream dataset D = {x i , y i } N i=1 , and a fine-tuned model f (•; θ ft ) which is trained on D. Below, we outline the implementation of the zero-shot and fine-tuned models:• Zero-shot models (ZS): We investigate CLIP models [20] as our zero-shot models. CLIP models are pre-trained using image-text pairs {(x 1 , t 1 ), ..., (x B , t B )} from the Internet. The objective of the CLIP models is to train a visual encoder Φ v and a text encoder Φ t such that the cosine similarity < Φ v (x i ), Φ t (t i ) > is maximized relative to unmatched pairs. CLIP models perform zero-shot inference for K classes by matching x with potential class names {c 1 , ..., c K }. Concretely, by extending the class name {c k } to a prompt "t k =a photo of a {c k }", the zero-shot model outputs the score (logit) for class k as f (x;θ zs ) k =< Φ v (x), Φ t (t k ) >.The predicted probabilities can be calculated using the softmax function, i.e., P(y|x; θ zs ) = softmax(f (x; θ zs )) y . The model outputs the label as pred(f (x; θ zs )) = argmax i f (x; θ zs ) i• Linear classifiers (LC): We learn a linear classifier on top of the visual embedding Φ v (x) while freezing the visual encoder Φ v . The parameters of the linear classifier are optimized to minimize the cross-entropy loss on D.• End-to-end fine-tuning (E2E-FT): We update both the linear classifier and the visual encoder by minimizing the cross-entropy loss on D.• Linear probing then full fine-tuning [15] (LP-FT): We employ a two-phase fine-tuning approach: initially training a linear classifier, followed by full fine-tuning starting from the solution derived from training the linear classifier.• Output-space ensemble (OSE): We perform linear interpolation of the outputs between a zero-shot model and a fine-tuned model (e.g., E2E-FT or LP-FT):EQUATION• Weight-space ensemble [28] (WSE). We combine the weights through linear interpolation between a zero-shot model and a fine-tuned model:EQUATIONAlgorithm 1 Variation Reduction Fine-tuning 4: Given a test sample x, compute its feature representation v, zero-shot prediction Pzs (y|x) and fine-tuned model prediction Pft (y|x). 5: Compute the k-NN distance to V as d(x) using Eq. ( 4).▷ Step 2: Distance Calculation 6: Compute the weight ω(x) using Eq. (6). 7: Return Pvrf (y|x) using Eq. ( 5)▷ Step 3: Sample-Wise EnsemblingWe now present our proposed method, VRF, which consists of three steps. First, before the inference stage, we collect the Zero-Shot Failure (ZSF) set. Second, for a given test sample, we calculate its distance to the ZSF set. Third, we assign weights to combine predictions from the zero-shot and fine-tuned models based on this distance.Step 1 (Identification). For each x i in the training dataset D, if the fine-tuned model correctly predicts the label while the zero-shot model fails, we collect its feature representation v i = Φ v (x i ; θ ft ) from the fine-tuned model to form the zero-shot failure set V. Specifically, V is defined as:EQUATIONHere, f zs (•) and f ft (•) are used to denote f (•; θ zs ) and f (•; θ ft ), respectively, for simplicity.Step 2 (Distance Calculation). The key empirical observation underpinning VRF is that in the vicinity of the ZSF set, a test sample typically exhibits lower zero-shot accuracy (Acc zs ) and higher fine-tuned accuracy (Acc ft ). Consequently, the Acc ft Acczs ratio demonstrates a monotonic decrease as the distance from the sample to the ZSF set increases. In this paper, we adopt non-parametric density estimation using nearest neighbors [24] to measure the distance of a test sample to the dataset V. Specifically, during inference, we derive the feature representation v of a test sample x, and compute the ℓ 2 distances ∥v -v i ∥ 2 w.r.t. v i ∈ V. We reorder V according to the increasing ℓ 2 distance and denote the ordered set in sequence as V ′ = (v (1) , v (2) , ..., v (|V|) ). The distance of x to V is defined as the ℓ 2 distance to the k-th nearest neighbor (k-NN), i.e.,EQUATION)If there is no ambiguity, we use d(x) to denote d(x; V, k) for readability. Since the features in CLIP models are ℓ 2 normalized, d(x) are bounded between [0, 2]. Step 3 (Sample-Wise Ensembling). We implement sample-wise out-space ensembling in the form:EQUATIONwhere ω(x) ∈ (0, 1). We use the distance to ZSF set d(x) to determine the weight ω. As shown by the blue line in Fig 2 , a smaller value of d(x) corresponds to a larger Acc ft Acczs ratio, and vice versa. Therefore, we set the weight ω to be inversely proportional to d(x). Given that ω is bounded between 0 and 1, we employ a sigmoid function σ(•) as:EQUATIONwhere a, b > 0 are two hyper-parameters sweeped using accuracy on ID validation set. We visualize the weight curve in green on Fig 2 , setting a = 1.5 and b = 0.6. We summarize the whole process in Algorithm 1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have uploaded the codes in supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Given the zero-shot and the fine-tuned models, the process of our post-hoc method is deterministic. Run multiple times will not introduce randomness. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
justification: 
 We now prove that our VRF can effectively reduce the variance of the combined model, resulting in lower errors compared to ensembling using a constant mixing coefficient.The outputs of a well trained classifier are expected to approximate the a posterior class distribution. Apart from the irreducible error (Bayes error), the residual error of a classifier can be broken down into bias and variance components. In specific, for a test sample x, the probability output of a classifier parameterized by θ can be expressed as:EQUATIONwhere P(y|x) denotes the true a posterior distribution, β y is the label bias of P(y|x; θ) which is independent to the input x, and η y (x) is related to the given input x. In this study, we primarily attribute the residual error to the variance term (i.e., β y = 0), as the label bias problem in foundation models has been effectively addressed by Zhu et al. [31] . Tumer et al. [26] have proven that the expected residual error E is given by:EQUATION)where s is a constant factor related to the derivative of the true a posterior distribution and is independent of the trained model, and V[η y (x)] is the variance.Let us shift our focus to the effects of combining the zero-shot and fine-tuned models. Let g zs (•) and g ft (•) be two functions that produce weights for ensembling the models. Subject to the constraint that g zs (x) + g ft (x) = 1, the residual error of the combined classifier is obtained by:EQUATIONwhere we omit the subscript y of η for readability. The variance of η vrf (x) can be expressed as:EQUATIONHere, we assume the residual errors are independent following the assumption of the previous studies of CLIP fine-tuning [14, 31] . We further explore the case of correlated residual errors in Section B. According to Eq. ( 8), the reduction in variance can be readily translated into a reduction in error rates. To obtain the smallest variance V[η vrf (x)], we minimize Eq. ( 10) using Lagrange multiplier to enforce the constraint that g zs (x) + g ft (x) = 1, and obtain the optimal weight function g ft as:EQUATIONSince Acc ft Acczs ∝ d(x) -1 (a smaller distance d(x) is associated with a larger Acc ft Acczs as shown in Fig. 2 ), we design the weighting function g ft (x) = ω(x) ∝ d(x) -1 as in Eq. ( 6). 
experimental setup: 
 Datasets with distribution shifts. We provide the results for ImageNet [3] and its five derived distribution shifts: (1) ImageNet-V2 (IN-V2) [21] : Test images sampled after a decade of the original ImageNet. (2) ImageNet-R (IN-R) [7] : Contains renditions (e.g., art, cartoons, graffiti). (3) ImageNet Sketch (IN-Sketch) [27] : Consists of sketches rather than natural photos. (4) ImageNet-A (IN-A) [9] : Collects real-world images that are misclassified by ResNet models. ( 5) ObjectNet [1] , a test set featuring objects with diverse backgrounds, rotations, and imaging viewpoints. We extend our analysis to include a standard distribution shift benchmark [15, 14, 4] : CIFAR-10 → STL-10, where the ID is CIFAR-10 [13] , and the OOD is STL-10 [2] . We removed the "monkey" class from STL-10, as it does not exist in CIFAR-10. In addition, we also consider subpopulation shifts, where the ID data contains a few sub-categories, and the OOD data comprises different sub-categories within the same parent category. Following [15, 14] , we adopt Entity30 dataset [23] , which aims to categorize images into one of 30 entity categories, such as "vehicle" and "insect".Baselines. We adopt two models: CLIP ViT-B/32 and a larger ViT-B/16 from OpenAI [20] . The default model used in ablation studies is the CLIP ViT-B/16. In addition to the zero-shot models, we compare our approach against five standard methods for adapting pre-trained models: (1) linear classifier [20] , (2) E2E-FT, (3) LP-FT [15] , (4) OSE, and ( 5) WSE [28] . The descriptions of these methods have been included in Section 3.1.Implementation details. When fine-tuning E2E-FT models, we adhere to Wortsman et al. [28] , employing the default PyTorch AdamW optimizer for 10 epochs with weight decay of 0.1 and a cosine-annealing learning rate schedule with 500 warm-up steps. Unless specified, we use a learning rate of 3 × 10 -5 , gradient clipping at norm 1. When fine-tuning LP-FT, we first adopt the settings of Wortsman et al. [28] to train the linear classifier, then full fine-tune the models at a learning rate of 1 × 10 -5 . For efficiently performing k-NN search, we use Faiss library [11] . Denote the size of the ZSF set is |V|, we scale the k according to a percentage p% of the sample set, where k = floor(p% • |V|). In this paper, p is set to 0.1%, a value consistent with the default setting proposed by Sun et al. [24] . Note that all the hyperparameters, e.g., α, a, b, are searched using the accuracy on the in-distribution (ID) validation set. Derived distribution shift datasets are only for evaluation and not for hyperparameter sweeps. See Appendix C.1 for the details of experimental details. ImageNet and its five shifted distribution results. In Table 1 and 2, we report the ID-OOD accuracies of fine-tuning baselines for CLIP ViT-32 and CLIP ViT-16 models, respectively. For OSE and WSE, we choose the mixing coefficient α with the highest ID validation accuracy. To enhance clarity in the results, we denote the improvement over OSE as ∆ in Tables 1 and 2 . We observe that our VRF boosts the accuracy of fine-tuned models, including ensembling baseline models, across five ImageNet distribution shifted datasets, while maintaining or improving the ImageNet in-distribution performance. For instance, in Table 1 , when ensembling with the E2E-FT model, our VRF outperforms the OSE model by 2.0% on distribution shifts while increasing the ID accuracy by 0.3%. Compared to WSE models, our VRF achieves a delta of 1.2% on distribution shifts, while maintaining ID performance within 0.2%, as shown in E2E-FT part of Table 2 .CIFAR-10 → STL-10 and Entity-30 results. We report the accuracy of various methods in Table 3 (a,b). We note that fine-tuning baselines can enhance the accuracy on CIFAR-10 compared to the zero-shot models. However, this improvement comes at the expense of reduced accuracy on STL-10. For instance, E2E-FT leads to a decrease of approximately 3.6% in STL-10 accuracy, as shown in Table 3 (a). Previous ensemble methods can mitigate the degradation to some extent, but the STL-10 performance still lags behind the zero-shot performance, e.g., In Table 3 (b), the accuracy of E2E-FT + WSE is 97.8% whereas the zero-shot performance is 98.4%. In contrast, our VRF simultaneously improves accuracy on both CIFAR-10 and STL-10. Similarly, for Entity-30, our VRF can further improvement the OOD performance when compared to WSE and OSE methods.In addition, we plot the ID-OOD frontier curves in Figure 3 (a.1&b.1), respectively. Similar to the results on ImageNet (Figure 1(a) ), the ensemble model achieves its best ID and OOD performances at different α values. For instance, on the CIFAR-10 benchmark, when the ensemble model attains its optimal ID value at α = 0.7, the OOD performance decreases by 2.0% relative to its peak. Conversely, when the optimal OOD value is reached at α = 0.3, the performance on ID diminishes by 2.7% from its best. In contrast, our VRF simultaneously attains the ID and OOD performance.We also analyze the relation between the ratio Acc ft Acczs and d(x) in Figure 3 (a.2&b.2). Consistent with the findings from ImageNet (Figure 1 (b)), we observe that the ratio decreases as d(x) increases, which further supports our design of assigning a higher weight to fine-tuned models if d(x) is smaller.VRF for linear-probed models. A drawback of the proposed method is its doubled inference and storage cost compared to WSE and other single-model robust fine-tuning methods. To address concerns regarding space-time complexity, we apply our VRF method to linear-probed models and present the results in Table 4 . We also compare with output-space ensembling, since the model is linear, it is equivalent to weight-space ensembling. We also compare it with output-space ensembling, which, given the linear nature of the model, is equivalent to weight-space ensembling. Consistent with the conclusions drawn from fully fine-tuned models, our VRF method further improves OOD performance while maintaining comparable ID performance to OSE/WSE ensembling. Distance to different datasets .FT Accuracy / ZS Accuracy In Figure 4 , we plot the Acc ft Acczs curves and report both ID and OOD accuracy using the two sets. We observe that the ratio curve using D does not exhibit a monotonic trend with d(x): it initially increases and then decreases as d(x) increases. Furthermore, the ratio Acc ft Acczs using D is less informative when d(x) is smaller than 1.2, as the curve relatively remains flat. As the zero-shot models can accurately predict a large proportion of the ID data (recall that the zero-shot accuracy is 68.3%), a smaller distance to entire training set D does not reliably indicate whether the fine-tuned model can make more accurate predictions. In comparison, our ZSF set contains only the samples where zero-shot models fail but fine-tuned models succeed. When a sample is close to V, it is more likely that the accuracy ratio will be high. Consequently, the performances using D are clearly outperformed by those using V. Comparison with selective prediction using OOD detector. A simple approach to address the ID-OOD trade-offs is to use an OOD detector for selective prediction. The OOD detector is a binary classifier G λ (•) to decide whether a sample is ID or OOD based on a threshold λ. For a test sample, predictions are made with the fine-tuned model if classified as ID, and with the zero-shot model otherwise: where instances with higher scores S(x) are classified as ID and vice versa. λ is typically chosen to achieve achieve a 95% true positive rate for ID samples. We report the results of several implementations of S(x) in Table 5 (Details are in Section C.6). We note that selective prediction achieves comparable ID performance to E2E-FT models and similar OOD performance to zero-shot models. However, its accuracy still falls significantly short of our VRF. This is because traditional OOD detectors are designed for scenarios where the OOD data have a completely disjoint label space from the ID data, i.e., Y OOD ∩ Y ID = ∅. However, in our setup, the zero-shot models show predictive power on ID data, and the fine-tuned models are effective on OOD data. Making binary selections may overlook the complementary knowledge from the other model. Instead, our weight function ω(x) intelligently selects the contribution of each model based on the distance to the ZSF set. Another reason why our method outperforms selective prediction is the effective use of the ZSF set, as illustrated in Figure 4 . Directly using all ID data as traditional OOD detectors (e.g., kNN and MD) leads to a weak correlation between the accuracy ratio and the distance d(x) (or score S(x))EQUATIONEQUATIONExamination of the averaged weight for ID and OOD test sets. Figure 5 (a) shows the average weight (E x [ω(x)]) of the E2E-FT model in ensembling for both ID and OOD test sets. As expected, higher average weights are observed in the ID test set, as the fine-tuned models excel in such domain.Logits-based ensembling. In this paper, we implement OSE by linearly interpolating the probabilities of the two models. Another common strategy for ensembling, known as Logits-Space Ensembling (LSE), involves interpolating in the logits space: f (x; θ lse ) = αf (x; θ ft ) + (1 -α)f (x; θ zs ). We aim to investigate whether our VRF can enhance the robustness of LSE without compromising the ID accuracy. The results depicted in Figure 5 (b) confirm that our VRF can indeed generalize to LSE.Effect of k in k-NN distance. In Figure 5 (c), to compute k = floor(p • |V|), we vary p across the range {0.0001%, 0.01%, 0.05%, 0.1%, 10%, 50%}. We note two observations: (1) Varying k slightly affect the ID performance: the fluctuations are less than 0.1%. (2) The OOD accuracy declines as p increases, but the degradation is very slight for relative small values of p (e.g., when p < 0.01%, the decline is smaller than 0.2%). In our implementation, we use the k-th nearest sample instead of the nearest one to reduce the potential impact of label noise. If the nearest sample is mislabeled, the distance may be unreliable. The k-th sample, being in a higher-density region, offers more stable distance estimates with lower variance, as it lies between the (k -1)-th and (k + 1)-th samples. This makes the measure more robust to outliers. Additionally, prior research [24] shows that using the k-th nearest distance improves density estimation, which we adopt here. Inference Time (ms) Inference speed of computing k-nearest neighbor distance.Thanks to the Faiss library [11] , the k-NN search can be efficiently implemented. When evaluated on ImageNet benchmarks using CLIP ViT/B-16 features, the inference speed is approximately 1.8 milliseconds per-image, which does not significantly improve the inference time. In Figure 8 , we further present the per-image inference speed of the k-nearest neighbor distance computation for various k values. The inference speed is less than 2 ms when k < 512.Effect of a and b in ω. We demonstrate the effect of a and b in Figure 6 (a&b). We highlight three trends: Other designs of ω(x). We further explore alternative designs of ω(x) beyond the sigmoid format in Eq. ( 6):• Binary weight: We report the results on ImageNet in Table 6 and plot the weight curves with the value of hyperparameters in Figure 6 (c). We find that the Linear and the Sigmoid weights show comparable performance and assign similar values of ω around d = 1.5.ω binary (x) = 1[d(x) < a],Visualization of samples x according to d(x). In Figure 7 , we randomly sample testing images with the top-100 smallest d(x) values in the range [0.40, 0.62] and the top-100 largest d(x) values in the range ∈ [1.59, 1.63]. Interesting, we observe that: (1) Samples with the smallest d(x) predominantly consist of fine-grained species, e.g., "Triturus vulgaris", "eft" and "lycaenid", where the fine-tuned models possess domain-specific knowledge, which is often lacking in the zero-shot models. (2) Images with the largest d(x) exhibit styles different from those of the fine-tuning samples, including tattoos, cartoons, and sketches, contrasting with the photos typically seen in fine-tuning. Zero-shot models are more skilled in non-real photo styles compared to fine-tuned models. 
impact, limitations and conclusion: 
 Impact. Zero-shot models inherit the weaknesses from pre-training data to downstream tasks, such as noisy and malicious samples. Our VRF might propagate the negative impact.Limitations. Our approach is built on the premise that zero-shot models posses predictive capabilities for downstream tasks. However, if the pre-training knowledge significantly differs from the downstream tasks, our algorithm might fail, which is also an open problem in transfer learning. In addition, the proposed method doubles inference cost compared to WSE and other fine-tuning methods, as it runs both the zero-shot and fine-tuned models. However, this overhead can be mitigated by parallel execution. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our method aims to improve the robustness when fine-tuning models, which poses no such risks to the best of our knowledge.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided the licences of each dataset in Section A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not involve such experiments.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Inspired by the ID-OOD trade-offs in ensemble-based fine-tuning, we propose VRF to simultaneously optimize the best ID and OOD accuracy. By leveraging the distance to the ZSF set, we assign sample-wise weights to the two models. Despite its simplicity, our VRF demonstrates strong empirical performance, offering a promising technique for solving ID-OOD trade-offs. All the datasets we considered are publicly available, we list their licences and URLs as follows:• CIFAR-10 [13]: MIT License, https://www.cs.toronto.edu/~kriz/cifar.html.• STL-10 [2]: Non-commercial, https://cs.stanford.edu/~acoates/stl10/.• Entity-30 [23] : Non-commercial, https://github.com/MadryLab/BREEDS-Benchmarks.• ImageNet [3] : Non-commercial, http://image-net.org.• IN-V2 [21] : MIT License, https://github.com/modestyachts/ImageNetV2.• IN-R [7] : MIT License, https://github.com/hendrycks/imagenet-r.• IN-Sketch [27] : MIT License, https://github.com/HaohanWang/ImageNet-Sketch. Our assumption of independent residual errors is based on the previous studies [28] (Section 5), where an empirical phenomena is observed: the zero-shot and the fine-tuned models produce diverse predictions. In general (i.e., the fine-tuned models are initialized from the zero-shot models), we cannot assume that the errors in the zero-shot and fine-tuned models are totally uncorrelated.Let C[η zs (x), η ft (x)] be the covariance between η zs (x) and η ft (x), the variance of η vrf (x) can be expressed as:EQUATION)Maintaining that g zs (x) + g ft (x) = 1, the optimal weight g * ft (x) to minimize Eq. ( 14) becomes: Recall that we are interested in using the distance to ZSF set, i.e., d(x), to surrogate g * ft (x). To understand the relationship between d(x) and g * ft (x), we first group test samples in ImageNet and its five distribution shifted datasets into bins based on the value of d(x). We then compute the averaged g * ft (x) for each bin and plot the relationship in Figure 9 . In specific, we use temperature scaling [14] to calibrate the zeroshot and fine-tuned models over the validation set. Afterwards, we define the true distribution P(y|x) as a one-hot vector, where the value of 1 corresponds to the true label for a given input x. We then calculate η ft (x) = P(y|x; θ ft ) -P(y|x) and η zs (x) = P(y|x; θ zs ) -P(y|x). Finally, we compute the average g * ft (x) for each bin as shown in Figure 9 . Interestingly, we observe the similar trend in Figure 1 (b) : the weight for fine-tuned models decreases as d(x) increases. This phenomena indicates that our weighting function ω(x) derived under the assumption of independent errors is also valid in the presence of correlated errors.EQUATIONFor CLIP ViT-32 based E2E-FT and LP-FT models, we use a batch size of 512. For CLIP ViT-16 based E2E-FT, we directly download the fine-tuned models from Wortsman et al. [28] 1 . The batch size for training CLIP ViT-16 based LP-FT models is set to 384, which is the largest batch size that fits into 2 A6000 GPUs. When performing linear probing, we use a batch of 512 and the initial learning rate of 0.1 for all experiments. The mixing coefficient α for OSE and WSE are searched over [0, 0.1, 0.2, ..., 0.9, 1.0]. The values of a and b for our VRF are reported in Table 7 . In Figure 10 , we present an ID-OOD scatter plot over a wide range of a and b values on ImageNet and its five variants using CLIP ViT-B/16. Specifically, a varies from 1.2 to 1.8, while b ranges from 0.5 to 1.0. Our method consistently achieves better ID-OOD trade-offs, as indicated by its points lying outside the OSE curve (represented by the magenta curve) across different configurations.Results for all mixing coefficient α for OSE and WSE are available in Table 8 and Table 9 , respectively. Results for values of a and b are available in Table 10 . In addition, we plot the ID-OOD trade-off curves for OSE and WSE on the CIFAR-10 and Entity-30 datasets in Figures 11 and 12 , respectively.We have conducted additional experiments where we optimized the hyperparameters for each test set of the ImageNet benckmarks. The results are summarized in Table 11 .Our VRF framework is orthogonal and complementary to existing fine-tuned models. To demonstrate this, we integrated FLYP [5] into our VRF framework. The results in Table 12 show that VRF improves OSE's performance under distribution shift by 1.1% without compromising in-distribution (ID) performance.We provide the breakdown performance for selective prediction using OOD detectors in Table 13 . compute the ratio. Note that the averaged ratio Acc ft Acczs on ImageNet-{A,R} and ObjectNet is undefined for d(x) = 0.6. This is because in these datasets, the zero-shot accuracy around d(x) = 0.6 is 0. We observe that the trend of the ratio Justification: We point out that the ID-OOD trade-offs still exist in existing fine-tuning methods, and propose VRF to simultaneously attain the best ID and OOD accuracy. Experiments on a variety of different models and tasks validate the effectiveness of our proposed method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://drive.google.com/drive/folders/1f56kjpRKPiNSaUxNDtETEDRkbDkZnpCQ 


Decision: This paper presents an output-space ensemble approach to robust finetuning of zero-shot model (i.e., weighted sum of outputs from zero-shot and finetuned models). The reviewers appreciate the novel, simple, and effective method and its strong performance, and the clarity of the manuscript. They however at the same time raised multiple nontrivial concerns. Most reviewers were concerned about the high space-time complexity due to the double forward propagations and the k-th nearest neighbor distance computation (e2mj, 8xce, 52xu, ydTd). Other concerns include additional hyperparameters that may have to be carefully tuned per dataset (vpPs, 52xu), lack of comparisons with latest techniques (vpPs, ydTd), lack of empirical evidences supporting the versatility of the method (52xu), experiments on limited datasets (vpPs), design choices that are not well justified (e.g., the use of k-th nearest neighbor distance) (e2mj, 8xce), and some presentation issues (e2mj). The authors' rebuttal and subsequent responses in the discussion period address most of these concerns; a borderline reviewer was not fully satisfied by the response and still concerned in particular about the lack of sensitivity analysis, but mentioned that it is not a ground for rejection. In the end, all the reviewers were supportive of the paper. 

The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers and to add new results and discussions brought up in the rebuttal and discussions, in particular a more thorough analysis on the sensitivity to hyperparameters and more experiments with a linear probed model; the latter is essential to fully resolve the concerns with the space-time complexity.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue of ID-OOD (In-Distribution vs. Out-of-Distribution) in the context of robust fine-tuning techniques commonly used in ensemble methods. The authors propose a sample-wise mixing approach to resolve this problem. The method involves creating a Zero-Shot Failure set, which contains samples that fail in zero-shot models but succeed in fine-tuned models, and mixing the predictions of the zero-shot and fine-tuned models based on the distance between test samples and the Zero-Shot Failure set. This approach demonstrates superior performance over traditional weight-ensemble and prediction ensemble methods on the ImageNet variants benchmark.
strength_weakness: strength_weakness: - The main drawback of the proposed method is the reliance on multiple hyperparameters (a, b, p), making tuning more complex compared to traditional ensemble methods that typically involve only one hyperparameter. Finding optimal values for these hyperparameters is challenging as they can vary significantly across different datasets, potentially affecting performance.
- To demonstrate the robustness of the proposed method, evaluations should be extended to more datasets beyond the ImageNet variants, similar to the experiments in the WiSE-FT paper, which include datasets like iWILDCam and FMoW. Differences in hyperparameter sweep ranges across datasets could highlight potential issues with the method.
- There is a lack of performance comparison with other robust fine-tuning methods, such as FLYP[1] and MaskFill[2], which limits the assessment of the proposed method’s effectiveness. In the ImageNet classificaiton experiments, the proposed method is inferior to [1,2] in performance.
- The paper need to report the hyperparameters found in experiments with different datasets and architectures to analyze their variability and sensitivity, providing deeper insights into the robustness of the proposed method.

[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et. al., CVPR 23
[2] Masked images are counterfactual samples for robust fine-tuning, Xiao et. al., CVPR 23
questions: questions: I am curious about the performance when finding hyperparameters for each test set as an oracle result.
limitations: limitations: The authors have adequately addressed some limitations of their work. However, they have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no negative societal impacts.
paper_summary: paper_summary: This paper aims to tackle the ID-OOD trade-off in the fine-tuning of pre-trained models with zero-shot abilities like CLIP. The proposed Variance Reduction Fine-tuning (VRF) is a sample-wise ensembling method concerning the zero-shot and fine-tuned models. The ensemble weights are determined by the distance from the test sample to the training samples that are incorrectly predicted by the zero-shot model. Theoretical analysis and experimental results justify the effectiveness of VRF.
strength_weakness: strength_weakness: 1. A major downside of VRF compared to previous ensemble methods is that the determination of the ensemble weights requires the storage of ZSF features at test time and computation of the distance from the test samples to each sample in ZSF set. This may bring a significant budget when the ZSF set is large. There should be a discussion on the theoretical and practical computational complexity regarding space and time.
2. The proposed distance calculation using the k-th nearest neighbor is borrowed from the previous work on OOD detection [24], and there is a lack of explanation for the specific choice of this metric. It seems that other metrics like the averaged k nearest neighbor distance discussed in [24] may also work in VRF.
3. It is claimed that the proposed method “can simultaneously attain the best ID and OOD accuracy without the trade-offs” (Line 9). However, this is not sufficiently justified. Specifically, it is unclear whether the “best ID and OOD accuracy” refers to the comparison with other existing methods or only considers the performance of the proposed method under different hyperparameter settings. Judging from Figure 3 (a.1), VRF does not always achieve the “best ID and OOD accuracy” compared with the peak ID/OOD accuracy of the existing ensembling method.
questions: questions: 1. To what extent is VRF less efficient than existing ensembling methods?
2. Why is k-th nearest neighbor distance selected for the distance calculation step of VRF?
3. What is the meaning of solving the ID-OOD trade-offs?
limitations: limitations: The authors have addressed some limitations and potential impacts of the work. I suggest additional discussions on the efficiency of the proposed method in the limitation part.
paper_summary: paper_summary: This paper studies the trade-off between in-distribution (ID) and out-of-distribution (OOD) performance of pre-trained models before and after fine-tuning. The authors observed that the sample distance is inversely proportional to $\frac{Acc_{ft}}{Acc_{zs}}$. After modeling the residual error of the model, they found that for a training sample $x$, the optimal ensemble weight is proportional to the sample's accuracy. Therefore, the authors set different ensemble weights for different samples based on the distance between the samples and the erroneous samples in the training set. Experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. VRF requires identifying and saving the zs classification error samples for subsequent use, which presents certain limitations.
2. The description of Step 2 is unclear: is it calculating the distance to the k-th nearest sample in $V$, or is it clustering the representations in $V$ first and then calculating the distance to the $k$-th cluster center?
3. Is the representation $ v $ of $ x $ calculated by ft or zs?
4. For each sample, recalculating and sorting the distance to $V$ incurs additional inference costs. How much extra computational cost does this introduce?
5. Since line 278 indicates that different values of $k$ have a minor impact on the model, why not use the nearest sample to calculate the distance? What is the reason behind choosing the $k$-th sample for distance calculation?
questions: questions: See Weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This study examines the robust fine-tuning of CLIP models. The proposed method, Variance Reduction Fine-tuning (VRF), employs a sample-wise ensembling technique to enhance both ID and OOD accuracy, reducing trade-offs between them. Experimental results on ImageNet and associated distribution shifts empirically validate the effectiveness of this approach.
strength_weakness: strength_weakness: - Although there has been considerable research on robust fine-tuning for CLIP (as noted by the authors referencing various studies), the baselines analyzed in Tables 1 and 2 are quite limited. A key advantage of the VRF method is its applicability to any (zero-shot, fine-tuned) pair, both retrospectively and prospectively. Therefore, applying VRF to other robust fine-tuning methods beyond FT and LP-FT would further demonstrate the effectiveness of the proposed VRF approach.

- The proposed method has a drawback of needing more than double the inference cost compared to WSE and other robust fine-tuning methods that yield a single fine-tuned model, since it requires performing a forward pass through the entire network twice (once for zero-shot and once for fine-tuned models) and calculating the ZSF-based weighting coefficient. However, an analysis of such inference costs has not yet been performed. It is important to note that Wortsman et al. (2022) favored WSE over OSE for combining zero-shot and fine-tuned models, highlighting this choice not only for its improved performance but also for its lack of additional costs.

- In the context of robust fine-tuning, we can only observe performance on in-distribution data. Therefore, the parameters 
$a$ and $b$ in the proposed Eq. (6) for sample-wise mixing coefficients should be determined based on the in-distribution validation set, as mentioned by the authors in lines 141 and 198-200. From Appendix C.1, it is understood that $\\alpha \\in \\{1.4, 1.5, 1.6\\}$ and $b \\in \\{0.5, 0.6, ..., 1.0\\}$ were considered. However, it seems that the range of values for $a$ and $b$ is too narrow. Based on the in-distribution results in Table 8, larger values for $a$ should be explored, but the authors did not do so. What happens if the value of $a$ is increased further? Would it still outperform OSE and WSE baselines?
questions: questions: 1. In combining zero-shot and fine-tuned models, WSE offers the advantage of requiring only half the inference cost compared to VRF by maintaining inference costs equivalent to that of a single model through weight averaging. When combining zero-shot and linear-probed models, since the image encoder part, which carries most of the inference cost, only needs to be processed once, it appears that VRF's concerns regarding cost could be mitigated. In this context, could the proposed VRF method also extend to "Linear classifier" models, alongside "E2E-FT" and "LP-FT"? It is worth noting that Wortsman et al. (2021) explores ensembling not only between zero-shot models and end-to-end fine-tuned models ("E2E-FT" in this work) but also between zero-shot models and linear classifier fine-tuned models (referred to as "Linear classifier" here).

2. In Figures 3, does "Ensembling with varying coefficient $\alpha$" refer to OSE? It would be helpful to also provide the curve for WSE.

3. Table 7 shows the results of exploring $\alpha$ values for OSE. It would be beneficial to provide the same results for WSE as well.

4. The proposed VRF approach also produces multiple points on trade-off plots (e.g., Figure 3(a.1)) depending on the values of $a$ and $b$ (especially $a$). It would be advantageous to visualize a scatter plot across a broad range of $a$ and $b$ values, allowing us to observe the trade-off curve similar to OSE and WSE, which interpolate between zero-shot and fine-tuned models.
limitations: limitations: The authors addressed the limitations and potential negative societal impact in Section 6.
paper_summary: paper_summary: This paper addresses the ID-OOD trade-off of the ensemble-based method under the pretrain-finetune regime. The authors propose a simple yet effective adaptive ensemble method that determines the ensemble coefficient based on the feature distance between a test sample and the zero-shot failure set. The proposed method has a theoretical foundation, and it consistently improves the OOD generalization performance compared to the vanilla ensemble baseline in diverse situations.
strength_weakness: strength_weakness: * The method requires 1) accessibility to the entire training dataset and 2) distance computation between each test sample and the entire train sample during inference time. These limit its application to resource-constrained (in terms of accessibility, memory storage, and runtime) situations.
* Limited implication of the theoretical result
  * While the current theoretical analysis provides a justification for the optimal strategy for determining the ensemble coefficients, which is realized by the authors, it does not say anything about the relative generalization error between the proposed method and other non-ensemble-based fine-tuning methods. 
* Lack of possible comparison with more advanced fine-tuning [4,5,6,7] or ensemble [1,2,3] methods.
  * While the proposed approach has its unique implication compared to the vanilla ensemble, missing comparison with advanced baselines raises concern about the practical utility of the proposal.

---
> Reference
* [1] SIMPLE: Specialized Model-Sample Matching for Domain Generalization, Li et al. 2023
* [2] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models, Zhu et al. 2023
* [3] Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization, Mavromatis et al. 2024
* [4] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, Kumar et al. 2022
* [5] Finetune like you pretrain: Improved finetuning of zero-shot vision models, Goyal et al. 2023
* [6] Trainable Projected Gradient Method for Robust Fine-tuning, Tian et al. 2023
* [7] Towards Calibrated Robust Fine-Tuning of Vision-Language Models, Oh et al. 2024
questions: questions: * Could the proposed method also be applied to the weight-space ensemble rather than the output-space ensemble?
* How sensitive is the proposed method against varying numbers of training samples that determine the quality of the zero-shot failure set?
limitations: limitations: The authors adequately stated the limitation in the paper.

Comments: title: Response to Reviewer ydTd
Response to A5
comment: Dear Reviewer e2mj,

We sincerely appreciate your invaluable comments, which have significantly enhanced the quality of our manuscript. We hope our responses have adequately addressed your concerns. If you find our replies satisfactory, we kindly ask if you would consider revisiting your rating of our paper.

Sincerely,
The Authors
We are delighted that our rebuttal addresses your concerns and greatly appreciate your valuable suggestions. We also thank you for acknowledging the effectiveness of our method. We will include the graphs for all datasets, along with an exploration of the hyper-parameters on validation sets, in the revised manuscript.

Sincerely,

The authors of Submission 3218
Thank you for your detailed response, I have carefully reviewed all the reviews and the rebuttal content, and all of my concerns have been resolved.  In particular, after revisiting Figure 6, where the certain hyperparameter $b$ achieves a high peak in ID performance,has reassured me that the hyperparameters are searchable and generalizable to OOD datasets.

To further strengthen your final paper, I would suggest including similar graphs for other datasets to confirm that these results are not unique to ImageNet. It would be good to show these in the appendix. Additionally, it would be valuable to provide graphs that illustrate the hyperparameters explored on the validation set, not just the test set's in-distribution performance.

The proposed method effectively show the capability of sample-wise out-of-space ensemble, and thus, I would like to raise my score.
Dear Reviewer vpPs,

We appreciate reviewer vpPs constructive feedback, which further helped us improve our draft.

We have submitted our responses to concerns, and we want to know if these replies address your concerns. Any further comments or questions are welcome to us.

Sincerely, 

The authors of Submission 3218
Dear Reviewer e2mj,

We appreciate reviewer e2mj's valuable comments that significantly contribute to improving our manuscript.

We want to know if the response address your concerns. Any further comments are welcome to us.

Sincerely, The authors of Submission 3218
I have updated my rating to 5 based on the authors' assurance that the issues will be addressed. Wishing you the best of luck.
Dear Reviewer 52xu,

Thank you for your thoughtful comments. We agree with your concerns about the impact of inappropriate choices for $a$ on performance. As you mentioned, setting $a$ too high biases the final predictions toward the fine-tuned models, while setting it too low favors the zero-shot models. In the revised version, we will address this issue directly and emphasize the need for careful selection of $a$. 

Regarding the use of the linear probed model, we have provided detailed results in **A3**. With a fixed backbone, OSE and WSE achieved 79.9% ID accuracy and 57.8% OOD accuracy. Our VRF further improved these results, reaching 80.0% ID accuracy and 61.0% OOD accuracy.

We appreciate your willingness to raise the score to 5. We will seriously take the reviewer’s advice to revise the paper and  promise that the revised paper will thoroughly address all concerns.

Sincerely,
Authors of Submission 3218
The authors have addressed most of my concerns in their rebuttal, including: 1) combining VRF with other robust fine-tuning methods, 2) using a linear probed model instead of a fine-tuned one, and 3) presenting ID-OOD trade-off plots.

Regarding the third point, I recommend that VRF be shown as a curve connecting the zero-shot and fine-tuned points on the trade-off plots, rather than as a single point. Since VRF, like OSE and WSE, is a post-hoc method that combines zero-shot and fine-tuned models, the authors should simply demonstrate that it appears in the top right position on the ID-OOD trade-off plots compared to OSE and WSE. Personally, I sense that the authors may be reluctant to show VRF underperforming the baseline when an inappropriate value of $a$ is chosen, but this is a natural outcome and does not need to be avoided (as the two extremes of $a$ would correspond to either the zero-shot or fine-tuned model).

While I have not received a response to the second weakness I raised, I reviewed the authors' comments to other reviewers. It seems that other reviewers have also expressed concerns about the additional computational burdens introduced by VRF. The current explanation is somewhat insufficient (which I understand might be due to the limited time for the rebuttal), and I believe the authors should provide a thorough analysis of this issue in the revised version. Therefore, if the revised version includes more detailed information on the ID-OOD trade-off plots and computational cost analysis (in addition to addressing the other issues mentioned in the rebuttal), I am willing to raise my score to 5.
We appreciate the reviewer's advice. Including the WSE frontier curve in Figure 12 is consistent with Figures 10 and 11. As the attached PDF is not editable, we are pleased to add the curve in the revised version.

Sincerely,
The authors of Submission 3218
Thank you to the authors for addressing most of my concerns. Would it be possible to revise Figure 12 to include WSE, with Zero-Shot and Fine-Tuned as the endpoints, consistent with the other figures?
We do appreciate the reviewer's positive support. We will seriously take the reviewer's advice to carefully improve our paper. Thank you very much again.

Sincerely, The authors of Submission 3218
Thanks for your responese. I don't have any other questions and I have changed my score from 4 to 6.
Thank you for the positive feedback and valuable comments. We appreciate your interest in comparing our method with other ensemble methods and we will consider it for future work.  We are delighted that you find our method novel and effective, and we look forward to the possibility of sharing this work as an official publication soon.
You are correct in noting that each sample has an equal risk of being mislabeled. However, the density around the k-th sample is typically higher compared to the nearest sample. This higher density leads to lower variance when measuring distance using the k-th sample. Specifically, because the k-th sample is positioned between the (k-1)-th and (k+1)-th samples, in a high-density region, the range between these distances tends to be smaller. This reduced variance provides a more stable measure, making it less sensitive to outliers or mislabeled instances.

Additionally, research[1] has demonstrated that using the k-th nearest distance is effective in density estimation in OOD detection, and we have adopted this approach.

We hope this explanation addresses your concern.

[1] Out-of-Distribution Detection with Deep Nearest Neighbors
Thanks for the kind answers!

Overall concerns are addressed by the author, while I still wonder about the relative superiority compared with another input-dependent advanced ensemble method (but whether this is addressed or not, I will keep my rating).
Anyway, the proposed method is quite novel and effective and has a good theoretical background.
I am looking forward to seeing this paper as an official publication soon!

reviewer ydTd
I don't understand A5. The nearest sample is the case where $k=1$, Why does calculating $k$ using $float(p \cdot |V|)$ help mitigate the mislabeled noise? I think every sample has the same risk of being mislabeled, right?