Towards Calibrated Robust Fine-Tuning of Vision-Language Models

Authors: changdae  oh
hyesu  lim
mijoo  kim
dongyoon  han
sangdoo  yun
jaegul  choo
alexander  hauptmann
zhi-qi  cheng
kyungwoo  song
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:39:05

Abstract: Improving out-of-distribution (OOD) generalization during in-distribution (ID) adaptation is a primary goal of robust fine-tuning of zero-shot models beyond naive fine-tuning. However, despite decent OOD generalization performance from recent robust fine-tuning methods, confidence calibration for reliable model output has not been fully addressed. This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. Firstly, we show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data: 1) ID calibration error and 2) the smallest singular value of the ID input covariance matrix. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value, which is further guided by the self-distillation of a moving-averaged model to achieve calibrated prediction as well. Starting from empirical evidence supporting our theoretical statements, we provide extensive experimental results on ImageNet distribution shift benchmarks that demonstrate the effectiveness of our theorem and its practical implementation.

Main paper: 
 introduction: 
 Foundation models [6] such as CLIP [47] have been extensively utilized on diverse domains via pretrain-finetune approaches. Their generalized knowledge shaped after large-scale pre-training enables them to easily adapt to downstream tasks through zero-shot inference or fine-tuning. However, it has been steadily reported that a naive fine-tuning approach comprises foundation models' strong out-of-distribution (OOD) generalization capability during adaptation to in-distribution (ID) data [61, 30] . To ensure robustness under distribution shifts, a wide range of research has followed [61, 30, 17, 32, 57, 43, 42] so-called robust fine-tuning. Despite the advancements of the robust finetuning methods, we are aware that an important criterion for trustworthy machine learning has been overlooked -confidence calibration [39, 8] that quantifies how close the confidence of our predictor is to the actual correctness of predictions. As shown in Figure 1 , existing robust fine-tuning methods hurt the confidence calibration in terms of expected calibration error (ECE) [40] on OOD data compared to the zero-shot evaluation while they show improvements on OOD accuracy. In this Figure 1 : OOD accuracy vs. ID accuracy (left) and negative OOD ECE (right). To maintain consistency in the plots, where desired values are shown on the right side of the x-axis, we report negative OOD ECE. ID ACC refers to ImageNet-1K top-1 accuracy; OOD ACC and ECE refer to the averaged accuracy and ECE of the five ImageNet distribution shifts (ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch, and ObjectNet), respectively. Detailed numbers are reported in Table 2 and 3 . Note that the competing methods -FLYP [17] , LP-FT [30] , and Lipsum-FT [42] -improve OOD accuracy over the zero-shot baseline (ZS) and naive fine-tuning (FT) but suffer from OOD miscalibration, presumably due to concerning generalization solely during fine-tuning. Our CaRot outperforms existing methods on both OOD accuracy and calibration by large margins.work, we introduce a calibrated robust fine-tuning method (CaRot) that simultaneously improves confidence calibration and accuracy of the classifier on OOD data.Confidence calibration is a key aspect of reliable machine learning, essential for avoiding highconfidence incorrect predictions in real-world decision-making systems. This is particularly crucial in high-stakes tasks like autonomous driving and healthcare applications. After a seminal work [18] revealed the miscalibration problem of high-performing neural networks, a plethora of attempts followed to improve the calibration of neural network models through post-hoc adjustments [65, 18, 29, 68, 19] or train-time regularizations [67, 52, 56, 38, 37] . However, many of them focus on improving calibration for ID samples, and methods for enhancing OOD calibration usually require OOD samples at train time [63, 16] . Moreover, these approaches commonly focus on calibration alone without ensuring improvement in other quantities, e.g., accuracy. In this work, we explore a unified framework that jointly considers calibration and accuracy (particularly on OOD data).To accomplish low classification and calibration errors on OOD samples with only ID samples in our hands, we conduct theoretical analyses of those OOD errors. To be specific, we derive an upper bound that is shared for OOD classification error and OOD calibration error composed with two quantities on ID samples, 1) the reciprocal of the smallest singular value of the normalized covariance matrix of ID data representation and 2) the ID calibration error. Different from the existing bounds focusing on either one of classification or calibration error [5, 72, 63] , we address both classification and calibration errors in a single unified bound. More importantly, the chief components of our bound can be computed solely with ID samples without relying on any OOD samples, which discerns our approach to existing work [4] .Motivated by our theoretical analysis, we propose a new multimodal contrastive loss that promotes the smallest singular value of input image representation to become larger by enforcing the orthogonality of the final projection matrix of the visual encoder. Furthermore, to understand the working mechanism in depth, we present an interpretation of our new multimodal contrastive loss as a process of seeking the low-rank approximation of cross-covariance matrix over image-text representations on a reduced solution space induced by the orthogonality constraint. Meanwhile, to enhance confidence calibration on ID samples during fine-tuning, we utilize a self-distillation (SD) with an exponential moving average (EMA) teacher model. This EMA SD encourages a student model to learn semantic similarity structures of in-batch data representations from teacher predictions across diverse contrastive pairs, appropriately adjusting confidence per instance.We first validate our new error bounds with synthetic datasets to show that the bounds hold empirically. Then, evaluate our method by conducting extensive experiments of fine-tuning CLIP [47] on ImageNet-1K [10] classification task under natural distribution shift (ImageNet-V2/R/A/Sketch and ObjectNet) and synthetic distribution shift (ImageNet-C). We demonstrate the effectiveness of our proposed framework for robust generalization and calibration by observing consistent improvements in terms of expected calibration error and accuracy on ID/OOD datasets. Summary of contributions. 1) We point out that existing fine-tuning methods do not adequately achieve satisfactory OOD generalization and calibration simultaneously. 2) We provide theoretical analysis for classification and calibration errors on the OOD data and show that they are both bounded from above by the ID calibration error and the smallest singular value of the covariance matrix over the ID input representation. 3) Based on our theoretical analyses, we devise a calibrated robust fine-tuning method, CaRot, as a practical realization of our theorem that reduces the upper bound of OOD classification and calibration errors by conducting constrained multimodal contrastive learning with EMA self-distillation. 4) We present empirical evidence for our theory on a synthetic dataset and demonstrate the efficacy of CaRot via extensive evaluations on ImageNet-1K distribution shifts in terms of accuracy and calibration error on ID and OOD domains. 
preliminary: 
 Robust fine-tuning aims to achieve consistently high performance on data from both training distribution (ID) and related but different test distributions (OOD). For validation, we commonly consider a covariate shift scenario for the classification task, where both ID and OOD domains share the class categories (Y ID = Y OOD ) and have the same conditional distribution P (Y |X), but have different marginal distributions over input X. That is, P ID (Y |X) = P OOD (Y |X) but P ID (X) ̸ = P OOD (X). Here, we evaluate a model that is fine-tuned on a training split of the ID domain, on a test split of ID, and on OOD domains. The term "OOD" is quite general, and we confine the scope of OOD to transformed and related distributions with ID [12] . For example, if our ID data is about an object recognition task with images, OOD data is about a sensor-noised version of ID, or independently collected data from a different domain targeting the same task.Confidence calibration is a concept of matching the prediction probabilities yielded for different inputs to the expected accuracy on these inputs. In a K-way classification setting, let X ∈ R d and Y ∈ {1, ..., K} be random variables indicating inputs and labels, respectively. A dataset with N independent samples from the joint distribution P (X, Y) = P (Y |X)P (X) is denoted by {(x n , y n )} N n=1 .Let f be a classifier and f (y|x) = p be a confidence, i.e., the maximum of probabilities among K dimensions corresponding to its prediction ŷ. We say a model is perfectlycalibrated when P(ŷ = y|p = p) = p, ∀p ∈ [0, 1]. As a quantitative measure, the model calibration can be derived as E[|P(ŷ = y|p = p) -p|]. In practice, we use expected calibration error (ECE) [40] as an empirical approximation of the model calibration, which is a weighted average of bin-wise miscalibration. The ECE divides the confidence score of N samples into M uniform confidence bins {B m } M m=1 and takes the mean of the gap between accuracy (acc) and confidence (conf) over the bins weighted by the number of samples in the bins, i.e., ECE= M m=1 |Bm| N |acc(B m ) -conf(B m )|. 
theoretical analysis on ood generalization and calibration: 
 We first identify the factors that affect OOD generalization and calibration errors under circumstances where only ID data is accessible. We take inspiration from the generalization bound of domain adaptation literature [5, 72] while remarkably adapting the analysis to consider both OOD classification error and OOD calibration error at the same time in a more practical way.Let D be a domain on input space X and Y = {0, 1} be a label space for a binary classification. Among the sufficiently expressive hypothesis functions h : X → [0, 1] in a class H, we define h 0 (•) as a desired calibrated predictor for y, which minimizes the calibration errorE x∼D [(h(x) -c(x)) 2 ] [39], where c(x) = E y [y|h(x)] is the expected value of y given a prediction h(x). That is, h 0 always produces the calibrated prediction for y given x so that the output confidence h 0 (x) matches the expectation of y over the subset of samples that have the same confidence value with h 0 (x). Our goal is to learn a hypothesis function h(•) that outputs reliable prediction probability on samples from the unseen OOD domain, which is defined by a distribution D OOD , as well as on the ID domain D ID , where the predictor is trained on. In essence, the error εD (h) = E x∼D [(h(x) -h 0 (x)) 2 ]should be small for two different domains D ∈ {D ID , D OOD }. Here, we focus on the covariate shift scenario ( §2) that only the marginal distribution over X changes while the distribution over Y is preserved.Let the optimal hypothesis h * , which minimizes a combination of errors on both ID and OOD, be h * := arg min h∈H ε DID (h) + ε DOOD (h), and ∆ denote the optimal joint error ∆ := ε DID (h * ) + ε DOOD (h * ). Now, we derive a new bound for the OOD calibration error and OOD classification error.Theorem 3.1. Let h : X → [0, 1] be a real-valued function which has structure h(x) = d i=1 h i (x[i]) where h i is an arbitrary one-dimensional function, and h(•) is in a hypothesis class H that has pseudo dimension Pdim(H) = d h , DID be an N -size empirical distribution on ID domain.If (x[i], x[j]) is a bi-variate Gaussian random variable for every i, j ∈ [d], then for any δ ∈ (0, 1) and for all h, the following bounds hold with probability at least 1 -δ:EQUATIONwhere ΣDID :=E DID [xx T ] is a covariance matrix with a strictly positive minimum singular value ofd-dimensional normalized input x = (x[1], ..., x[d]), where x[i]:=(x[i] -E[x[i]])Var(x[i]) -1/2and σ min (M ) is the smallest singular value of a matrix M ∈ R d1×d2 . These theoretical results can be directly applied to the intermediate or penultimate layer's representation of a neural network by setting the input variable x as a representation vector as in [4, 73] . From now on, we will assume the input as an image representation from the last layer of the visual encoder in the following sections. Note that 1) the LHS of the first inequality (ineq.( 1)) indicates OOD calibration error, 2) two terms in the LHS of the second inequality (ineq.( 2)) denote OOD classification error in terms of L 2 loss and prediction sharpness on OOD domain, and 3) both inequalities have the same RHS, which contains the empirical estimate of ID calibration error, the reciprocal of the smallest singular value of ID input covariance matrix, and remaining irreducible terms that depend on the problem setup. Intuitively, Theorem 3 implies that pursuing diverse input features while maintaining the calibration of the classifier contributes to improving OOD calibration and generalization simultaneously. We defer the proof and discussion on the tightness of the bound and its assumptions in Appendix C.Based on our analysis, we expect the potential of reducing the upper bound of OOD calibration error and the sum of OOD classification error and prediction sharpness by minimizing the first two terms of RHS in both bounds: the empirical ID calibration error and the reciprocal of minimum singular value of the normalized ID covariance matrix. In §4, we devise a realization of this theoretical concept. 
method: 
 Our goal is to achieve good OOD generalization and calibration during the ID adaptation of pretrained models. Motivated by Theorem 3, we propose a new fine-tuning method that increases the smallest singular value of the ID input covariance matrix while improving ID calibration, thereby lowering the upper bound of OOD calibration and generalization errors. By following [61, 30, 17] , we set a vision-language model (VLM), CLIP [47] as our target, which serves as a remarkably strong backbone for zero-shot inference and fine-tuning with ease. We limit the scope of validation to image classification tasks. Note that our theorem is not confined to specific domains or model architectures and thus can be applied beyond CLIP's image classification. See Figure 2 for an overview.To adapt a pre-trained VLM on image classification tasks, the cross-entropy loss is the most common choice as an objective function. However, there are emerging shreds of evidence supporting the use of contrastive loss (CL) for robust adaptation [70, 50, 17] , especially when the model is pre-trained via CL. Witnessing its empirical success on OOD generalization [17] , we leverage a CL-based learning strategy for VLM fine-tuning. CLIP consists of an image encoder f θv (•) = f θv (•)W v and a text encoder g θ l (•) = g θl (•)W l , where encoders are composed with backbone models (f θv (•), g θl (•)) and projection matrices (W v ∈ R dv×r , W l ∈ R d l ×r ). The encoders produce L 2 -normalized representations to compute the similarity between image and text inputs. Given N pairs of (image, text) {(I i , T i )} N i=1 , a common form of multimodal contrastive loss (MCL) can be written asEQUATIONFigure 2 : Overview of CaRot. We fine-tune a VLM using a multimodal contrastive loss with an orthogonality constraint on visual projection layer (eq.( 4)) and self-distillation LSD (eq.( 5)) that takes predictions of EMA teacher ψ as soft target labels to train the student model θ. where θ = {θ v , θ l } are the parameters of image and text encoders and R(θ v , θ l ) reflects a general regularization strategy in CL [7, 21] . We update both image and text encoders during fine-tuning as done in the pre-train phase and use OpenAI templates [47] to create (image, text) pairs from a downstream classification dataset that consists of (image, class) pairs.Meanwhile, the basic form of L MCL does not inform anything about the singular value distribution of learned representation. In §3, we showed that the reciprocal of the smallest singular value constitutes the shared upper bound, i.e., the larger the smallest singular value is, the lower the upper bound becomes. To encourage this, we put a soft constraint term to L MCL that enforces the final projection matrix W v of the visual encoder to be orthogonal and hence the output image representation matrix to have a large effective rank, as in below:EQUATIONwhere I is an identity matrix that has the same shape with W T v W v and λ OC is a strength of the orthogonality constraint 1 . While recklessly increasing the singular values might hinder ID adaptation, our orthogonal constraint mitigates the degradation of performance by pursuing not only the smallest singular values to be large but also the largest singular values to be small which is important for generalization on ID data [69] . Interestingly, this contrastive loss with regularization terms can be viewed as a constrained singular value decomposition (SVD) with a cross-covariance matrix of image-text representations where the orthogonality constraint is applied.To be specific, by following Nakada et al. [41] , under a linear representation assumption, a gradient descent step of L MCL-con boils down to the maximization of the SVD objective, which aims to find a low-rank approximation of the normalized cross-covariance matrix S(β) 2 as follow:arg min Wv ,W l LMCL-con(W ) := 1 2N N i=1 -log exp(Wv Îi • W l Ti) N j=1 exp(Wv Îi • W l Tj) + 1 2N N i=1 -log exp(Wv Îi • W l Ti) N j=1 exp(Wv Îj • W l Ti) + R(Wv, W l ) + λOC||W T v Wv -I|| 2 F ≈ arg max Wv ,W l SVD(S(β)) := tr(W T v S(β)W l ) -(ρ/2)||WvW T l || 2 F subject to ||W T v Wv -I|| 2 F = 0,where we adoptR(W v , W l ) = (ρ/2)||W v W T l || 2F for ρ > 0 as a regularization term to promote the encoders to capture diverse features as in Ji et al. [27] . Here, we assume that the input of L MCL-con is the penultimate representation of VLM's encoders, i.e., ( Îi , Ti ) = (f θv (I i ), g θl (T i )), and 1 While it is also possible to inject constraint on the text projection matrix W l , we only do it for Wv because our concern is about the singular value of image representations for downstream tasks. We observed degradations of accuracy (ID:-0.01, OOD:-0.15) and ECE (ID:-0.002, OOD:-0.02) by adding the constraint on W l .2 S(β) := 1 N N i=1 βi Îi T T i -1 N N i̸ =j βij Îi T T j, where βi and βij depend the choice of non-linear function over dot product between image and text representations. See Nakada et al. [41] for a detailed derivation W = {W v , W l } is a set of projection matrices (i.e., last layers of each encoder). This connection between L MCL and SVD allows us to understand the working mechanism of our proposed objective. That is, minimizing L MCL-con can be interpreted as finding a good rank-r (dimensionality of the imagetext projection space) approximation of the cross-modal covariance matrix by seeking the direction of large co-variation among image-text representations, while the solution space is constrained by enforcing an orthogonality condition on the collection of vision-side singular vectors W v to achieve the larger effective rank of both the projection matrix W v and the image representation matrix (See Appendix §B for further explanation on the effective rank and the smallest singular value). In §5, we validate that L MCL-con significantly increases the smallest singular values and results in better OOD generalization and calibration on downstream tasks.In the previous section, we devise a new multimodal contrastive loss that promotes large σ min ( ΣDID ). We now address the next component standing for ID calibration, which is another crucial component according to our theoretical analysis. While there are numerous approaches to enhance calibration during neural network training [38, 56, 52 , 1], we notice the promising results of knowledge distillation-based calibration approaches [64, 71] . These approaches encourage the model to learn from input-dependent smoothed labels that effectively mitigate the overconfidence issue, which is commonly associated with miscalibration. Therefore, we employ a self-distillation (SD) method for ID calibration in that distilling the similarity score map would help avoid overconfidence. Specifically, we first initialize both teacher and student networks with a pre-trained CLIP model (including both image and text encoders); update the student model using gradient descent for every iteration while slowly updating the teacher model that has ψ = {ψ v , ψ l } as parameters using EMA with the momentum of α at every t > 1 iteration, i.e., ψ ← αψ + (1 -α)θ. Rather than hosting another VLM or fixed pre-trained CLIP as a teacher model, we adopt a self-evolving EMA network as a teacher, observing its successful usage on the weight-space ensemble between homogeneous models [61, 48] , robust self-supervised learning methods [2, 45] , as well as regularization [71] . With the EMA teacher {f ψv (•), g ψ l (•)} and the learning student {f θv (•), g θ l (•)}, we construct a self-distillation loss term for N data pairs as:EQUATIONwhere KL denotes Kullback-Leibler divergence,q I i = softmax({f θv (I i ) • g θ l (T j )} N j=1 ) and q T i = softmax({f θv (I j ) • g θ l (T i )} N j=1) are student outputs, and qI j and qT j are teacher outputs which are similarly defined by replacing the student parameter θ with that of teacher's ψ. Presumably, label smoothing (LS) [55] behaves similarly to what we intended, but we argue that LS would be less effective than EMA SD in terms of mitigating overconfidence issues. See Appendix §B.We complete the learning objective as a summation of L MCL-con and L SD with a coefficient λ SD , i.e., L = L MCL + λ OC L OC + λ SD L SD . The novel combination of these two components contributes to ensuring a larger smallest singular value of image representation and ID calibration simultaneously, which induces calibrated robust fine-tuning (CaRot) on distribution shifts. Note that this objective function is just one of the possible realizations of our upper bound (Theorem 3) on OOD generalization and calibration errors. Further exploration can spawn a more practical algorithm in the future. 
experiments: 
 In § 5.1, we first show empirical evidence of the error bounds that we derived in §3. We then provide experimental setup and main benchmarking results ( §5.2) and present further empirical studies ( §5.3).Our theoretical analysis in §3 revealed the possibility of managing OOD classification and calibration errors simultaneously by leveraging a shared quantity over the ID domain (sum of calibration error term and the singular value term). Before conducting real-world evaluations, we verify the theoretical analysis with a toy experiment that simulates distribution shifts. To be specific, we generate a binary classification dataset with 1000-dimensional Gaussian random variables as features where the mean of features are partly shifted across different test environments (ID, OOD). We train a three-layer network with regularization terms: Born-Again-Network (BAN)-style self-distillation [13] for calibration (L SD ) and the orthogonal constraint (shown in eq.( 4)) for the singular value (L OC ). Detailed descriptions of the experimental setup are provided in Appendix §A.1. 1 , which also indicates that reducing the upper bound results in better OOD generalization and calibration. These analyses demonstrate that Theorem 3 empirically holds.Training and evaluation. We adopt CLIP ViT-B/16 as our VLM backbone and evaluate each finetuning method, including CaRot, in terms of calibration (with ECE) and accuracy under distribution shifts. For downstream tasks, we consider the ImageNet-1K (IN) classification and regard it as our ID domain. For all methods, we optimize the model parameters using the AdamW with a batch size of 512 over 10 epochs. Fine-tuned models are evaluated under varying distribution shifts.Benchmark datasets. We consider IN-V2 [49] , IN-R [23] , IN-A [24] , IN-S [60] , and ObjectNet [3] as natural shifts of the in-distribution dataset (IN). We refer to the average performance over these five datasets as Avg. Shifts or OOD throughout the following sections unless it is specified as a different dataset, e.g., IN-C [22] which we adopt as a synthetic shift scenario occurred by sensory noises. Baseline methods. We benchmark CaRot alongside zero-shot inference (ZS) and fine-tuning methods: standard fine-tuning (FT), LP-FT [30] , FLYP [17] , and Lipsum-FT [42] . Refer §A for further details.In Appendix Table B , C, and D, we compare results with post-hoc robustification method (weight ensemble; WiSE-FT [61] ) and post-hoc calibration (temperature scaling; TS [18] ) method.Results on natural shifts. Table 2 and  7 (See Table H and I for details). CaRot consistently outperforms the baseline methods for these backbones, too. Furthermore, in Table 6 , we provide additional comparisons with CAR-FT [34] , Model Stock [26] and ARF [20] 3 . Results on synthetic shifts. In real-world applications, distribution shifts are commonly occurred by sensory noises. To evaluate different fine-tuning methods under such synthetic shifts, we adopt a corrupted version of ImageNet (IN-C) with 15 types of image corruptions over five severities. In Figure 4 , we provide corruption-wise accuracy and ECE of each method averaged by five severities. Overall, CaRot consistently outperforms the baseline methods and the actual amount of improvements varying depends on the type of corruptions. Specifically, on the relatively coarser granular corruptions such as Snow, Frost, Fog, Brightness, and Contrast greatly change the semantics of the image (similar to natural shift), CaRot shows remarkably good performance compared to others. Meanwhile, on the finer granular corruptions such as Elastic transform and JPEG compression, the improvements achieved by CaRot become smaller. We present zoom-in results on these two cases in Figure 5 .Ablation study. In Table 4 , we provide results of the ablation study to show the impacts of each component of CaRot. In line with our hypothesis, results confirm that all three components boost OOD accuracy and calibration performance. The comparison of adopting and not adopting L MCL (we followed the naive fine-tuning approach for the latter) ascertains that employing contrastive loss as a fine-tuning objective is superior to cross-entropy loss for ID/OOD accuracy, consistent with the previous observations [17] , and even extends to improvements in calibration as well. The ablations of We speculate that learning diverse features by the singular value regularization while being enforced to contribute to reflecting the in-batch similarity structure by EMA SD induces well-restricted solution space [25] otherwise has risk converged to bad solutions (learning diverse features but noise-sensitive). Besides, Table 5 shows the impact of each component by varying the strength coefficients. We observe that the increased intensity of constraint improves ID and OOD performance to some degree, but there is a slight decline when the intensity is too strong. Meanwhile, the strength of self-distillation positively correlated with OOD accuracy and ECE, but there are negative effects on ID accuracy and ECE, which reflects the inevitable trade-off between ID adaptation and OOD generalization (Table G and F in Appendix provide further details). The smallest 20 singular values Analysis on singular values. Figure 6 illustrates the last 20 singular values of the covariance matrix ĪT Ī where Ī is a standardized image representations over N samples. Our proposed constrained contrastive loss L MCL-con increases the small singular values compared to the vanilla contrastive loss L MCL . This result verifies that adding the orthogonality constraint successfully reduces 1/σ min ( ΣDID ), the component of the shared upper bound we derived in §3, following our intention. Robust fine-tuning for visual foundation models. Beyond the ID generalization, there are a lot of works aiming at improving the generalization of fine-tuned models on the OOD domain. Some of them leverage the strong robustness of pre-trained model through weight-average [61, 26, 53, 44] or regularization [53, 57, 58, 42] whereas others attribute to the robustness during fine-tuning from different part of model backbone [30, 32] . Besides, Goyal et al. [17] claims that aligning the learning objective during pre-training and fine-tuning is crucial for retaining the remarkable OOD generalization capability of the pre-trained model. Although the above methods have provided insights into the extrapolation of foundation models regarding accuracy, confidence calibration has been unexplored, which is crucial for reliable ML applications. We investigate the OOD calibration of fine-tuned CLIP as well as accuracy and propose a unified fine-tuning strategy with theoretical support to achieve superior ID and OOD calibration alongside OOD generalization for the first time.Confidence calibration. After some early research on calibrated prediction [39, 8] , lots of follow-up studies have been conducted. As a seminal work, Guo et al. [18] revealed the miscalibration problem of neural networks, then, Minderer et al. [35] and LeVine et al. [33] provided a comprehensive analysis on the calibration of modern vision models with consideration on distribution shift. To improve the calibration of predictive models, Temperature Scaling (TS) [18] and Label Smoothing (LS) [55] are two representative methods in practice. TS-based approaches learn a temperature parameter itself [18, 16] or model [63, 28] to estimate the temperature to adjust the output probability of models, whereas LS-based methods focus on producing soft labels to mitigating overconfidence issues by a fixed [55, 38] , randomized [56] , or model-based [71, 66] smoothing strategies. However, existing approaches do not consider distribution shifts [71] , assume accessibility to target domain [16, 63] , assume specific type of distribution shift [59] , cannot adjust confidences individually [18, 55] . In this work, we adopt EMA self-distillation as an effective input-dependent calibration method and show that the superior calibration results on in-domain samples can be transferred to other domains (without data from those domains) by pursuing the larger smallest singular value together.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: At the end of abstract, we provide a URL link for anonymized code repository that contains example command for reproduction. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Train and test details are provided in Section 5.2 and in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and discussion: 
 While there have been numerous research endeavors to improve reliability during the model adaptation in the wilds, almost all of them meet the desired criteria only in half: OOD generalization or confidence calibration. This work attempts to address both OOD generalization and OOD calibration in a single framework. We first derive a shared upper bound for OOD classification and calibration errors which is constructed with the ID calibration error and the smallest singular value of ID input representation.We then devise a novel fine-tuning method CaRot, which promotes a larger smallest singular value and calibrated prediction through constrained multimodal contrastive loss and self-distillation. Our theoretical statements and proposed method are empirically validated through extensive experiments.Limitation and future work. Due to resource constraints, our research reached the scale of ViT-L.Exploring validation on the larger models such as ViT-G or ViT-H where the assumptions behind our theory become more realistic would be necessary. Our scope of validation was also limited to CLIP-like VLMs, but Theorem 3 is not specific to VLMs, and investigating the applicability to other types of models such as language models would be an exciting future work direction.Impact statement. Our method enhances the foundation models' reliability in multiple dimensionsaccuracy and confidence calibration, and many downstream applications for society can enjoy benefits from the improved reliability. However, inherent biases learned from ID data can not be removed with our method, and thus may have risk raising potential harms in real-world applications.Standard fine-tuning (FT [61] ): Linear classification head is initialized with text representation vectors for each class encoded by pre-trained CLIP text encoder. Image encoder and linear classifier parameters are fine-tuned for 10 epochs with a learning rate of 3e-5.LP-FT [30] : Randomly initialized linear classification head is first trained upon frozen image encoder for 5 epochs and then both image encoder and linear head parameters are updated for 5 epochs. For each phase, we use a learning rate of 1e-2 and 3e-5, respectively.Fine-tuning with contrastive learning (FLYP [17] ): Both image and text encoders are updated without additional linear classification heads. To create text representations of training samples, we use the OpenAI template similar to the evaluation data. Unlike for evaluation, we do not take an average of 80 different templates. Instead, we use 80 versions of text prompts for each class to build training pairs. The training pairs are randomly selected throughout the training steps. We fine-tune the model for 10 epochs (in total, 25K steps) with a learning rate of 1e-5.Lipsum-FT [42] : Cross-entropy loss with a regularization term that minimizes the energy gap between image and text is used as fine-tuning objective. We followed the details described in the original paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We report statistical significance for toy experiments. However, for large-scale dataset benchmarks (i.e., ImageNet with CLIP ViT-B/16), we do report error bars due to expensive computational costs.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Both positive and negative impacts are discussed at the end of the paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not pose such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper of dataset and methods. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We set the orthogonality constraint coefficient λ OC as 0.2 and self-distillation coefficient λ SD as 1.5, update frequency for EMA teacher as 500, and EMA final target momentum as 0.9. We linearly increased the EMA momentum α by 0.05 for the first 20% iterations. We followed all the other details from FLYP.In addition to the comparisons of our CaRot with zero-shot, naive fine-tuning, and robust finetuning methods in §5, we provide results when applying post-hoc techniques for robustness (weight ensembling) and calibration (temperature scaling). Moreover, we compare our self-distillation-based soft label with uniform constant label smoothing.Weight average of fine-tuned and zero-shot (WiSE-FT [61] ): Zero-shot and fine-tuned model weights are averaged with a strength of ensembling coefficient. This ensembling technique can be applied to any fine-tuning method. We tune the ensembling coefficient for each method based on the ImageNet ECE value (we picked the value having the lowest ID ECE for each method).Temperature scaling (TS [18] ): Before applying the softmax function to compute output probability distribution, TS divides the logit by temperature τ ∈ (0, ∞). τ → 0 makes the probability similar to point masses (sharpening), τ → ∞ makes uniform distribution (smoothing). Scaling the output distribution does not affect accuracy since it does not change the model prediction (i.e., the probability rank remains the same). Temperature value was tuned for each method on the ID validation set based on the ECE value.Label smoothing (LS [55] ): LS is a regularization strategy that pursues the generalization of classification by utilizing soft labels, which are derived by adding uniform distribution to the hard label distribution. The soft label can be viewed as a new target probability distribution where the value of 1 to the target pair is reduced and the value of 0 for the non-target pair is increased by the smoothing parameter ϵ ∈ (0, 1). Since utilizing soft labels allows the model to pull negative pairs with limited strength, LS is beneficial for calibration beyond generalization by addressing the over-confidence issue [38] .Results with WiSE are reported in Table B and C. We observe that weight ensembling, which aims to align zero-shot models and fine-tuned models in the model weight space, boosts the overall performance. Still, CaRot shows superior results to the baselines.Results with TS can be found in Table D . Aligning with previous observations, applying TS significantly improves ID ECE. However, since TS adjusts the logits assuming that all data instances would have a similar extent of overconfidence (or underconfidence) issue, its positive effect is limited under distribution shifts. Hence, TS does not guarantee to attain low calibration error on OOD datasets. It is noteworthy that applying TS even harms OOD ECE of CaRot (compare results with Table 3 ), which is already calibrated using data-dependent confidence adjustment during fine-tuning.Results comparing with LS are reported in Table E . In accordance with its capability of improving calibration, LS successfully reduces calibration errors. We observe that our adopted approach, EMA SD, remarkably outperforms LS, especially on OOD ECE as well as in ID ECE. While LS addresses the over-confidence problem during train time by dispersing the concentrated confidence on the target label to non-target labels with a constant amount, self-distillation shows a similar behavior but in a dynamic approach, considering the diversity of data instances [71] . We interpret that reflecting the difficulty of input batch and distilling such information is crucial to achieving robust calibration.We further provide an intuitive interpretation of self-distillation as an input-dependent approach to label smoothing as elaborated in Zhang and Sabuncu [71] . EMA SD provides soft labels considering the variation of data instances. For example, classifying a dog image from images of airplane or car could be less challenging than classifying it from images of cat or wolf. Ideally, a calibrated model should output higher confidence for the former case than for the latter. Instead of providing supervision of constantly adjusted confidence (as in LS), teacher predictions provide confidence reflecting the input data difficulty. Thereby, the student model can be supervised with high confidence diversity, which leads to a better-calibrated model. Please refer to Zhang and Sabuncu [71] for detailed discussions.Other types of regularization for larger smallest singular value. We adopted an orthogonality constraint over visual projection matrix W v ∈ R dv×r via ||W T v W v -I|| F term which pursues the larger effective rank of the visual projection matrix. While we adopt this kind of indirect soft constraint to achieve balanced performance over both ID and OOD generalization, one may wonder about 1) the implication of increased effective rank of W v on the smallest singular value of the covariance matrix and 2) the possibility of leveraging more direct constraints to increase the smallest singular value of the input covariance matrix. This paragraph answers those questions.Given n samples of d-dimensional visual features and r as a pre-defined projection dimension, let Zv ∈ R n×r and Ẑv ∈ R n×r denote visual representations obtained by an arbitrary projection matrix W v ∈ R dv×r and an orthogonal projection matrix O v ∈ R dv×r multiplied with pre-projected visual feature Z v ∈ R n×dv , respectively. We can show that the rank of the covariance matrix from Ẑv is always greater or equal to that of Zv as below,EQUATIONDuring minimizing ||W T v W v -I|| F , the visual projection matrix W v becomes closer to an orthogonal matrix O v , and the effective rank of the visual representations' covariance matrix increases. A larger effective rank implies a non-diminishing (relatively larger) smallest singular value of a matrix, which justifies our implementation of the method.Meanwhile, we can adopt other constraint terms alternative to orthogonality constraint over visual projection matrix. Using the negative value of the smallest singular value of the visual representation's covariance matrix as an additional loss term may be the most natural candidate. The result is provided in Table B .2. While both SVD and orthogonality constraint methods show remarkably better performance in terms of OOD classification and calibration, SVD requires much heavier computation compared with the orthogonality constraint term. For simplicity, we advocate the use of orthogonality term thus. Exploring other types of efficient constraint terms will be a promising future work direction.Coefficient terms ablation. In Table G and F, we present the detailed ablation results of hyperparameters associated with the methodologies addressed in our paper. These results supplement Table 5 . In this section, we provide proof for Theorem 3 and some discussions. Theorem C.1 (Restatement of Theorem 3.). Let h : X → [0, 1] be a real-valued function which has structure h(x)= d i=1 h i (x[i]) where h i is an arbitrary one-dimensional function, and h(•) is in a hypothesis class H that has pseudo dimension Pdim(H) = d h , DID be an N -size empirical distribution on ID domain. If (x[i], x[j]) is a bi-variate Gaussian random variable for every i, j ∈ [d], then for any δ ∈ (0, 1) and for all h, the following bounds hold with probability at least 1 -δ:EQUATIONiiEQUATIONwhere ΣDID DID [xx T ] is a covariance matrix with a strictly positive minimum singular value ofd-dimensional normalized input x = (x[1], ..., x[d]), where x[i]:=(x[i] -E[x[i]])Var(x[i]) -1/2 and σ min (M ) is the smallest singular value of a matrix M ∈ R d1×d2 .Proof. The proof for the above theorem is divided into three steps: 1) the derivation of OOD calibration error bound, 2) the derivation of OOD generalization bound, and 3) the replacement of domain discrepancy term over ID and OOD into an ID-depend singular value term.We first define a domain discrepancy measure H-sqaure disagreement, sd H , between two distributions as below:Definition C.2 (H-sqaure disagreement). Given two probability distributions D OOD andD ID over input space X , H as a hypothesis class containing the hypothesis h(•) : X → [0, 1], the discrepancy between D OOD andD ID is defined asEQUATIONThe H-square disagreement can be regarded as a variant of H-divergence [5] which adopts mean square error term rather than 0 -1 classification loss or mean absolute error term as in [72] .Proposition C.3 (OOD calibration error bound). Let h : X → [0, 1] be a real-valued function in a hypothesis class H with a pseudo dimension Pdim(H) = d h . If DID is an empirical distribution constructed by N -size i.i.d. samples drawn from D ID , then for any δ ∈ (0, 1), and for all h, a bound below hold with probability at least 1 -δ.EQUATIONNow, likewise Lemma 3 of [72] , we start to review a triangle inequality for all h, h ′ , h ′′ ∈ H, for any D on X , and for error functionε D (•, •), the inequality ε D (h, h ′ ) ≤ ε D (h, h ′′ ) + ε D (h ′′ , h ′ ) holds.Here, we useε D (h, h ′ ) to denote E x∼D [(h(x) -h ′ (x)) 2 ]. Now, given h * := arg min h∈H ε DID (h) + ε DOOD (h) and ∆ := ε DID (h * ) + ε DOOD (h * ), we haveEQUATION)where ε D (h) is defined as ε D (h) := E x∼D [(h(x) -h 0 (x)) 2 ] and h 0 (•) := arg min h∈H E x [(h(x)c(x)) 2 ] denotes a desired calibrated predictor of label y when c(x) := E D [y|h(x)]. Here, the first and fourth inequalities in Eq (15) are held by triangular inequality.The above bound is defined over the true population distribution D ID and D OOD . For the ID domain, we can confine our analysis to empirical distribution DID with N i.i.d. samples generated from D ID , by leveraging a generalization bound on a single domain regression setting [36, 72] . If Pdim(H) = d h , for all h ∈ H, below bound holds with probability at least 1 -δ (See Lemma 5 of Zhao et al. [72] ).EQUATION)The above bound is similar to a regression bound proposed by Zhao et al. [72] , but we build the theory based on the squared error rather than the previously adopted absolute error. This slight change allows us two attractive extensions that we will introduce to achieve calibrated robust fine-tuning.While Proposition C.3 provides guidance to pursue OOD calibration, it does not say anything about OOD classification error, which is our primary goal before calibration. Here, we pay attention to the decomposition of the Brier score [39, 46] :EQUATIONwhere E[(h(x) -y) 2 ] is an expected mean-squared classification error and E[c(x) 2 ] denotes the sharpness [51] term rewarding the predictor h(•) to produce outputs towards zero or one. By assuming that h(•) is expressive enough to estimate the ground truth expectation function for the label, i.e., c(•), plugging Eq. ( 17) into the LHS of Proposition C.3 derives the same (except a constant) upper bound for the sum of classification error and prediction sharpness on OOD samples as in Proposition C.4.Proposition C.4 (OOD generalization error bound). Let h(•), H, and DID have the same definition as in Proposition C.3, then for any δ ∈ (0, 1), and for all h, a bound hold with prob. at least 1-δ,EQUATION)From domain discrepancy to minimum singular value. The second term sd H (D OOD , D ID ) of RHS of Proposition C.3 and Proposition C.4 is defined on both ID and OOD samples, so it is hard to control the quantity directly. While existing approaches attempt to learn domain-invariant representations for reducing the similar quantity (e.g., H-divergence) by using unlabeled OOD data [14, 72, 73] , the OOD data are commonly inaccessible on many real-world applications. Therefore, we need a quantity that is solely defined with ID data. Recently, Dong and Ma [11] proved that the domain discrepancy ratio can be bounded from above by a reciprocal of the smallest singular value of a covariance matrix of ID data as below:EQUATIONwhere ΣDID := E DID [xx T ] is a covariance matrix of the d-dimensional nomarlized input x = (x[1], ..., x[d]) where x[i] := (x[i] -E[x[i]])Var(x[i]) -1/This can be easily shown by the definition of sd H (D OOD , D ID ),EQUATIONEQUATION)Where the first inequality is held by the assumption of the Lemma C.5 and the second inequality is held by Theorem 3 of [11] . Discussion on the tightness of proposed bound Our theoretical analysis was inspired by two previous works that provide OOD generalization error bound [5] and the domain discrepancy bound via the smallest singular value [11] . Intuitively, the inequality of our bounds (ineq.( 1)) approaches to equality (becomes tight) under the following conditions:1. When the outputs of the learned classifier are the same as the outputs of the ideal joint classifier that is trained to minimize both ID and OOD errors [5] . 2. Whether the outputs of our classifier or ideal joint classifier are perfectly calibrated [5] .3. The number of training samples N approaches infinite [5] . 4. If our classifier is a linear model when the weight vector is the same as the eigenvector corresponding to the smallest eigenvalue of the input covariance matrix [11] . 5. When the OOD and ID calibration errors satisfy the relationship:ε DOOD (h, h * ) = ε 2 D ID (h,h * ) (ε D ID (h,h * )-1) .Although our bound seems quite loose given the high dimensionality of modern machine learning setup, empirical validation in the paper strongly supports the validity of our theory-grounded method.Discussion on the gap between reality and assumption Some ground assumptions derive our bound. First, we assume that ID and OOD have overlapping marginal distributions, whereas the joint distributions do not overlap much. Second, we assume that our hypothesis function has the structure of h(x) = d i=1 h i (x[i]). Finally, we assume the pair-wise bi-variate normality of (x[i], x[j]) for every i, j ∈ [d]. Given that the input x here is the last layer representation of the visual encoder, ID and OOD have overlapping marginals, but we can not guarantee that the joint distributions of input do not overlap much. For example, there would be a high overlap between ImageNet (ID) and ImageNet-V2 (OOD) in terms of the input joint distributions, but there would not be much overlap in the case of ImageNet-Sketch as an OOD. While it depends on the type of OOD, our evaluation shows robust results across datasets. The second assumption over the structure of our hypothesis function is naturally met if our h(•) is a linear classifier, which is realistic under the modern representation learning paradigm with a heavy feature extractor and a light task-specific head. Lastly, about our last assumption, the representation of neural networks is not usually guaranteed to be pair-wise Gaussian. However, there is rich evidence revealing that the outputs of infinitely wide neural networks (whether they are MLP [31] , convolutional neural network [15] , or Transformer [62] ) are Gaussian processes (GPs) and GPs ensure that every finite collection of their elements are jointly Gaussian. While our numerical analyses are limited to ViT-L scale models, we believe that the current large-scale modeling regime spurs us to explore larger widths [9] where our second assumption becomes more valid [31] .• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.We report the average accuracy over four shifted datasets in Table6. Results with an asterisk (*) are taken from the original papers. ECE values are not included due to the missing evaluations from the original papers. 


Decision: The paper proposes a method for improving out-of-distribution confidence calibration in vision-language models. The proposed method involves finetuning with a constrained multimodal contrastive loss that minimizes an upper bound on out-of-distribution calibration. Reviewers were generally positive about the paper, highlighting the significance of the theoretical result and the excellent presentation of the paper, but also raised concerns about the fact that the proposed method only leads to small improvements in performance and that the proposed regularization term is not specific to vision-language models. While I sympathize with these concerns, I believe that---although the framing of the paper likely reduces the impact of the theoretical results on the broader community---the contributions of the paper are sufficiently significant and of sufficient interest to the NeurIPS community to warrant publication. In conclusion, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This research paper presents a novel fine-tuning approach for improving out-of-distribution (OOD) generalization and calibration in Vision Language Models (VLMs). By identifying a shared upper bound for OOD accuracy and calibration errors, the authors develop a constrained multimodal contrastive loss framework enhanced by self-distillation. Empirical validation and tests on ImageNet benchmarks demonstrate the method's effectiveness in enhancing both OOD accuracy and calibration error.
strength_weakness: strength_weakness: - The theory part is a bit obscure. Is there an intuitive interpretation of the smallest singular value of ID input covariance matrix in the studied context? How tight is the upper bounds (Theorem 3.1)?
- Were the main experiments repeated multiple times? If so, can the authors provide the error bars for the main experiments?  - The paper limits the scope to VLM fine-tuning, but does it also apply to fine-tuning other kinds of models?
questions: questions: Please see the weaknesses.
limitations: limitations: The authors discussed one limitation of the work. I don't notice any potential negative societal impact.
paper_summary: paper_summary: VLMs have shown to be effective in a wide-area of applications, though they can fail under certain domain shifts. In this work, the authors first observe that the the upper bound of generalization and calibration under domain shifts is bounded by the ID calibration error and the smallest singular value of the ID covariance matrix. Building upon this intuition, the authors then propose a novel fine-tuning scheme for VLMs in the shape of a contrastive objective and a self-distillation technique. Experimental evaluation on Imagenet and quite a few of its variants are presented to show the effectiveness of the proposed approach.
strength_weakness: strength_weakness: - In certain cases, such as Imagenet-R on Table 2 and 3, it appears that the zero-shot CLIP is better than _any_ of the fine-tuning methods including the proposed approach in terms of _both_ the accuracy and calibration. Furthermore, the calibration after fine-tuning with CaRoT is worse than the zero-shot CLIP under harsher domain-shift benchmarks, namely the ObjectNet, Imagenet-{A, S, R}. I wonder how would the Table 2 look like had the model had access to multiple training environments (e.g maybe having a subset of Imagenet-S during fine-tuning, then evaluated on Imagenet-R) as it is often the case for domain generalization benchmarks [A].
- On Tables 2 and 3, the proposed approach seems to be falling behind of the other methods in terms of Imagenet accuracy, which may limit its usefulness for a wider range of applications.
- It would have been good to have Imagenet-C [B] here as well, as it is perhaps the most commonly used among these variants for domain shift benchmarking. In particular, having some analyses based different types of corruptions and under different severity levels could have provided more insights into the limitations and strengths of the proposed approach.
- One of the minor issues I can see with the work is regarding the usage of the term "OOD", especially since the benchmarks used vary significantly from Imagenet-V2 (which was designed to be distributionally as similar as possible to the original Imagenet) to Imagenet-S. While I acknowledge that the authors clarify what they mean under Section 2, I still encourage them to check out [C] for a nice template on works involving the term. 



[A] Guljarani et al., "In search of lost domain generalization", ICLR 2021

[B] Hendrycks et al., “ Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”, ICLR 2019

[C] Farquhar et al., "What ‘Out-of-distribution’ Is and Is Not", NeurIPS-W 2022
questions: questions: I wonder what the authors think about the first weakness I have described above and it would be great to see detailed Imagenet-C results.
limitations: limitations: The limitations are somewhat discussed in the conclusion. I appreciate the fact that they have stated that they could not include larger models due to computational constraints which prompted me to avoid asking for how the proposed method would perform with larger models trained with much larger data.
paper_summary: paper_summary: his paper aims to improve the accuracy and reduce the calibration error on OOD data for fine-tuning VLM models. The authors first demonstrate that the OOD calibration error and the OOD classification error can be bounded by the ID calibration error and the smallest singular value of the ID input covariance matrix. To address this, the authors apply orthogonal regularization to increase the smallest singular value and use self-distillation to improve the ID calibration. Several experiments on distribution-shifted datasets validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: (1) Some of the notations are misleading. In line 86, the subscript of $x$ denotes the  $n$-th sample in $\mathcal{D} $. However, in line 113, the subscript of $x$ denotes the value of the $i$-th dimension of $x$.

(2) Some of the assumptions are not realistic. In line 113, the authors suppose the hypothesis $h_i$ is a one-dimensional function, each applying to one dimension of the input, which does not hold most of the time. Besides, the authors assume each dimension pair is Gaussian. I am unsure whether the conclusion of Theorem 3.1 still holds if these two assumptions are violated.

(3) The orthogonal constraint is too strong. The finding of Theorem 3.1 states that lifting the smallest singular value is enough to reduce calibration and classification errors. However, the authors implement an orthogonal constraint in the learning procedure, enforcing the features to be an orthogonal basis, which leads to a smaller largest singular value. In Table 3, we can see that applying such regularization alone reduces ID accuracy. I wonder whether applying a regularization that only increases the smallest singular value could achieve higher performance.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Limitations have been discussed and no negative societal impact has been identified.
paper_summary: paper_summary: The paper proposes a novel framework for robust fine-tuning for CLIP. To enhance out-of-distribution accuracy and calibration, the author incorporates a singular-based constraint term, self-distillation, and EMA. Extensive experiments on synthesized data and ImageNet demonstrate that CaRot can achieve better OOD performance and reliable predictions.
strength_weakness: strength_weakness: 1. The overall framework is not well-motivated. Why do we need to incorporate self-distillation and exponential moving average (EMA)? These two techniques are not the main contributions of this paper and are not directly relevant to the primary theoretical analysis of singular value constraints.
2. The connection to Vision-Language Models is weak. The paper does not justify why the proposed soft constraint term is specifically tailored to Vision-Language Models like CLIP. In other words, does this regularization only apply to visual language models?
3. The result analysis is insufficient. The paper does not report the average confidence level, which may lead to the confusion that the calibration improvement may only come from the accuracy improvement.
questions: questions: 1. Can the proposed singular value-based constraint improve other fine-tuning methods, such as FLYP or WISE-FT?
limitations: limitations: No

Comments: title: Sincerely looking forward to your feedbacks
Reminder for discussion, Reviewer a8to
Further Suggestions for Improvement
comment: Dear Reviewer yVZD,

We would like to express our huge gratitude for your invaluable comments so far.
Your comment has definitely improved the quality of our manuscript and led us to refine our statement further.
For now, we are wondering if our responses address your remaining concerns. Could you check our responses by any chance?

* We clarify that our theory is not confined to VLM for its generality and broader impact, but our method—integration of multimodal contrastive learning with orthogonality constraint—enables us to interpret our method as a constrained singular value decomposition on the cross-covariance matrix of image-text representation pairs (Please refer to Section 4.1 of our manuscript) during fine-tuning of VLM.
* Regarding your second concern, we provide counter-examples showing that improved accuracy does not translate to better calibration (with some references) and clarify the experiment results to fix misunderstanding, thereby demonstrating that our CaRot is not just an accuracy-improving method but also promotes non-trivial confidence calibration.

If these replies address your concerns, could we politely ask you to reconsider your rating on our paper?

Sincerely,

The Authors
The authors have addressed my concerns effectively, and I have increased my score accordingly.
Dear reviewer yVZD,

We are delighted that some of your concerns have been addressed.

For the remaining concerns, 
* [first concern] As you have pointed out from the original review, we have derived our theory in a common classification setting which is not confined to the vision-language model for the generality of the theoretical statement, and we observe that our OOD error bound-based regularization method is also effective on a vision-only model during the rebuttal.
  * About the connection between VLM and the proposed method, as described in Section 4.1. of the draft, our orthogonality constraint on the visual projection layer is seamlessly combined with a multimodal contrastive loss so that enables it to be interpreted as a constrained singular value decomposition (SVD) on the cross-covariance matrix of image-text representation pairs. 
  * We appreciate you found that our theory and corresponding method have broad potential applications, and respect your worry. we will reflect your comment to refine the presentation of our revised manuscript. Thanks again.
* [second concern] We would like to respectfully refute your claim: is CaRot's improved confidence calibration merely due to accuracy improvement?
  * Allow us to clarify that Table 2 of the attachment is the ablation study of the orthogonality constraint (OC) on FLYP and WiSE-FT, which shows the consistent effect of OOD improvement of OC. **The result of CaRot is not included in that table**. We attached the Table with the results of CaRot below, and demonstrate that **CaRot improves OOD accuracy and ECE as 5.05 (8.7%) and 0.1395 (63.8%). Thus the improvement in terms of ECE is far more significant than the improvement in accuracy.**
  
| Method     | WiSE-FT | ID Acc | ID ECE | OOD Acc | OOD ECE |
|------------|---------|--------|--------|---------|---------|
| FT         | X       | 81.53  | 0.0884 | 57.50   | 0.2186  |
| FT w/ OC   | X       | 81.45  | 0.0826 | 59.10   | 0.2051  |
| FLYP       | X       | 82.69  | 0.0635 | 59.46   | 0.1831  |
| FLYP w/ OC | X       | 82.51  | 0.0651 | 59.51   | 0.1803  |
| FT         | O       | 82.16  | 0.0820 | 61.22   | 0.1920  |
| FT w/ OC   | O       | 82.03  | 0.0770 | 61.97   | 0.1829  |
| FLYP       | O       | 82.98  | 0.0798 | 61.27   | 0.1788  |
| FLYP w/ OC | O       | 82.80  | 0.0627 | 61.41   | 0.1682  |
| **CaRot**      | X       | 83.13  | 0.0470 | 62.55   | 0.0791  |

* [second concern] Meanwhile, we partly agree with your statement that improvement in accuracy could contribute to calibration (or vice versa) somewhat. **However, even though the accuracy and calibration are correlated sometimes, they are not a causality.** For example, Figure 2 of Guo et al. 2017 shows that improved accuracy hurts calibration, and Table 1 of Levine et al. presents that CLIP ViT-H-14 and CLIP ViT-B-16 have worse calibration than CLIP ViT-L-14 and CLIP ViT-B-32, respectively, even though they achieve far better classification accuracy. Figure 9 of Dehghani et al. also implies that increased accuracy does not result in improvement in calibration.
  * We observe the same evidence in Figure 6 of the attached PDF file under our experimental setup. For instance, from the left-most point of FT to the left-most point of FLYP, Accuracy is roughly improved from 81.0 to 82.0 (+1.2%), but the ECE is increased (`worsen`) from 0.06 to 0.08 (+33.3%). Meanwhile, CaRot achieves Accuracy and ECE of 83.0 (+2.4%) and 0.05 (-20%; `become better`). This indicates that **CaRot is not just a method for accuracy improvement, but it is a method for accuracy and calibration simultaneously in a single theory-motivated framework.**

### Reference

1. Guo et al. 2017, On Calibration of Modern Neural Networks
2. Levine et al. 2023, ENABLING CALIBRATION IN THE ZERO-SHOT INFER- ENCE OF LARGE VISION-LANGUAGE MODELS
3. Dehghani et al. 2023, Scaling Vision Transformers to 22 Billion Parameters

We thank you again for the reviewer yVZD's intensive commitment to reviewing our paper and we appreciate valuable comments that contribute to improving the quality of our draft.

Best regard
Dear reviewer a8to,

1. We truly appreciate your suggestion on fixing notation, clarifying assumptions of the theory, and discussing potential alternative design choices of constraint terms. We promise to update our manuscript thoroughly, as you pointed out, and this will significantly refine the presentation quality from now on.
2. We also appreciate your further interest in the behavior of orthogonality constraint (OC) and the consideration of raising the score!


Allow us to clarify the experimental setup for Table 3 and Table 4 of our manuscript. The left side of **Table 4 is the results of the ablation study where the self-distillation (SD) regularization term is applied with a 1.5 multiplier**. That is, the first and third rows of Table 4 are equal to the seventh and eighth rows of Table 4. For a clear comparison, we insert a table below that includes results with and without SD under varying OC magnitudes. Meanwhile, As you can see **by comparing the first and second, and the fifth and sixth rows of Table 3, orthogonality constraint (`without SD term`) slightly tradeoffs ID accuracy**. This phenomenon is also observed through additional ablation studies with the WiSE-FT method in Table 3 of the rebuttal attachment PDF.

| Objective | OC  | ID Acc | ID ECE | OOD Acc | OOD ECE |
|-----------|-----|--------|--------|---------|---------|
| MCL       | 0   | 82.69  | 0.0635 | 59.40   | 0.1836  |
| MCL       | 0.1 | 82.48  | 0.0652 | 59.41   | 0.1807  |
| MCL       | 0.2 | 82.51  | 0.0651 | 59.51   | 0.1803  |
| MCL w/ SD | 0   | 83.03  | 0.0523 | 62.28   | 0.0772  |
| MCL w/ SD | 0.1 | 83.18  | 0.0511 | 62.42   | 0.0779  |
| MCL w/ SD | 0.2 | 83.13  | 0.0470 | 62.55   | 0.0791  |


**Therefore, we derive non-contradictive conclusions from Tables 3 and 4 of the manuscript, indicating that orthogonality constraint alone somewhat compromises ID adaptation capability for OOD generalization, while this tradeoff is mitigated when the SD is applied together, which is our final learning objective.**

We speculate that,
1) When the orthogonality constraint is applied alone, the model is enforced to capture diverse features for OOD generalization yet without any restriction on the type and priority of learned features. While this contributes to enhancing OOD generalization, **diverse features without prioritization might compromise strong ID performance.**
2) However, the SD regularization produces input-dependent soft labels that hold similarity structures between classes. This allows the model to learn diverse features while **putting a higher priority on features shared across similar classes** (judged by the EMA teacher model) so that the features are beneficial not only for OOD generalization but also for ID adaptation. We can understand the joint use of the orthogonality constraint and self-distillation regularization induces a narrower solution set, which potentially induces better generalization (Huh et al. 2024).

[Huh et al. 2024] The Platonic Representation Hypothesis

Thanks again for your constructive comment so far.

Best regard
Dear Reviewer g5ML,

We sincerely appreciate your further clarification, and we want to express our huge gratitude for the positive rating incline to acceptance! 

In this work, 
* we **provide a novel theoretical bound on both OOD generalization and calibration errors in a single unified formulation that solely requires ID-related terms**, i.e., ID calibration error and the smallest singular value of the ID covariance matrix. 
* **CaRot, which is designed to minimize this error upper bound, was extensively validated on the large-scale dataset across diverse backbones**. 
* Through the ablation studies, we confirm that the **adoptions of each component**, i.e., orthogonality constraint and calibration regularization, of CaRot **induce consistent improvements in the OOD generalization and calibration**.

We respect your viewpoint on the effectiveness of the orthogonality term and thank you for the opinion of direct regularization on the feature singular value. While our orthogonality constraint was introduced to promote a larger smallest singular value of the feature covariance matrix, **it also allows us to view our multimodal contrastive loss as a constrained singular value decomposition (SVD) on the cross-covariance matrix of image-text representation pairs that helps interpretation of our method**.

In Table 1. of the attached PDF, we included a result of the direct feature regularization method based on the singular value decomposition (SVD) and using the negative smallest singular value as an additional loss term. This method achieved promising outcomes in terms of OOD performance. We will further investigate the other direct constraint terms in future work.

About your second concern 'weak linkage between the method and VLM, framing of paper', while the combination of each component has originality in terms of theoretical motivation and interpretation, we respect and will accommodate your opionion to increase the impact of our work.

Thanks again for your invaluable feedback so far. We believe that our manuscript will be significantly improved. 

Best regard
My initial concerns are addressed and I really appreciate the authors' detailed, thoughtful responses. They helped me better understand the paper and see clearer its merits and defects.

The reason I would like to keep my score is that the improvement brought by the proposed orthogonality constraint appears to be somewhat marginal. The constraint only deals with the loss of feature diversity at the last projection layer, disregarding potential losses at earlier stages, which are probably more consequential. For future improvement, I wonder if a similar constraint can be directly applied to the features themselves (as the theory implies) rather than the layer weights.

Finally, the overall method, CaRot, feels like a naive combination of three separate components. In particular, as pointed out by Reviewer yVZD, the connection of the method to vision-language models is weak. Although the authors have presented further results on vision-only models, this does not fix the framing of the paper (fixing it would probably require a major revision).

In all, while I am inclined to accept the paper, the impact of the paper in its current form may be limited so I think a borderline accept is adequate.
Thanks for the response, and some of my concerns have been addressed.

However, I still have the following two concerns:

1. The main issue in this paper is the calibration for CLIP. However, the connection between the proposed method and vision-language model (CLIP) is still weak. I think that the proposed method can be applied to most classification models and is not limited to CLIP.

2. The calibration improvement brought by CaRot seems to come only from accuracy improvement. It does not address the overconfidence issue of CLIP caused by fine-tuning. For example, in Table 2 of the attachment, compared to the baseline, the accuracy of FT increased by 4%, but the ECE only decreased by 3%. Hence, It appears that CaRot is merely a method to improve accuracy rather than a confidence calibration method.

I will keep my original rating.

Best regards,

yVZD
Dear Authors,

Thank you for your detailed responses, which have addressed most of my concerns. I would like to kindly request that you revise the notations, add an explanation of the assumption of the one-dimensional function, and discuss other possible design choices for the smallest singular value regularization in the updated version.

Before I raise my rating, I have one additional question regarding the ablation study in Table 4 and Table 3: In Table 3, applying orthogonal regularization seems to reduce ID accuracy. However, in Table 4,$\lambda_{OC} = 0.1$ results in higher ID performance than $\lambda_{OC} = 0$, which appears to contradict the conclusion of Table 3. Could you please further explain this discrepancy?

Best regards,
Dear Reviewer yVZD,

We appreciate reviewer yVZD's constructive feedback, which helped us improve our draft further.

We have submitted our responses to concerns raised by reviewer yVZD, and we are eager to know if these replies address your concerns!

Any further comments or questions are welcome to us.

Thank you
Dear Reviewer a8to,

We appreciate reviewer a8to's valuable comments that significantly contribute to improving our manuscript.

We have submitted our responses to reviewer a8to's concerns, and we want to know if these replies address your concerns!

Any further comments or questions are welcome to us.

Thank you
We are so delighted that our rebuttal addresses your concerns and questions!
Thank you for your valuable comments and for taking the time to review our paper thoroughly.
Thank you for your detailed feedback and inquiry so far! 

Your comments were highly valuable to us. We have thoroughly addressed the issues raised by Reviewer g5ML.
We believe these revisions will enhance our manuscript significantly.

**We respectfully inquire if you could consider raising the score, given that all of your concerns are addressed.
If not, we are interested in understanding if there are any additional concerns or suggestions that we could address to improve the submission further.**

Again, we would like to express our gratitude for your commitment to reviewing our paper thoroughly and providing constructive feedback, which helps us refine our work.
I appreciate the hard work the authors have put into the rebuttal. They seem to have answered my points fairly well, especially with the detailed DomainBed and Imagenet-C experiments and visuals. I am increasing my score accordingly.
Thank you for addressing my queries. I have no further questions or comments. I would like to keep my score for now.
Thank you for your active query!

### 1. Rank comparison

> Allow us to recap the rank of matrix multiplication, 
$\\text{rank}(AB) \\le \text{min}(\\text{rank}(A),\\text{rank}(B))$
where the equality only holds whether matrix $A$ or $B$ has full rank.

Then, for the neural network representation of our case, **the post-projection feature always has a smaller rank compared to the pre-projection feature if the projection matrix $W$ is not full rank**. There is a solid theoretical background about this so-called **rank diminishing phenomenon** [Feng et. al 2022].

As you said, our constraint can't increase the rank of the pre-projection feature and just preserve it in the ideal case; however, this preservation induces the higher rank of the post-projection feature compared with the baseline, which does not encourage the projection matrix to be full-rank.
Therefore, our method induces a higher effective rank of the feature covariance matrix compared to baseline methods, which do not enforce rank preservation, and our method induces learning of diverse features that contribute to better OOD generalization.

_We really appreciate this valuable query, and will revise our manuscript to clarify further the connection between singular values, rank, and OOD generalization._

### 2. Significance of performance gain
We would like to emphasize that, in Table 2 of the attached PDF file, our orthogonal constraint, OC for short, **consistently improves the OOD accuracy and ECE in all 8 cases**, as well as a case of the toy experiment in the left side of Figure 3 in the manuscript. While the amount of improvement might be seen as small, we believe **this consistent improvement (9 out of 9) indicates strong evidence of the effectiveness of our theory-motivated rank-preserving constraint for OOD generalization**, which is complementary to the benefits from the FLYP or SD. Moreover, the amount of performance gain is sometimes significant, e.g., +1.6 in terms of accuracy of FT w/ and w/o OC, -0.01 in terms of ECE of FLYP-WiSE w/ and w/o OC.

Please do not hesitate to ask us any further questions; we will be delighted to have an extended discussion with the reviewer g5ML.

---

### reference 
[Feng et al. 2022] Rank Diminishing in Deep Neural Networks
Thank you for the further clarifications.

I get the idea that a higher effective rank of the feature covariance matrix implies higher feature diversity. However, I'm still confused about the role of minimizing $\lVert W_v W_v^T - I\rVert^2_F$ in achieving such a goal.

Suppose $W_v = I$, which is the best we can get. This means the projection layer preserves the effective rank $r$ of the feature from the previous layer, so we have $r' = r$ where $r'$ is the effective rank of the projected feature. The question is, does this have any effect on $r$ itself? To me, it seems the answer is no because the effective rank $r$ of the pre-projection feature may be low even if $\lVert W_v W_v^T - I\rVert^2_F = 0$, i.e., they are independent.

Empirically, the ablation study (Table 3) suggests that the proposed constraint, i.e., $\lVert W_v W_v^T - I\rVert^2_F = 0$, plays a small role. The main improvement is brought by FLYP and SD but they are not the main contribution of this work (as partly mentioned by Reviewer yVZD). Could the authors please further comment on this?