Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance
Authors: Giung Nam
Byeongho Heo
Juho Lee
Venue: ICLR 2024 poster (2024)
Keywords: computer vision
vision-langauge model
transfer learning
fine-tuning
distribution shifts
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a novel robust fine-tuning method tailored for vision-language models.

Abstract: Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across a range of image classification tasks without requiring training on downstream data. Recent works have confirmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descriptions based on feature distortion theory and joint energy-based models. Subsequently, we propose a novel robust fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language modeling aspect of the vision-language pre-trained models. Extensive experiments conducted on distribution shift scenarios in DomainNet and ImageNet confirm the superiority of our proposed Lipsum-FT approach over existing robust fine-tuning methods.

Published as a conference paper at ICLR 2024
LIPSUM-FT: ROBUST FINE-TUNING OF ZERO-SHOT MODELS USING RANDOM TEXT GUIDANCE
Giung Nam
Byeongho Heo? NAVER AI Lab
Juho Lee AITRICS
KAIST AI {giung,
juholee)ekaistt.c..
th.hho@navercorp.com
ABSTRACT
Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across range of image classification tasks without requiring training downstream data. Recent works have con- firmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's s robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descrip- tions based on feature distortion theory and joint energy-based models. Subse- quently, propose a novel robust fine-tuning algorithm, L,i taipuum-Pa FT,, that ef- fectively utilizes the language modeling aspect of the vision-language pre-trained models. Extensive experiments conducted on distribution shift scenarios in Do- mainNet and ImageNet confirm the superiority of our proposed Li Lipsum-FT ap- proach over existing robust fine-tuning methods.
INTRODUCTION
Recent advances visual representation learning have been achieved through large-scale contrastive Ausion-language pre-training (Radford et al., 2021; Jia et al., 2021; Pham et al., 2023). A prominent instance that exemplifies this trend is the Contrastive Lanuuage-Imag Pre-training (CLIP; Radford et al., 2021), methodology that leverages natural language supervision to enhance visual repre- sentation learning. Capitalizing on its language modeling component, the zero-shot CLIP model, in conjunction with a classification head tailored using class label text for downstream tasks, adeptly conducts image classification tasks without requiring extra optimization specifically on the down- stream images.
Although the vision-language models achieve remarkable performance without extra training in downstream tasks, fine-tuning on the reference data still proves be effective way improve downstream performance significantly. Nonetheless, this fine-tuning procedure compromises ro- bustness: the accuracy of the fine-tuned model decreases across distribution shifts compared the accuracy of the initial zero-shot model (Radford et al.. 2021; Pham et al., 2023). Thus, it is worth ex- ploring novel technique for robust fine-tuning that can alleviate the performance trade-off between reference and distribution shift data.
While recent works have been conducted in developing robust fine-tuning methods applicable to models (Wortsman et al., 2022b; Tian et al., 2023; Mao et al., 2023), these ap- proaches tend to neglect the incorporation of the language aspect during the fine-tuning process. Since the nature of lision-language models is their multimodal structure encompassing both images and text, we argue that the existing robust fine-tuning methods do not effectively harness the infor- mation embedded within the language modeling aspect. Consequently, propose novel strategy for robust fine-tuning that effectively utilizes the language model, aiming to enhance robustness.
The main contributions of our work can be outlined as follows:
We re-verified the performance trade-off the reference and distribution shift data af- ter fine-tuning of CLIP- VIT models (Radford et al., 2021). Our empirical investigation revealed that the recently introduced feature distortion theory (Kumar et al., 2022) lacks

---

Published as a conference paper at ICLR 2024
Foo Zero-shot Vision model
Foo (x)
Ego.0 & (x, tl...)
seiionlanguagg.cconeee before fine-tuning (Zero-shot)
Random texts
'wy prism ayat harold' 'locks sasha tte aras'
(o(ti) (o(ts)
EnergyGap(Pn) 8n) = -Eor, - =bo.
G Zero-shot- langüäge madel
'amt shield gam ghead'
(a(tsr)
Vision-language connection after tuning Eer (2, tl...)
For Fine-twoed -iision model
Fe (x)
Figure 1: Overview. We first verify that standard fine-tuning declines the vision-language connec- tions in the pre-trained CLIP model, as evidenced by changes in the energy function E after T fine-tuning steps, denoted as Og - Or ($ 4.2). Subsequently, we propose a simple yet effective novel robust fine-tuning method, Lipsum-Er which regularizes the EnergyGap( 0r, 0o) easily derived from language model outputs for random texts during fine-tuning ($ 5.1).
a comprchensive explanation for the robustness observed in distribution shifts during the fine-tuning of zero-shot CL.P--iT models.
In lieu the feature distortion theory, els (Grathwohl et al., 2019) investigate the underlying factors contributing the robust- ness of the existing robust fine-tuning methods. To summarize, our research showed that the fine-tuning process disturbs the connection between the vision and language models, as evidenced by alterations in the energy values.
we introduced a concept of joint energy-based mod-
We propose a novel robust fine-tuning method, Lipsum-FT, tailored for vision-languee models. Lipsum-FT utilizes language model outputs to align the fine-tuned model with the zero-shot model. Notably, Lipsum-FT is the pioneering effort that considers the language modeling component of vision-language models for robust fine-tuning.
Figure depicts an overall idea of our proposed L.i psum-FT approach. Through empir- ical validation using the DomainNet and ImageNet datasets simulate distribution shift scenarios, we demonstrate that Lipsum-1 FT outperforms previous methods in terms of both prediction accuracy and uncertainty estimation ($ 5.2).
PROBLEM SETTING
Throughout the paper, we consider image classification tasks using pre-trained vision-language models (Radford et al., 2021; Jia et al., 2021; Pham et al., 2023). To elaborate on our problem setting, we will primarily discuss CLIP (Radford et al., 2021), yet is worth mentioning that other lasion-language models can be described in similar manner.
Zero-shot image classification. CLIP consists of two separate models; (1) a vision model Fo parameterized by 0, which maps images into the D-dimensional representation space, and (2) language model G6 parameterized by Φ. which maps texts into the same representation space. For K K-class classification problem taking images æ, CLIP demonstrates the ability to predict the target label in zero-shot manner. Specifically, we can compute the class logits as u(x) = WFo (x), where the classification head weights W is configured by the language model G& and the class label text f,,...,4 tx (e.g., 'a photo class name"). )
W (t)(t))
Go(tR)] KND
2

---

Published as a conference paper at ICLR 2024
The logits u(x) IK* for imput transformed into K-class probability vector using the softmax function, denoted as p(y|z)) = Softmax(u(ar) (x)))))(()....)))))))))))))))))) Namely,
exp (ul) for I....,K. K.
ply k|z) Softmax (u(x))
(2)
>K 2n=1 exp (u()(x))
Fine-tuning of the zero-shot model. The zero-shot CLIP model demonstrates competitive perfor- mance in image classification tasks. However, fine-tuning the zero-shot model can further improve the downstream performance of the zero-shot model. A conventional fine-tuning procedure involves simultaneously updating the vision model parameters 0 and the downstream classification head W minimize the cross-entropy loss function,
Ce O, W)
log (Softmax (K) (WFo(x))) (m,,))ED
Σ
(3)
where æ and & respectively are an image and its corresponding label index in the training dataset D. We refer readers to Appendix B.1 for more detailed description.
RELATED WORK
This section mainly focuses the works that follow the advent of the large-scale vision-language model (Radford et al., 2021). Appendix A provides further discussion of previous work in transfer learning, out-of-distribution generalization, and other relevant areas.
Robust fine-tuning of vision-language models. In line with the recent progress in large-scale vision- laiiunla models, several works have recently delved into the realm of robust fine-tuning of these models: Wortsman et al. (2022b) demonstrated that simple weight averaging strategy, which ensembles the weights of the zero-shot and fine-tuned models, effectively enhances robustness; Tian et al. (2023) proposed method where the fine-tuned weights are projected into the neighboring region of the zero-shot model using different projection radii for each layer. work to ours, Mao et al. (2023) utilized context information derived from the language model.
closely related
Theory of transfer learning. While much the current research on transfer learning focuses pri- marily on linear probing due to the complexity involved in analyzing the fine-tuning process (Tripu- raneni et al., 2020; Du et al., 2021). a recent investigation by Kumar et al. (2022) introduced the concept termed feature distortion theory. This theory proposes that fine-tuning could potentially distort the features learned during pre-training, potentially resulting in decreased performance on data that falls outside the distribution of the training data. However, real-world scenarios may not always align with theoretical assumptions, the feature distortion theory alone does not provide comprehensive explanation for robustness practical applications (Trivedi et al., 2023).
UNDERSTANDING FINE-TUNING OF ZERO-SHOT MODELS
This section examines the implications of fine-tuning for zero-shot vision-language models on vi- sion tasks, particularly concerning distribution shifts. To this end, present extensive array of experimental outcomes across the following two scenarios using CLIP-ViT: (1) DomainNet, where DomainNet R employed as the reference training data, and evaluation encompasses four distri- bution shifts Doma ComainNet..o - (P, C, I, SJ: (2) ImageNet, where ImageNet serves as the reference training data, and evaluation entails four distribution shifts ImageNet -(V2, R, A, S). Please refer to Appendix C for more details on these datasets.
4.1 STANDARD FINE-TUNING
It is common practice to initialize downstream classification head with zero-shot weights derived from the language model when fine-tuning vision-language models (Wortsman et al., 2022b; Tian et al., 2023; Mao et al., 2023). We denote this standard fine-tuning approach FT throughout the paper and refer readers to Appendix B.1 for supplementary fine-tuning results using alternative head initialization strategies.
3

---

Published as a conference paper at ICLR 2024
B/32
B/16
L/14
B/32
B/16
L/14
95
95
DomainNet- R-R
90
DomainNer-
90
85
85
80
75
45
50 55 60
65
50
55
60
65
DomainNet- (P, S,
DomainNett (P,C,I,S))
Figure 2: Trade-off plots on DomainNet. In plots, the vertical axis shows accuracy on the ref- erence data, whereas the horizontal axis indicates average accuracy on distribution shifts, i.e., the top-right represents a better case. The star markers * correspond to the zero-shot model, while the square markers O denote the model fine-tuned with 5000 training steps and a learning rate of 1e-05. Left: The number of training steps is kept at 5000, while the learning rate varies as 1e-06, 3e-06, 1e-05, and 3e-05 (the leftmost point denotes 3e-05). Right: The learning rate is constant at 1e-05, while the number of training steps is altered to 1000, 3000, 5000, and 10000 (the leftmost point denotes 10000). Refer to Figure 7 for the results on ImageNet.
Decrease in performance on distribution shifts. Radford et al. (2021) observed that fine-tuning CLIP-ViT models leads to improved accuracy on the source distribution, but it also results in reduction in accuracy on distribution shifts. Figure 2 verifies this trade-off, where more fine-tuning of zero-shot models (represented by star markers s;) through higher learning rates more training steps leads to enhanced performance on the reference data (moving upwards), but also corresponds to decrease in performance on distribution shifts (moving the left).
Examining the feature distortion theory. We first investigate this phenomenon using the feature distortion theory (Kumar et al., 2022). According to the feature distortion theory, fine-tuning mainly modifies features for the reference data rather than for distribution shift data. Kumar et al. (2022) assert that this is the underlying reason for the performance trade-off observed between reference and distribution shift data and empirically validate it in the context of fine-tuning ResNet-50 that is pre-trained using MoCo v2 (Chen et al., 2020c). They further demonstrate that LP-FT perform- ing fine-tuning after linear-probing, effectively alleviates feature distortion, resulting in enhanced performance on distribution shifts.
However, our findings demonstrate that fine-tuning of CLIP-ViT models does not exhibit greater feature distortion in reference data compared distribution shift data. In Figure 3, we computed the Euclidean distance between features before and after the fine-tuning procedure to quantitatively measure the extent of distortion (y-axis), line with the experimental design presented by Kumar et al. (2022). Surprisingly, even in the DomainNet scenario, the features on distribution shift data exhibit larger distortion than those on reference data, contradicting the feature distortion theory. This can be attributed to the notion that the assumptions made in the theory, including 1) the over- parameterized linear network assumption and 2) the rigorous fittof--dittibution assumption about the distribution shift data being orthogonal to the reference data's row space, might not align with real-world practical situations. For a more comprehensive understanding the results and further verification of LP-FT, refer readers Appendix B.
4.2 REGULARIZATION TOWARDS ZERO-SHOT FOR ROBUST FINE-TUNING
Differentiating pre-trained vision-language models from vision-only models, e.g., Caron al. (2021); Chen et al. (2021); Bao et al. (2022); He et al. (2022), is their ability in zero-shot classi- fication utilizing the language model. Notably, the zero-shot model without any training already demonstrates superior performance in handling shifts in data distribution, as illustrated earlier in Figure 2. Consequently, the primary task in achieving robust fine-tuning of vision-language models maintaining their zero-shot performance distribution shift data (Wortsman al., 2022b).
In this section, we delve into comparative analysis between the existing techniques for achieving robust fine-tuning and the conventional standard fine-tuning approach, denoted FT. To this end,
4

---

Published as a conference paper at ICLR 2024
DomainNet-RR
DomainNet- (P, C,),S)
1.5
1.5
Distortion
+8.5%
+11.4% 4%
+8.5%
+6.7%
1.0
+4.1%
1.0
+6.6%
+3.2%
+6.1%
0.5
0.5
0.0
0.0
1e-06
3e-06
1e-05
3e-05
1000
3000
5000
10000
Figure 3: Bar plots depicting the feature distortion on DomainNet. A taller bar represents larger degree of feature distortion after fine-tuning. The number on the bar denotes the relative difference in distortion values between reference and distribution shift data. Left: The number of training steps is kept at 5000, while the learning rate varies as 1 le-06, 3e-06, 1e-05, and 15,05, while the number of training steps is
3e-05. Right: The learning rate is constant altered to 1 1000, 3000, 5000, and 10000. These plots are with B 6 on DomainNet, and refer to Figures and for the results with B/ 32, B/16 6, and L/ on DomainNet and ImageNet.
explore the following three groups of existing algorithms that can as methods for achieving robust fine-tuning of zero-shot models: (1) EMA (Exponential Moving Average) and L2SP (Xuhong al., 2018) apply regularization steer the fine-tuned solution towards the zero-shot model in the weight space throughout the fine-tuning process: (2) KD (Hinton et al., 2015) and CAR-FT (Mao al., 2023) utilize regularization guide the fine-tuned solution towards the zero-shot model in the output space throughout the fine-tuning procedure; (3) WiSE (Wortsman et al., 2022b) and TPGM (Tian et al., 2023) are post-hoc approaches that obtain robust solutions by adjusting the fine- tuned model to be positioned close to the zero-shot model in the weight space. All experiments are with B/16 6 architecture, and we refer readers to Appendix B.2 for more details on these methods.
Interpretation via joint energy-based model. In the pursuit of developing novel robust fine-tuning methods, it becomes essential to consider the following questions: What exactly differentiate the existing robust fine-tuning approaches from standard fine-tunin fine-tuning, and what particular outcomes realized by employing regularization towards the zero-shot model? To explore this further, we per- ceive the zero-shot and fine-tuner models through the lens of a joint energy- based model (Grath- wohl et al., 2019), which reveals the inherent capacity for generative modeling within conventional discriminative models. To be specific, pre-trained vision-language models offer discriminative mod- eling for image data x and text data f,
Pe & (t/=)) = exp (
Er exp (fe (xx where So. (o, (Fo(z),
(4)
and an energy based model for the joint distribution of æ and t can be defined as
P (x. exp (o, exp (f) 2(0
(5)
④))
where the energy function is given Fo. (xr, - fa (xx,t) and 7 700, Φb) is unknown normal- izing constant. That is, the inner product value of fe. (xx, t) utilized as class logits in discriminative modeling (as described in Eq. 4) be reused to define the energy function (xx,). t).
Originally, the vision model Fa, before fine-tuning is strongly associated with the language model Ge through vision-language contrastive pre-training to the extent that enables zero-shot discrim- inative modeling. However, after fine-tuning, the representations of the vision model undergo dis- tortion, resulting in a weakening of the connection between the vision and language models. More precisely, we confirm this through an increase in the energy gap, defined as follows:
EnergyGap(e. 0, 0o) Eg (Ee (x. E t))²
(6)
Here, we compute the expectation involving by employing set of 10,000 text tokens, each having a sequence length of 8. These tokens are generated randomly from the vocabulary the pre-trained language model. We hypothesize that the existing robust fine-tuning techniques alleviate the decline in pre-trained associations between vision and language models by indirectly reducing the energy gap during their regularization procedure towards the zero-shot model.
5

---

Published as a conference paper at ICLR 2024
FT
EMA TPGM
L2SP Lipsum-F FT
CAR-FT Lipsum-FT
WiSE
Lipsum-FT WiSE
TPGM
6.004
PCC=-0. 796
PCC=- 764
PCC=- 0. 799
ECC=-0.860
EnergyCap
0.002
0.000
14
0.
0.74
0. DomainNet-CC
0.76
0.48 0.50
0.52
0.58
0.60 0.62
DomainNet-
DomainNet-I
DomainNet-5
Figure 4: Energy gaps and distribution shift accuracy on DomainNet. It represents the energy gap (y-axis) and the relative accuracy of distribution shift data the reference accuracy (x-axis). The model attained through the fine-tuning procedure is represented by square markers, while the model obtained through post-hoc approaches combining the fine-tuned and zero-shot models is denoted by diamond-shaped markers (i.e., Wi SE and TPGM). The dashed line depicts linear trend, and it comes with specific details provided the Pearson correlation coefficient, denoted as PCC. We refer readers to Figure 13 for the results on ImageNet.
Figure 4 provides clear evidence for our hypothesis regarding the energy gap minimization aspect in a range of robust fine-tuning approaches. We draw scatter plots illustrating the relative accu- racy for distribution shifts (i.e., the distribution shift accuracy divided by the reference accuracy) in connection with the energy gap. The Pearson Correlation Coefficient (PCC) values in each subplot further clarify a distinct correlation between the energy gap and the robustness to distribution shifts. Notably, our method, labeled Lipsum-FT and elaborated upon in 5, effectively leverages this connection and achieves superior performance in addressing distribution shift data.
LIPSUM-FT: FINE-TUNING WITH RANDOM TEXT GUIDANCE
In this section, we propose novel method for robust fine-tuning called Lipsum-FT, tailored to hance the robustness of vision-language models after fine-tuning. Briefly speaking, the core concept of our approach is to minimize the change in energy during the fine-tuning process in the context of discriminative modeling with the language model.
5.1 LIPSUM-FT
Alongside the cross-entropy loss, which is main objective of the fine-tuning procedure, our pro- posed I.i psum-FT method also minimizes the following regularization term,
R(())
(.)) vo, (<:))
2M
where (x) is computed using the language model G and M text tokens t,,...,tM
(m)) 0. (x) (G (tm) Fo(x)) for = I.
(8)
The name Lipsum FT comes from the process of generating text tokens t,,..., ta in a random manner using the vocabulary of the pre-trained lingunelanguage model. worth mentioning that the minimization of the regularization term described in Equation is stochastic way to minimize the energy gap defined before in Equation 6, which involves M samples for every fine-tuning iter- ation. Here, we have defined our regularization term in the logit matching (Ba & Caruana, 2014) format to facilitate a comparison with the closely related CAR-FT. More detailed discussions re- garding our proposed Lipsum-FT regularization will be provided in the forthcoming $ 5.3.
5.2 EVALUATION RESULTS
To begin, we present the evaluation results clearly showing the efficacy our approach compared to the baseline methods. For more comprehensive view of the results, please Appendix B.3.

---

Published as a conference paper at ICLR 2024
Table 1: Main results on distribution shift tasks. It summarizes the accuracy of various methods for B/1 16 6 in distribution shift scenarios on DomainNet and ImageNet. An underline highlights the top two values in each column, where all values are averaged from five measurements. See Table for the results with B/ 323 and L/14.
DomainNet
ImageNet
Method
V2
R
S
Zero-shot
82.5
66.7
65.6
43.8
56.9
68.2
62.0
76.3
52.1
46.7
PT
88.5.6.1
62.5.a2
64.3
41.2-0s
50.6
82.8-6.1
72.6.0.3
68.5
39.2.03
48.0402
EMA
88. 94.0
65. 55.s
67.3
44.4:06
54,8a08
83.0ano
72.6.0.
70.2
39.6,03
49,4
L2SP
88 6a0.1
63. 2e0.2
65.0e02
42.0pn2
51 51.4
82.9
72.6,0.2
68.8
39.7eu2
48, 48.2an.1
6
64.2.02
65.7403
42.9.02
52.4.04
83.1-0.1
73.140.3
72.9km
42.3+04
49.9.02
CAR-FT
88. 94.00
64.4
65 85.8
43.3
53.2ans
83.2-00
73.0ea.
71.3
43.7su2
49.5.01
Lipsum-FT
89.0so.0
66.3
68.0eam
46.0s02
56.2an2
83.3.00
73.6ea.1
75.9
49.9tm3
51.4
Table Uncertainty quantification results on distribution shift tasks. summarizes the un-
certainty metrics, including (a) expected calibration error and (b) negative log- bog-likelhhood, of various methods for B/16 in distribution shift scenarios on DomainNet and ImageNet. All fine-tuning re- sults are averaged from five measurements, and the corresponding standard deviation values provided. An underline highlights the top two values in each column.
(a) Expected calibration error (lower better).
DomainNet
ImageNet
Method
R
P
IN
V2
Zero-sbot
0.025
0.020
0.028
0.107
0.021
0,019
0.023
0.039
0.076
0.045
FT
0.024
0. 0.13300001
0.122a0m4
0.204e000l
0.184
0.034a0001
0.075
0.066
0.234an00c
0.159
EMA
0.018
0, 0.104400066
0.101
0.19622006
a.151
0,027
0.0682a00s
0.050
0.220an:04
0.142ac.004
L28P
0.024m000
123
0.124a0
0.204g2006
0,18330004
0.033=000
0.074 0.063
0.228eno0z 0,158200002
RD
0,006
0.078a000s
0.073
0.170ea006
0.123
0,007a000
0.029
0.009a00001
0.169an004
0.083a0.002
CAR-FT
0.018.am:
0.1 10
a.107
0.2021000
0.156m0000
0.026
0.066u000
0.045
0.197mno0
0.1400000
1.4 psum-FT
0.007--000:
0.055
0.043-4mm
0. 1464a001
40.0
0. 0.034-aco 0.008-0mm
2.01240007
0. 0.122-000
0.088-c.0m
(b) Negative log- log-likelihood (lower is better).
DomainNet
ImageNot
Method
P
TN
V2
Zero shot
0.731
1.497
1.527
3.071
2.047
1.184
1.490
0.927
1.925
2.245
0,445:000
1.842s0010
1.748.002
3.64120088
2.818a0041
0.624
1.129000
1.439a0ms5
2.765an017
2.550a0.00
EMA
0.425.nmm
1.67310040
1,54
3.435rn0w
2.469.0.88
0.608 1003
1.089
1.330
2.660:n008
2.399
L25P
0.447-m002
1.846x00th
1,736=0013
3.6354n0m
2.806
0.621
1.126a00r
1.42940m
2.727-0012
2.546+oms
KD
0.435.nmm
1 1.68300m
1.575m0m
3.427inoss
2.511
0.601
L.47-n0m
1.137
2.407:00w
2.257
CAR-FT
0.423-nom:
1.732s00ts
1.624.0033
3.5084m004
2.579
0.599
1.087:0c05
1.261
2.424.0014
4.38
Lipsun FT
0.425=too:
1.521
m,400emms
3.079en011
2.150m0.11
0. 0,595
1.010eoos
0.973e0.04
1.986en200
2.12300004
Evaluation results for classification accuracy. Table 1 presents comparative results in the Do- mainNet and ImageNet distribution shift scenarios for B/16. 6. It clearly shows that Lipsum- FT surpasses all other fine-tuning baselines across all types of shifts. Table 5 in Appendix B.3 pro- vides additional confirmation that this excellence extends beyond the B/16 6 architecture and also observed in the case of the 3/32 and L/14 architectures.
Evaluation results for predictive uncertainty. While classification accuracy serves as the primary metric for assessing image classification models, it does not offer insights into the model's abil- ity in uncertainty quantification when processing distribution shift data. Considering the tendency of classification models to make incorrect predictions when faced with distribution shift data (as demonstrated by lower distribution shift accuracy compared to reference accuracy), evaluating the model's capacity to quantify predictive uncertainty becomes notable concern.
To account for this, we also present the evaluation results using the following two prominent un- certainty metrics (Guo et al., 2017): (a) Expected calibration error (Pakdaman Nacini et al., 2015), which quantifies calibration by calculating the overall disparity between prediction accuracy and confidence. (b) Negative logllike,,,ood, which directly assesses the likelihood of predictive cate-
7

---

Published as a conference paper at ICLR 2024
WiSE
TPGM-C
Lipsun-FT
Lipsum-FT
WiSE
Lipsun-FT
TPGM-C
89.0
89.0
89.0
89.0
Doma inNet-RR
88.5
88.5
88.5
88.5
88.0
88.0
88.0
88.0
65
68
65
68
42
45
52
55
58
DomainNet-P
Domain C
DomainNet-11
DomainNet-S
Figure 5: Scatter plots for post-hoc methods on DomainNet. Moving in an upward and right- ward direction signifies improved accuracy for the reference and distribution shift data, respectively, making the top right corner more desirable. We refer readers to Figure 14 for ImageNet results, as well as Tables and 7 for numerical results.
Soup
Lipsum- FT
Soup
Soup
Lipsum
Soup
Ensemble
Lipsum- Ensemble
Ensemble
Lipsum- Ensemble
90
86
DomainNet
89
Imagellet
88
V
86
78
48
50
52
54
56
58
60
50
52
54
56
60
62
DomainNet- (P, S,
ImageNet A,
Figure Scatter plots for ensemble and soup methods. Moving an upward and rightward di- rection signifies improved accuracy for the reference and distribution shift data, respectively, making the top right corner more desirable. Small markers depict set of fine-tuned models with varying hyperparameter configurations. Within this collection, Soup ingredients are identified by triangu- lar marker, while Ensemble members are distinguished by an inverted triangular marker.
gorical distribution. Table 2 shows that our suggested L Li Lipsum-FT approach attains lower numbers in these metries, indicating its superiority in uncertainty quantification.
5.3 DISCUSSIONS
Combining with post-hoc methods. Given that the Wi SE and TPGM techniques belong to the class of post-hoc methods, which involves merging zero-shot and fine-tuned models, they have the potential to work in conjunction with our proposed Lipsum-F' FT. Accordingly, we further present the results for comparison and combination of Lipsum-FF with WiSE and TPGM-C in Figure 5; TPGM-C is a variant of GPGM introduced by Tian et al. (2023) for a fair comparison with WiSE. Please refer Appendix B.2 for more details on these methods.
Figure 5 clearly shows that Lipsum-FT is positioned in favorable region at the upper right, demonstrating its superiority in both the reference data (upper) and distribution shift data (right). Furthermore, the plots labeled as Lipsum-F FT Wi SE and Li Lipaum-FT + TPGM-C illustrate that the existing post-hoe methodologies SE and TPGM can operate independently to improve the distribution shift performance in conjunction with newly introduced method, Lipsum-FI.
Combining with ensemble and soup methods. Another strategy acknowledged for its ability to withstand distribution shifts is the concept of model soup (Wortsman et al., 2022a), where the weights of multiple models fine-tuned with different hyperparameter configurations are averaged. In this context, also present the results comparing and combining Lipsum-FT with Soup and Ens ensemble in Figure 6. Soup and Ensemble represent weight-space and output-space ensem- bles, respectively, using the following greedy procedure descrihed in Wortsman et al. (2022a): (1)
8

---

Published as a conference paper at ICLR 2024
Table Ablation results on loss function and text guidance. It summarizes the accuracy of methods located within the range from CAR-FT to Lipsum-FT. The values enclosed in brackets indicate the difference in performance when compared to CAR-FT, where all values are averaged from five measurements. An underline highlights the top two values in each column. Please refer to Table 9 for ImageNet results, including standard deviations.
Loss function
Text guidance
DomainNet
Method
KLD + MSR fixed random
R
P
C
S
CAR-FT
88,9
64.4
65.8
43.3
53.2
CAR-FTHsE
88,9
65.7 (+1.)
67.1 (+13)
44.5 (+1.2)
55.2 (+2.0)
Lipsum-FTxzn
89.0 (+0.1)
66.0 (+1.6)
67.7 (+1.)
45.8 (42.5)
55.7 (+2.5)
Lipsum-FT
89.0 (+0.1)
66.3 (+1.9)
68.0 (+2.2) 46.0 (+2.7)
56.2 (+3.0)
arranging the fine-tuned solutions in order of their validation accuracy, and (2) gradually expanding the pool of candidates for ensembling in a greedy manner to improve the validation accuracy.
Figure 6 illustrates that when our approach is combined with the model soup method, denoted as Lipsum-BE FT + Soup, it results improvements both reference and distribution shift perfor- mance, positioning it favorably in the upper right region. It is worth emphasizing that the soup models outperform the ensemble models, which come with an extra inference cost (approximately four times more, constrained the maximum ensemble size four). These results align with the observations reported by Wortsman et al. (2022a) and demonstrate that our Li pssmpFFF FT approach functions independently from the soup method.
Further comparison with Mao et al. (2023). In a related study conducted by Mao et al. (2023), which introduced the CAR-FT method, they empirically investigated the impact of fine-tuning on the context awareness of pre-trained CLIP features. Their findings align with our central idea that fine-tuning disrupts the pre-trained connection between vision and language models. However, our research extends beyond context awareness, encompassing broader spectrum of general relation- ships for all potential discriminative models derived from pre-trained vision- laiiguge models. We enhance our understanding of the critical factors behind LipsummFFi FFT' success by conducting ad- ditional ablation study involving a comparative analysis between CAR-FT and I.i L.psum-FT.
More precisely, the points mentioned above lead to the following methodological distinctions be- tween CAR-FT and our proposed Lipsum-Fri FT: (1) CAR-F2 FT utilizes the Kullback-Leibler Diver- gence (KLD) loss for handling categorical outputs in the context of discriminative model for con- text classification. In contrast, Lipsum-FT adopts the Mean Squared Error (MSE) loss for energies (i.e.. logits) in the context of arbitrary discriminative model. (2) CAR-FT employs a fixed text guidance to explicitly address context awareness, whereas L.i L.ppsu-FT relies on random text guidance strategy covering all potential aspects inherent in pre-trained vision-language models. Ta- ble 3 conducts an ablation analysis on these two factors, illustrating the roles played by both the MSE loss and the random text guidance strategy in enhancing performance. For more detailed implementation and connection between MSE and KLD losses, refer to Appendix B.3.
6 CONCLUSION
In this study, we introduced Li Lipsum-FT, straightforward yet powerful method for enhancing the distribution shift robustness of fine-tuned lision-language models. Lipsum-F's goes beyond the conventional notion that fine-tuning distorts the features learned during pre-training; it advances the concept in the context of pre-trained vision-language models that fine-tuning can disrupt the pre- trained connections between vision and language models. From this, Lipsum-FT minimizes the energy gap across all potential discriminative models derived from the pre-trained vision-language model. The extensive experimental results strongly support these claims, and Lipsum-Fi FT has also demonstrated its capability to enhance other methods using weight averaging strategies. This research represents pioneering efforts in involving the language model aspect into the fine-tuning procedure for vision-language models. An intriguing avenue for future research is investigating a novel text guidance strategy tailored for robust fine-tuning instead of random text guidance.

---

Published as a conference paper at ICLR 2024
Ethics statement. In this work, utilized CLIP, notable contemporary example of large-scale pre-trained models. Such large models potentially raise ethical concerns due to unfiltered data during their pre- pre-training phase (Weidinger et al., 2021). Nevertheless, it is worth clarifying that this work fundamentally an analytical paper and does not inherently carry significant ethical risks.
Reproducibility statement. Appendix C provides specific information about the experiments, and we plan to release the experiment code accessible to the public to facilitate reproducibility.
ACKNOWLEDGEMENT
This work was partly supported by Institute of Information communications Technology Promo- tion(IITP) grant funded by the Korea goverimen (TTMM//TINNNNONO-NOM-.. 2019-0-00075, Artificial Intelligence Graduate School Program(KAIST) (AISTT) the National Research Foundation of Korea(NRF) grant funded by the Korea (overnment(MSIT) (NRF-2022RI
and I KAIST-NAVER Hypercreative Al Center. This material based upon work supported by the Google Cloud Research Credits program with the award GCP19980904 and Cloud TPUs from Google's TPU Research Cloud (TRC).
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz. Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015, URL https: /ww. wensorrrrffrr www.oorflwwoorrr.oooooo 33
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Informa- fion Processing Systems (NIPS), 2014. 6, 27
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacei, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzie, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojeiech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL ht thub com/ deepmind. 33
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transform- ers. In International Conference on Learning Representations (ICLR), 2022. 4, 15
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015. 15
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 15
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems (NIPS),
2011. 15
James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. cwm wwtgg//ogggeeeooooo jax. 33
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 15
10

---

Published as a conference paper at ICLR 2024
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 4, 15
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, 2020a. 15
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020b. 15
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020c. 4, 15
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision
(ICCV), 2021. 4, 15
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations (ICLR), 2021. 3
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 15
Henry Gouk, Timothy Hospedales, and massimiliano pontil. Distance- regularisation of deep networks for fine-tuning. In International Conference on Learning Representations (ICLR), 2021. 15
Will Grathwohl,
Wang, foern-Hentte Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations (ICLR), 2019. 2, 15
Baun-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 15
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020. 15
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022. 4, 15
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and duitr--disttiuutton examples in neural networks. In International Conference Learning Representations (ICLR), 2017. 15
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021a. 33
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021b. 33
11

---

Published as a conference paper at ICLR 2024
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025:: 2015. 5, 24, 27
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference Machine
Learning (ICML), 2021. 2, 15
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 15
Taehyeon Kim, Jaehoon Oh, Nak Yil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021. 27
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015, 16
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform duttoo--iittibuutinn. In International Con- ference on Learning Representations (ICLR), 2022. 1, 3, 16
Yann LeCun, Sumit Chopra, and Raia Hadsell. A tutorial on energy-based learning, 2006. 15
Hao Li, Pratik Chaudhari. Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations (ICLR), 2020. 15
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. DELTA: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations (ICLR), 2019. 15
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- autoof-distribution detec- tion. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 15
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 15
Xiaofeng Mao, Yuefeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, 2023. 1, 3, 5, 9, 15, 24, 33
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 15
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on Machine Learning (ICML), 2013. 15
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014. 15
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference Artificial Intelligence, 29(1), Feb. 2015.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV) 2019. 33
12

---

Published as a conference paper at ICLR 2024
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputiigg pp. 126658, 2023. 2,
Joaquin Quinonero-Candela, Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008. 15
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. 1, 2, 3,4, 15,
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classi- fiers generalize to imagenet? In Proceedings of the 36th International Conference on Machine Learning. 2019. 15, 33
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pp. 400-407, 1951. 16
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3), 2015. 33
Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 35(7):1757-1772, 2013. 15
Manli Shu, Weili Nie. De-An Huang. Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances Neural Information Processing Systems (NeurIPS), 2022. 16
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 15
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 15, 33
Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 123-)):135-11260 2003. 15
Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng I Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 3. 5, 8, 15, 25, 33
Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011. 15
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
Puja Trivedi, Danai Koutra, and Jayaraman Thiagarajan. A closer look model adaptation using feature distortion and simplicity bias. In International Conference on Learning Representations (ICLR), 2023.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):2579-266 2008. URL http: / jmlr. www. http//ppprs///////
vandermeaten html. 29
13

---

Published as a conference paper at ICLR 2024
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 33
Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3032-30444 3042, 2023. 31
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprini urXiv:21/2.04335, 2021. 10
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https: /ww..w................./....ww c web. org/ anthool ogy/ 2020. emn mmmnp-demos... 6. 33
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups:
averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time. In Proceedings of the 39th International Conference Machine Learning (ICML), 2022a. 8, 9, 25, 33
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022b. 1,3,4, 4, 5, 15, 24,33 33
Yongqin Xian, Tobias Lorenz, Bernt Schicle, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pp. 5542-5551, 2018, 31
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In Proceedings of the 33rd International Conference on Machine Learning (ICML). 2016. 15
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedingy of the 35th International Conference Machine Learning (ICML), 2018. 5, 15,
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep
neural networks? In Advances in Neural Information Processing Systems (NIPS), 2014. 15
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), 2014. 15
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: A baseline for network adaptation via additive side networks. In Proceedings of the European Conference Computer Vision (ECCV), 2020. 15
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2022a. 15, 16
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 13309)23377-2488 2022b. 15, 16, 31
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORK
Transfer learning. The notion that training convolutional deep neural networks on extensive up- stream data leads to obtaining generalizable feature representations (Zeiler & Fergus, 2014; Yosinski et al., 2014; Oquab et al., 2014) has established the foundation for the
brraining-and-fine regime in computer vision. Numerous works have proposed different pre-training strategies for vi- sion models (Sun et al., 2017: Mahajan et al., 2018; He et al., 2020; Chen et al., 2020a;e;b; Grill et al., 2020: Caron et al., 2020; 2021: Chen et al.. 2021; Bao et al., 2022; He et al., 2022), enabling effective adaptation of pre-trained knowledge to downstream vision tasks. This adaptation process typically involves fine-tuning, and various methods for regularizing towards the initial pre-trained model have also emerged to preserve the pre-trained knowledge during the fine-tuning process (Xuhong et al., 2018; Li et al., 2019; 2020; Zhang et al., 2020; Gouk et al., 2021). The advent of pre-trained vision- language models capable of zero-shot classification (Radford et al., 2021; Jia et al., 2021; Pham et al., 2023) has triggered the development of this regularization concept towards transferring the distribution shift robustness from the zero-shot model to the fine-tuned model (Wortsman et al., 2022b; Mao et al., 2023; Tian et al., 2023).
Energy-based model. An energy- based model belongs to the category of generative models, which gain an understanding of an underlying data distribution by examining a sample dataset (Teh et al., 2003). One notable characteristic of energy- based models is their ability assess the likelihood that a given set of data originates from a specific data distribution (LeCun et al., 2006). The seminal work of Xie al. (2016) demonstrated that there is mutual derivation between the discriminative model and the energy-based model. Following this, Grathwohl et al. (2019) suggested reinterpretation of a conventional discriminative classifier as an mergy-based model for the joint distribution.
Handling out-of- -if-----istiitttinn data. Given the inability to handle every potential instance during the training phase, mainly due the dataset shift (Quinonero-Candel et al., 2008), accounting for unknowns in testing becomes a notable issue in machine learning (Scheirer et al., 2013; Bendale & Boult, 2015; 2016). Within this context, the following problems have emerged as a significant research topic in machine learning:
Out-of--istribution detection problem aims to answer the following (Hendrycks & Gim- pel, 2017): can we identify whether test sample originates from a distribution distinct from the one employed during the training process? As an answer, one way to quantify the probability that given data came from specific data distribution is through the notion of energy (LeCun et al., 2006), where increased energy levels suggest that the data points exhibit lower probability density (Grathwohl et al., 2019; Liu et al., 2020). Notably, re- cent work by Ming et al. (2022) demonstrated the effective use of CLIP-like models for zero-shot -iitt--distiibution detection, capitalizing on the rich pre-trained feature repre- sentations obtained through a multi-modal contrastive loss.
Out-of-distribution generalization problem explores whether a model can effectively han- dle test sample that shows specific variations from the training dataset while still being part of a comparable overall distribution. From the perspective of domain shift, assuming underlying latent representation of the covariate space that is unchanged (Quinonero- Candela et al., 2008), there are closely related research topics such as domain adaptation and domain generalization. However, the outtof-distribution generalization encompasses broader scope than domain adaptation, which assumes access to the test data distribution, or domain generalization, which involves utilizing data from diverse domains (Muandet et al., 2013; Blanchard et al., 2011). Especially in computer vision, many studies have approached the -otoo-diistrbution generalization problem by focusing on the distribution shift robustness of image classification models; models trained on reference data should provide reliable predictions even when dealing with shifted data (Torralba & Efros, 201 I; Recht et al., 2019; Taori et al., 2020).
Prompt learning. The seminal work of Radford et al. (2021) has already demonstrated the signifi- cance of crafting effective prompts for enhancing zero-shot classification with CLIP. Utilizing suit- able prompt text through prompt engineering improves zero-shot classification performance com- pared to the baseline of using context-free class text. Consequently, a branch of prompt learning research has arisen, focusing on adapting pre-traincd vision- vasion-language models to downstream vision tasks without fine-tuning vision model parameters (Zhou et al., 2022b;a; Du et al., 2022; Jia al.,
15

---

Published as a conference paper at ICLR 2024
2022: Shu et al., 2022). Empirical findings from prior research have validated that prompt learning methods exhibit competitive performance in out-of-distribution generalization, including the Ima- geNet scenario discussed in this paper (Zhou et al., 2022b;a; Shu et al., 2022). However, their reference performance is considerably lower than the fine-tuning techniques we considered, with the highest accuracy in Shu et al. (2022) reaching 73.6, while FT baseline achieves 82.8.
B SUPPLEMENTARY MATERIALS
B.1 STANDARD FINE-TUNING
Details on standard fine-tuning procedure, Starting from an initial parameters Oo = =(0, Wo), a standard fine-tuning procedure obtains the fine-tuned model parameters 0r =(r,, Wrr) after T iterations. Specifically, the update rule for the (th iteration is
θ - - Ve.c (0)lo-o0) for t I,,..,,
where 1) learning rate and (Cr(O) is a cross-entropy loss computed for given mini-batch. Although we practically employed the Adam optimizer (Kingma & Ba, 2015) and a cosine decay learning rate scheduler with linear warm-up. we present here the basic form for stochastic gradient methods (Robbins & Monro, 1951) for the sake of clarity. Unless specified, the mini-batch size was set to 2566, the peak learning rate was configured as 1 e-05 with the initial 10% of steps were ded- icated to the warm-up phase. We also evaluate fine-tuned models every 000 iterations and choose the model that demonstrates the highest performance on the reference validation data, which corre- sponds to Doma inNet R for the DomainNet scenario and ImageNet for the ImageNet scenario.
Head initialization matters. The feature distortion theory of Kumar et al. (2022) underscores how the fine-tuning procedure is influenced by the initial weights of the head. Consequently, we explore three distinct strategies for initializing the head during the standard fine-tuming of CLIP-VI ViT models: (1) Scratch-Fi fine-tuning with head weights initialized to zero; (2) LP-FT: fine-tuning with head weights; (3) FT: fine-tuning with zero-shot head weights. We obtain the zero- shot head weights by following the procedure described in the official code base.
Table 4 summarizes the baseline results for standard fine-tuning with three different head initializa- tions. Across the board, FT consistently exhibits superior performance distribution shifts com- pared to Scrat ch ch-FT and LP-F7 on average. It highlights the considerable influence exerted by the distinctive aspect of zero-shot head initialization involving the language module the zero-shot vision- lssiun-language model fine-tuning scenario.
Supplementary plots for Figure 2. In Figure 7, we present the same plot for the ImageNet scenario. As discussed in 1, the performance on distribution shifts deteriorates as we engage in further fine- tuning via increased learning rates or additional training steps. We also provide detailed results for each distribution shift in Figure 8.
Supplementary plots for Figure In Figures and 10, we present outcomes from parallel experi- ments for 3/32, B/1 6, and T./14 across both DomainNet and ImageNet scenarios. As discussed in 4. there is no significant difference in the extent of the feature distortion between reference and distribution shift data, and even in the case of DomainNet, the feature distortion for distribution shift data is larger than that for reference data, Figure also provides detailed results for each distribution shift (bottom plots in each subfigure).
Further examination on the feature distortion theory. Continuing from $ 4.1, we delve into an empirical exploration to ascertain whether the feature distortion theory can elucidate the scenario of CLIP-ViT fine-tuning. Across B/32, B/1 6, and L,/1 4 architectures on DomainNet and ImageNet datasets, Figures 11 and 12 demonstrate: (1) I.P-FT effectively reduces feature distortion, nonethe- less, (2) there exists no noteworthy discrepancy in the degree of distortion between reference and distribution shift data on average. Considering the fact that LP-FT's performance on distribution shifts is not superior to other methods, as evident in Table 4, we conclude that LP-FT does not achieve the goal of robust fine-tuning in the scenario of CLIP-ViT ViT fine-tuning.
httpp:://gi thugh wtw.o...oo/ooaaaaaaaaaaaaaaaaaaa /CI.IP
/CLIP
16

---

Published as a conference paper at ICLR 2024
Table 4: Baseline results for standard fine-tuning. Classification accuracy for both reference and distribution shift data. The number of training steps 5000 for DomainNet and 50000 for ImageNet, using a learning rate of 05. The highest value across fine-tuning methods is indicated with an underline. Moreover, we provide the results for zero-shot ("Zero-shot') and linear-probed models ("Lincar-probed for further comparisons.
(a) DomainNet.
Distribution shifts
Model
Method
I
AVG
Scratch-FT
85. 15.0av.1
55.820.2
57.2s0.3
32.1
43.9a0.2
47.3
LP-FT
84,4
50.5+0.3
51.4+0.3
27.2
36.540.3
41.4
B/32
86.5+n.1
58.9 28.9402
59.7+0.3
34.8+0.2
45.9+0.3
49.8
Linear-probed
83.2
51.4
50.7
27.2
38.6
42.0
Zero-shot
79.4
62.4
59.8
37.1
50.3
52.4
Scratch-25 FT LP-FT
87.7+0.0
60.8+0.2
62.720.3
39.3=0.2
49,2+0.5
53.0
86. 20.7
54.5+0.1
56.2+0.2
32.3+0.5
40.8
46.0
B/16
88.5.0.1
62.5.0.2
64.3.0.4
41,2a0.5
50.640.5
54.7
Linear- ehearr-probed Zero-shot
86.0
56.0
57.4
32.3
44.2
47.5
82.5
66.7
65.6
43.8
56.9
58.3
Scratch-FT 91.0a0.1
67.420.3
73.920.3
46.4
62.320.4
62.5
LP-FT 89.3+n.2
59.6+0.5
66.7+0.5
39.5+0,3
53.3+0.4
54.8
L/14
FT
91.2+n.1
68.1+0.2
74.3+0.4
48.6+0.6
63.5+0.5
63.6
Linear- Linear-probed Zero-shot
88.8
60.5
68.0
39.1
56.1
55.9
86.6
71.3
75.7
49.3
68.8
66.3
(b) ImageNet.
Distribution shifts
Model
Method
IN
V2
A
AVG
Scratch-
79.0s0.1
67.2+0.3
55, 1003
19.1
40.8+0.1
45.6
LP-FT
78. 78.9+0.1
66.5+0.1
54.4.0.2
19.0+0.2
39.5+0.3
44.9
B/32
FT
79.2.0.1
67.2.0.2
59.8.0.
20.2.0.3
42.240.2
47.4
Linear-probed
74.7
62.8
55.3
24.8
37.0
45.0
Zero-shot
63.2
56.4
68.4
32.4
41.2
49.6
Scratch-FT
83.1
73.1
64.220.3
37.820.
47.420.3
55.6
LP-FT 8
70.8+0.2
62.2+02
35.4+0.4
44.6+0.2
53.3
B/16
FT
82.8+n.1
72.6+0.3
68.5+0.3
39.2+0.3
48.0+0.2
57.1
Linear-probed Zero-shot
78.6
67.6
63.2
44.3
42.3
54.4
68.2
62.1
76.3
52.1
46.7
59.3
Scratch- FT LP-FT FT
86.2.0.1
77.0
76.920.6
60.8
57.9+0.3
68.2
84.5.0.0
73.9+0.1
73.8s0.2
55.8
54.5+0.2
64.5
L/14
85.4
75.5.0.1
80.4
60.4.0.1
57.940.1
68.6
Linear- Linear-probed Zero-shot
83.0
73.1
77.4
65.4
53.5
67.4
75.2
69.9
86.9
71.8
58.6
71.8
17

---

Published as a conference paper at ICLR 2024
B/32
B/16
L/14
B/32
B/16
L/14
90
90
ImageNet
ImageNett
80
70
70
60
45
50
55
60
65
70
45
59
55
68 65
70
ImageNet (V2, R,,
ImageNet- R,
Figure 7: Trade-off plots on ImageNet. The star-shaped markers correspond to the zero-shot model, while the square-shaped markers denote the model fine-tuned with 50000 training steps and learning rate of 1e-05. It supplements Figure Left: The number of training steps is kept at 50000, while the learning
-0,, 06, le 05, and 3e-05 (the leftmost point denotes 3e- 05), Right: The learning rate is constant at 1e-05 05, while the number of training steps altered 10000, 3C000, 50000, and 100000 (the leftmost point denotes 100000).
varies
18

---

Published as a conference paper at ICLR 2024
B/32
B/16
L/14
DomainNet-
90
85
80
75
60
70
60
70
30
40
50
40
60
DomainNet-P
DomainNet
DomainNet-1Z
DomainNet-S
(a) DomainNet results with varying learning rates of 1e-06, 3e-06, 1c-05, and 3c-05.
B/32
B/16
L/14
95
DomainNet-
90
85
80
60
60
70
49
50
50
70
Doma Comaineet-2 -P
DomainNet-C
DomainNet-II
DomainNet-S
(b) DomainNet results with varying training steps 1000, 3000, 5000, and 10000.
B/32
B/16
L/14
90
ImageNet
80
70
60
60
70
60
80
25
50
40
50
60
ImageNet-Ve -V2
ImageNet-R -R
ImageNet AA
ImageNet S
(c) ImageNet results with varying learning rates of 1c-06, 3e-06 0,006, 1e-05 05, and 3e-05.
B/32
B/16
L/14
ImageNet
70
60
60
88
20
50 ImageNet-S
60
ImageNet- v2
ImageNet
ImageNet-A
(d) ImageNet results with varying training steps of 10000, 30000, sccce, and 100000.
Figure 8: Trade-off plots in detail. This figure presents individual results for each distribution shift. See Figures and for results summarized single plot.
19

---

Published as a conference paper at ICLR 2024
DomainNet-
DomainNet (P, C,I,5)
1.5
1.5
+10.
+5.8%
+6. 1%
+6. 1%
Distortis
1.0
+2.9%
1.0
+5.0%
+2.3%
+5.3%
0.5
0.5
0.0
1e-06
3e-06
1e-05
3e-05
1890
3098
5008
18909
DomainNet-* R
DomainNet-PP
DomainNett-C
DomainNet-11
DomainNect-
Distortion
1.5
1.5
0.5
0.5
0.0
0.0
1e-05
3e-05
1000
3000
5000
10090
(a) Results for B/32 on DomainNet.
DomainNet-RR
DomainNet (P,C, 1,),
1.5
1.5
Distertion
+11.4%
+6.7%
+8.5%
+8.5%
1.0
+4.1%
1.0
+6.6%
+3.2%
+6.1%
0.5
0.5
0.0
0.0
1e-05
05
5000
10090
DomainNet-RR
DomainNet-PP
DomainNet--C
DomainNet
DomainNet-S
1.5
1.5
Distortion
1.0
1.0
0.5
0.5
0.0
0.0
1e-06
3e-06
1e-05
3e-05
1000
3000
5600
10000
(b) Results for B/16 DomainNet.
DomainNet-AR R
DomainNet (P,,,
1.5
1.5
Distortion
+8.5%
+5.4%
1.0
+4.5%
1.0
+5.5%
+4.5%
0.5
+3.8%
+2. 0%
0.5
+5.2%
0.0
0.0
le- 06
36-06
1e-05
3e- 05
1090
3000
5080
10008
DomainNet-At RR
DomainNet--P
DomainNett-G C
DomainNet I
DomainNet-S
1.5
1.5
Distortion
1.0
1.0
0.0
0.0
1e-06
3e-06
1e-05
3e-05
190G
3000
5000
10000
(c) Results for 1.1 on DomainNet.
Figure 9: Feature distortion DomainNet in detail. supplements Figure presented in the main text. The bottom plot for each subfigure presents individual results for each distribution shift.
20

---

Published as a conference paper at ICLR 2024
ImageNet
ImageNet R,,,
1.5
1.5
Distortion
1.0
-5.6%
-7.6%
5.7%
-0.9%
1.0
-2.9%
-5.3%
-5.7%
-3.7%
0.5
0.5
0.0
0.0
06
1c-05
05
30900
50900
100000
InageNet
ImageNet-V2
ImageNet-R
ImageNet
ImageNet-S S
1.5
1.5
Distortion
1.0
1.0
0.5
0.5
0.0
0.0
10000
(a) Results for B/32 on ImageNet.
ImageNet
ImageNet RV,,)eett-V R,2,
1.5
1.5
Distortion
1.0
15.
7%
+1.3%
1.0
-3.3%
-2.7%
-2.8%
-4.6%
-2.5%
0.5
0.5
0.0
0.0
1e-06
3e-06
1e-05
3e-05
10800
38000
50060
160000
ImageNet
ImageNet
ImageNet
ImageNet
ImageNet S
1.5
1.5
Distortion
1.0
1.0
0.5
0.5
0.0
0.0
1e- 06
3e- 06
1e-05
35-05
10000
30000
50000
100000
(b) Results for H/ 1 ImageNct.
ImageNet
ImageNet RV2,R,A,S)
1.5
1.5
Distartion
10.
-4.5%
5%
-5.9%
1.0
-8.3%
1.0
-5.7%
-1.4%
0.5
0.5
0.0
0.0
le- 86
36-06
1e-05
3e-05 05
18900
38000
58060
160000
ImageNet
ImageNet-V2
ImageNet-BR R
ImageNet
ImageNet
1.5
1.5
Distortion
1.0
1.0
0.5
0.5
0.0
0.0
10-
06
3e- 06
1c-05
3c- 95
10000
38000
50000
169008
(c) Results for L/14 ImageNet.
Figure 10: Feature distortion on ImageNet in detail. It supplements Figure presented in the main text. The bottom plot for each subfigure presents individual results for each distribution shift.
21

---

Published as a conference paper at ICLR 2024
DomainNes neet-R R
DomainNet (P,C,I,))
R
1.8
1.8
+3.4%
Dist Distor
1.2
+6.1%
1.2
+7.7% 7%
0.6
0.6
0.0
0.0
Scratch-FT
LP-FT -62. 7%
Scratch-FT
LP-FT
FT
-26.0%
(a) Results for 8/32.
DomainNet--N R
DomainNett (P,
R
S
pistor Tor 00
1.8
1.8
+2.6%
1.2
+8.5%
Distort
1.2
+9.9%
0.6
0.6
0.0
0.0
Scratch-FT
LP-FT -61.0%
Scratch-FT
LP-FT
FT
-27.8%
(b) Results for B/16.
DomainNet-
DomainNet- (PP,C
R
1.8
Distor rt
+2. 0%
1.2
+4.5%
1.2
+5.6%
0.6
0.6
0.0
Scratch-FT
LP-FT -53.
FT -31.4%
Scratch-FT
LP-FT
FT
(c) Results for L/14.
Figure 11: LP-FT within the context of feature distortion theory on DomainNet. The number on the bar denotes the relative difference distortion values between reference and distribution shift data, while the value under the x-labels indicates the relative difference in distortion from the Scratch-FT method. Left: Results summarized single plot. Right: A detailed results for each dsitribution shift.
22

---

Published as a conference paper at ICLR 2024
ImageNet
ImageNet MmageNeet-(V2,RAA.2 AV,
IN
V2
Dist sstorto OP
1.8
1.8
-0.4%
Distor
1.2
-5.7%
1.2
-1.9%
0.6
0.6
8.0
0.0
Scratch-FT
LP-FT -61.5%
FT -29.1%
Scratch-FT
LP-FT
FT
(a) Results for 8/32.
ImageNet
ImageNet AVVgeNlet((V2,RAA...
IN
V2
S
Distor ion
1.8
1.8
-1.0%
1.2
-2.7%
Distort
1.2
-2.4%
0.6
0.6
0.0
0.0
Scratch-FT
LP-FT -59.2%
Scratch-FT
LP-FT
FT
-27.6%
(b) Results for B/16.
ImageNet
ImageNet ImageNett(V22RRAAA AV,
V2
1.8
Distor 17
-9. 1%
1.2
-8.3%
-5.9%
1.2
0.6
0.6
0.0
Scratch-FT
LP-FT
FT -25.4%
Scratch-FT
LP-FT
FT
(c) Results for L/14.
Figure 12: LP-FT within the context of feature distortion theory ImageNet. The number on the bar denotes the relative difference distortion values between reference and distribution shift data, while the value under the x-labels indicates the relative difference in distortion from the Scratch-FT method. Left: Results summarized single plot. Right: A detailed results for each dsitribution shift.
23

---

Published as a conference paper at ICLR 2024
B.2 ROBUST FINE-TUNING
Details on robust fine-tuning methods. This paragraph provides concise overview of the robust fine-tuning techniques we have considered in this work. For more comprehensive details, kindly consult the corresponding references:
(1.2 regularization towards Starting Point; Xuhong et al., 2018) L.2P applies an 1.2 regu- larization towards O, i.c., the update rule becomes
+ - 1 A125P R12se (
(10)
where the regularization term is given by
1 ク/ 25P (
1 001 1
(11)
We swept over ÀLSSS E (3e-03, 03,0,, 3e-04, 1e-04) and set AL2sp 3e-04 both for DomainNet and ImageNet scenarios.
(Knowledge Distillation; Hinton et al., 2015) KD applies regularization towards O using the update rule
e = 0 -V Axo Rec (0))le-o...
(12)
where the regularization term is given by the distillation loss,
R2 2 Σ Softmax (k) (WoFe, log Softmax (4) (WFe(x)).
(13)
We swept over Agn E 10.1, 0. 0. 2,0, and set ARn 0.1 both for DomainNet and ImageNet scenarios.
(Context-Aware Robust Fine-Tuning, Mao et al., 2023) CAR-FT applies a regularization towards O, using the update rule
θ = 0 (e( Acae-Fr (.ak-r.())) (0))
(14)
where the regularization term is defined for the vision model parameters 0,
M Σ
R RI (0)
Softmax (m)) (W (We.xFo. (x) log Softmax (11) (W WoxxFo(x)) (15)
We swept over ACAR-PT é f0. 1, 0. 0, and set AEAR- = 1.0 both for DomainNet and ImageNet scenarios.
(Exponential Moving Average) EMA takes an exponential moving average over the opti- mization trajectory, i.e., for every 7th iteration,
LKA Auma
OxxA + -1-AAEN)) - -
(16)
using the decay factor AEva [0, 1], where we initialized the buffer 6RNa + Og- In- stead of the fine-tuned solution Or, EMA utilizes OEna for evaluation purposes. We swept over Arna E {0. 9999, ,,,9,,,,,,,, 0. 9997, 09977000 0. 9995) and set Arsz = 0. 9995 for DomainNet and Azza 0. 9999 for ImageNet.
(Weight-Space Ensembles for Fine- Fine-Tuning; Wortsman et al., 2022b) WiSE post-hoc simple weight averaging scheme that takes an weighted average between the zero-shot parameters 0, and the fine-tuned parameters 0r,
i Oasee = (1 >wise) 0 AgISE Or:
(17)
using the mixing coefficient Awise [0, 1]. Unless otherwise mentioned in the main text, we used AgisE 0.9 because it performs satisfactorily on the reference data, as demon- strated in Tables 6 and 7.
24

---

Published as a conference paper at ICLR 2024
FT WiSE
EMA TPGM
L2SP
CAR-FT Lipsum-FT TPGM
Lipsum-F
Lipsum-FT WiSE
600
PCC=-6 9.915
PCC=- 0.861
PCC=-0.877
EnergyCap
0.002
0.000
0.88
0.89
0.85
0.90 ImageNet R
0.50
0.60
0.58
0.60
0.62
ImageNet V22
ImageNet-
ImageNet-S
Figure 13: Energy gaps in robust fine-tuning baselines on ImageNet. It supplements Figure 4.
(Trainable Projected Gradient Method; Tian et al., 2023) TPGM is post-hoc optimization scheme that takes a layer-wise projection of the fine-tuned parameters to be within the straint | Ool & "Yege in the layer-wise manner. Let Or be the fine-tuned parameters and On be the zero-shot parameters for each layer, with slight abuse of notation. Then, TP TPCM defines the projected parameters as
OPGN = Og + max (1, 0r Ool2 /T)
(0r 0o),
(18)
and carries out the additional optimization using the validation data Dval.
Vieax = arg min
Σ
-log Softmax (k) (W Fo
(19)
Traax
(Dr,)JEDOS)
Following the implementation detailed by Tian et al. (2023), we carried out this optimiza- tion process using the Adam optimizer. The learning rate was set to 1e-02, and the num- ber of training steps was 200. Additionally, we experimented with TPGM-C, controlled variant of TPGM, where regularization over Yiras was introduced during the optimization.
Caveat: To streamline the experiments, we employed the test split of Doma iNet R as Dvai in the DomainNet scenario. Similarly, in the ImageNet setup, we used the validation split of ImageNet as Dva1. Thus, we note here that it is challenging to categorize the reference data performance of the TPGM presented in the main text purely test perfor- mance, raising point of contention. However, we would like to emphasize that there are no issues with the experimental comparison.
(Model Soups; Wortsman et al., 2022a) Soup and Ensemble refer to the greedy soup and greedy ensemble algorithms introduced in Wortsman et al. (2022a). These techniques op- erate a scenario where multiple models are created with varying hyperparameter setups. In contrast to the conventional approach of selecting the best-performing model from this set, Soup and Ensemble respectively attain improved performance by averaging their weights and ensembling their outputs.
In our experiment, we varied the following three hyperparameters: (1) mini-batch sizes {128, 160, 192, 224, 256 66), f. (2) initial learning rates 1,, 2e-06,.... 05-05), and (3) the number of training iterations {3000, 5000, 7000} for the DomainNet sce- nario and (30000, 50000, 70000} } for the ImageNet scenario.
Caveat: Similar to the situation with TPGM, it is hard to label the reference data perfor- mance of the Soup and Ensemble methods discussed in the main text solely test per- formance since the test data have been involved in the creation process of those methods. Also, it is worth mentioning that we have conducted additional hyperparameter searches as well. Again, however, we want to highlight that there are no concerns regarding the experimental comparison.
Supplementary plots for Figure 4. In Figure 13, present the same plot for the ImageNet sce- nario. As discussed in $ 4.2, it clearly depicts a distinct correlation between the energy gap and the robustness to distribution shifts.
25

---

Published as a conference paper at ICLR 2024
Table 5: Full results on distribution shift tasks. It summarizes the accuracy of various methods for (a) 3/32 and (b) L/14 14 models in distribution shift scenarios on DomainNet and ImageNet (for B/16, 6, please refer to Table presented in the main text). All fine-tuning results are averaged from five measurements, and the corresponding standard deviation values are provided. An underline highlights the top two values each column.
(a) B/32.
DomainNet
ImageNet
Method
R
I
S
IN
V2
R
8
Zero-shot
79.4
62.4
59.8
37.1
50.3
63.2
56.4
68.4
32,4
41.2
PT
86.5.o.1
58.9
59.7Ga3
34.8-02
45.9v03
79.2.6.1
67.2x0.2
59.8
20.2
42.2m02.
EMA
86.8
61.0aa2
62.1x02
37.3-02
48.9.03
79.1.0.1
67.4x0.2
60.8s04
19.9.03
42.7s04
86.4s0.1
58. 88.9.02
59.8
35.0an3
46.0an2
79.2.0.1
67.3a0.1
60.1
20.6.a3
42.3:03
KD
86.6
60.0kaa
60.9
36.0-03
47.3-0a
79.4-6.1
68.0
63.7G02
22.2:a2
43.5
CAR-FT
86.7.0.1
60.0ea.
60.4s04
36. 1602
47.1a03
79.5.00
67.8s0.2
62.4s0z
23.2
43.6s02
Lipsum-PT
86.6x0.1
61.9.0.1
62.1xa
38.4.n.1
49.6.n.
79.4.0.1
68.4.0.1
67.2x02
27.7:03
45.0saa
(b) L/14.
DomainNet
ImageNet
Method
R
I
IN
V2
R
S
Zero-shot
86.6
21.3
75.7
49.3
68.8
75.2
69.9
86.9
21.8
58.6
FT
91.2a0.1
1a02
74.3an4
48.6ans
63.5ans
85.4
75.5
80.4203
60.4anu
57.9
ΣMA
91.5a00
70.8
76.8
$1.2
67.3a07
86.2.0.1
27.1.0.3
83.6
60.1a03
L2SP
91.2a0.
68.9202
74.8e02
48.9203
63.92n.
85.8
76.3
81.9203
62.7ans
58.5
KD
91.3a0.1
70.0s0.
75.8a02
49.5
65.9un.4
86.0ao.1
77.0a0.1
85.2s0.2
65.5am4
60.0an2
CAR-FT
91.4.01
69.8 19.1
76.0xa3
50.2an.
16.
86.3-0.1
76.8.0.1
84.2
66.62n2
60.0an.1
Lipsum
91.3s00
71.3
77.2401
$1.9.22
68.2an2
77.7a0.2
87.0so.1
71.7a03
61.2s02
WiSE
TPGM-C
Lipsum-t
Lipsum-FF
WiSE
Lipsun-FT TPGM-C
83.5
83.5
83.5
83.5
Iinageeet
83.0
83.8
82.5
82.5
82.5
82.5
73
74
70
75
40
50
50
52
ImageNet-
ImageNet
ImageNet-A
ImageNet-
Figure 14: Scatter plots for post-hoc methods on ImageNet. It supplements Figure 5. See Table for numerical results with standard deviations.
B.3 LIPSUM-FT
Full table of results for Table 1. In Table 5, we performed additional comparative assessments of robust fine-tuning techniques using both 3/32 and L/14 architectures. For L/ 14, experiments were conducted using batch size of 1 28 due to hardware memory constraints. We also applied the same hyperparameters from R/1 to B B/32 and T./14 without any adjustments. Overall, our proposed Lipsum-FF approach demonstrates enhanced distribution shift performance while main- taining performance the reference data.
Supplementary plots and tables for Figure 5. In Figure 14, we present the same plots for the ImageNet scenario. Tables and further provides the numerical results averaged over five mea- surements along with its standard deviation values. Again, Lipsum-FT exhibits superior perfor- mance both on the reference and distribution shift data. We could further enhance the performance in distribution shifts by combining with the existing post-hoc methods Wi SE. and TPGM.
26

---

Published as a conference paper at ICLR 2024
Table 6: Results for post-hoe methods DomainNet. supplements Figure 5, An underline highlights the top two values in each column for each group.
Method
WiSE (AwisR 1.
88.6=0.1
63. 1
65. 45.0
41. 11.900.3
51.2+0.5
WiSE (AaisE = 9)
88.8=0.1
64.5s0.1
66.5
43 620,3
53.2
WiSE (Awise = -0
88.8
65.
67.6 07.6103
44, 4,0.
54.920.4
WiSE (AwssE 7)
88.6=0.
66.7+0.2
68.4
46. Os0.2
56.3+0.3
WiSE (AwisE 6)
88.3
67.4.0.2
68.9402
46. 46.62
57.2+0.3
WiSE (WISE 0.5)
87.8=0.1
67.9.0.2
69.0s0.2
47 1..00..
57.9+0.2
Lipsum-FT WiSE (Arisz = 1. 1.0)
89.0=0.0
66.3+0.2
68. 68.040.0
46. 46.020.2
56,2+0.2
Lipsum-FT + WiSE (Amisz = 0.99
88.8=0.0
66.8+0.
68.4+0.1
46.4=0.1
56.9+0.2
Lipsum-FT + WiSE (Awiss = =0.8)
88.5=0.0
67. 2
68.6+0.0
46. 46.7
57.5+0.
Lipsum-FT WiSE (Aris -0.7)
88.1
67.5+0
68.7+0.1
46.9+0.1
57.8+0.2
Lipsum-FT WiSE (Aiss = -0.6)
87.7=0.0
67.7+0.1
68.6+0.1
46.8+0.1
58.1+0.2
Lipsum-FT + WiSE (Ariss = =).5)
87.1
67. 17.8
68.410.1
46.720.1
58.310.2
TPGM
89.0=0.0
63.820.2
67.310.1
44.920.3
55.420.1
TPGM-C CPGM-C(xe
88.9=0.0
65.8u0.2
67.340.2
44.920.3
55,420.2
TPGM-C (reg=0.2)
88.9
65.920.2
67.3a0.2
44.920.3
55.420.1
TPGM-C (reg=0.5)
88.9
66.020.2
67.4102
45.1
55.520.1
TPGM-C (reg=1.0)
88.8=0.1
66. 1
67.6202
45 45.420.1
55.820.2
TPGM-C (reg=2.0
88.7
66.620.1
68. 18.1
46.2=0.1
56.4.0.1
TPGM-C TPGM-C(reg=5) 0)
18.1
67.440.2
68.640.1
46.9.0.1
57.4.0.1
Lipsum-FT + TPGM
89.0=n.1
66.420.2
68.1
46.2#0.2
56.6s0.2
Lipsum-FT TPGM-C (reg=0.1)
89.0=0.1
66.5+0.2
68.1
46.2+0.1
56.6.0.3
Lipsum-FT + TPGM-C (reg-0.2)
89.0.0.1
66.410.2
68. 1
46.220.1
56.6s0.2
Lipsum-FT + TPGM-C (reg-0.5)
89.0=a.0
66.510.2
68.1
46.220.1
56.620.2
Lipsum-FT TPGM-C (reg-1.0)
89.0.0.0
66.510.2
68.2
46.320.2
56.720.3
Lipsum-FT TPGM-C (reg=2.0)
88.820.1
66.7.0.2
68.3
46.630.2
56.9:0.2
Lipsum-FT TPGM-C (reg-5.0)
88.0.0.
67.310.1
68.5.0.2
46.920.1
57.6.0.1
Hyperparameters in Lipsum-FT. There are two hyperparameters Lipsum-FT: (1) length of randomly generated text tokens L and (2) the number of text tokens M to be generated for each fine-tuning iteration. Table 8 provides ablation results regarding these hyperparameters, indicating that both substantial increase and decrease in the length of the random text can result in decline performance, as illustrated cases where I, and I, 32. implies that the effectiveness of text guidance can fluctuate depending the input text and further motivates future works aimed at creating a novel input text for robust fine-tuning. We used L = 8 and M 80 in the main text.
Connection between KLD and MSE. For CAR-FT and Li Lipsum-FTee methods presented in Tables and 9, we utilized the following KLD loss formulation,
R
Softmax (m) (
(m) (vo.0/T),
(20)
m==1
where T is a temperature hyperparameter for smoothing categorical outputs (Hinton et al., 2015). Although kept this temperature fixed connecting the KLD loss and the MSE loss. More precisely, taking a limit + oo to the KLD loss RELo gives us the following (Kim et al., 2021):
throughout all experiments, it serves bridge
lim RJ ALD
1
1 v 12 2M
M
M
()) 0o.
Constant.
(21)
2M
8
The first term aligns with our regularization defined in Equation 7, which can be understood as logit matching (Ba Caruana, 2014). Here, the second term causes an increase in the values in vo.0 and hinders complete logit matching pointed out in Kim et al. (2021). In the context of our Lipsum-FT framework, the scale of Vg.0 contains the energy information associated with the pre-trained relationship between vision and language models. Consequently, MSE is better choice than KLD from this perspective, as validated by empirical results in Tables and 9; CAR-2 FT TMSE surpasses CAR-FT, and I. psum-FT outperforms L.i psum-FTxxx
27

---

Published as a conference paper at ICLR 2024
Table Results for post-hoe methods on ImageNet. supplements Figure 14. An underline highlights the top two values in each column for each group.
Method
TN
V2
$
WiSE (AwisR 1.
82.8=0.1
72.6x0.3
68.5
39.2a0.3
48.0s0.2
WiSE (AaisE = 9)
83.2=00
73.3x0.2
71.3
42. 62.63
49.9x0.2
WiSE (Awise = -0
83.1
73.520.2
73.5
45 62..444
51.2s0.1
WiSE (Awss 7)
82.8=0.1
73.4
75.2+0.2
48.
52.2s0.1
WiSE (AwisE 0)
82.1
72.940.1
76,6102
50.3
52.5e0.1
WiSE (WISE = 0.5)
80.9=0.0
72.1
77.6+0.1
51.820.4
52.5+0.1
Lipsum-FT WiSE (Arisz = 1.0)
83.3=0.0
73.6+0.1
75.940.1
49, 19.920.3
51.4.0.1
Lipsum-FT + WiSE (Arisz = 0..9)
83.0=0.0
73.660.
76.7+0.1
51.1
51.9+0.1
Lipsum-FT + WiSE (Auise = =0.8)
82.6=0.0
73.4
77.3+0.1
52.0+0.2
52.2+0.1
Lipsum-FT WiSE (Arss 7)
81.8=0.1
72.9+0.1
77.8+0.1
52.8+0.1
52.2+0.1
Lipsum-FT + WiSE (Arss = 6)
80.9-0.1
72. 1
78.1
53.4-0.2
52. 10.1
Lipsum-FT + WiSE (Awiss = =).5)
79.620.1
71.1
78.2a0.1
53.7.0.2
51.820.1
TPGM
83.0=a.1
73.610.1
74.1102
47 47.820.6
51.720.2
TPGM-C CPGM-C(xee=0..2
82.9
73.6u0.1
75.0202
49.5.0.5
52.020.1
TPGM-C (reg=0.2) .2)
82.5.0.1
73.220.1
75.61022
50.420.4
52.220.2
TPGM-C (reg=0.5)
81.3g0.1
72.320.1
76.7202
52.0a0.1
52.1
TPGM-C (reg=1.0)
78,8=0.1
70.4
77.6.0.1
53.020.3
51.420.1
TPGM-C (reg=2.0 00)
75.2
67. 27.7
77.940.1
53.2a0.2
50.1
TPGM-C TPGM-C(reg=5) (reg=5.0 0)
71.3a0.0
64.6s0.1
77.3
52.8.0.1
48.3+0.0
Lipsum-FT + TPGM
82.8=n.0
73.5
76.940.1
51.420.3
52.3
Lipsum-FT TPGM-C (Ieg=0.1)
82.2=0.1
1
77.2+0.1
52.0=0.2
52.4.0.1
Lipsum-FT + TPGM-C (zeg-0.2)
81.6=0.1
72.710.1
77.410.1
52.5.0,3
52.4.0.1
Lipsum-FT + TPGM-C (reg-0.5)
79.8=0.1
71.3203
77.610.1
53.0.0.1
52.0s0.1
Lipsum-FT TPGM-C (reg-1.0)
77.2.0.0
69.3x0.2
77.5u0.1
53.1
51.020.1
Lipsum-FT TPGM-C (reg=2.0)
73.7:0.0
66.6u0.1
77.110.1
52.920.2
49.320.1
Lipsum-FT TPGM-C (reg-5.0)
70.620.0
64.110.0
76.710.
52.520.1
47.8
Table 8: Ablation results on hyperparameters. There are hyperparameters L and M in Lipsum-FT. An underline highlights the top two values in each column for each group.
DomainNet
ImageNet
V2
80
89.0+0.0
66.0+o.1
67.7+01
45.4+0.
55.6.0.1
83.4-0.1
73.7-01
74.9.a.1
48.4.04
51.1
89.0an.
66.1
67.7203
45.9
55.9202
83.4.00
73.7=01
75.5
49.0am.
51.4201
39.0en.1
66.3.o.1
68.0sa.
46.0sn
1a.1
83.3=0m
73.6=0.
75.9
49.5am.
51.4.02
89 0.0.
66,3
6800200
46.0a02
56.2an.2
83.2a0.1
73.5
49.5
51.4
16
80
89.0.a.1
66.1
67.7w02
45.7
55.6x0.1
83,4+0.
73.8+02
75.1
48.6.0.5
51.3+02
32
80
88. 921.0
1a.2
67.7zou
45.7au2
55.920.2
83,42om
73.6.0.1
74.6
48.6a114
51.0201
88. 93..1..1
66.420.2
67.920.1
46,3402
56.320.4
83.2a0.1
73.5.02
75. 75.8
49.5
51.4.0.1
20
88. 9...0...
66.4-o.1
1
46.3
56.4
83.2+00
73.4+02
75.8
49.7.0.
51.5
40
88. 9a10
66.3.0.1
68.0so2
16.1
56.42n.
83.3=0.1
73.6
75.8
50.0am.3
51.4.0.
8
80
89.0+a.
66.3.0.2
68.0son
46.0en2
56,2+n.2
83.3=on
73.6=0.1
75.9
49.9.n.3
51.4.o.
160
89.0an.
66.320.1
68 0202
46.2
56,3an2
83. 33.3.00
73.7=02
75.940.1
49.8
51.4
320
89.0+o.1
66.3.o.1
68.0
46.2x02
56,3.0.2
83.3
73.6+0
76.0.a.
50.0.0.2
51.4-0
Allowing certain level of feature distortion. One can notice that distinction might exist between features before and after fine-tuning, i.c., Fo, (x) and Fo(x) for the input æ, even when employ- ing the suggested Li psum-FT regularization, as Li Lipsum-F" specifically regulates inner product values related to text features instead of simply regularizing the image features to be similar to their original values. be effective for robust fine-tumn fineituning, To be more specific, Lipsum FT goes beyond simply reducing feature distortion in all directions, e.g., through a simple Feat KD regularization which minimizes Reeateo (0) [=(x)) Fe, (x) Instead, it explicitly addresses feature distortion that negatively impacts the pre- preitrained vision- languala connection, quantified by inner product values related to text features. In other words, psipsum-F7 FF allows degree of flexibility for the feature vector to undergo
a valid point and, in fact, acts as the primary reason why Lipsum- FT can
28

---

Published as a conference paper at ICLR 2024
Table Ablation results on loss function and text guidance. It supplements Table 3.
(a) ImageNet results.
Loss function
Text guidance
ImageNet
Method
KLD MSE fixed random IN
V2
A
S
CAR-FT
83.2
73.0
71.3
43.7
49.5
CAR-FTwsR
83,2 (+0.0)
73.2 (+0.2)
74.4 (+3.1)
49.2 (+5.5)
51.1 (+1.6)
Lipsum-FIae
83.4 (+0.2)
73.7 (+0.7)
75.6 (+4.3)
48.4 (+4.7)
51.2 (+1.7)
Lipsum-FT
83.3 (+0.1) 73.6 (+0.6)
75.9 (+4.6)
49,9 (+6.2)
51.4 (+1.9)
(b) Numbers with standard deviations.
DomainNet
ImageNet
Method
R
P
I
S
IN
V2
R
A
S
CAR-FT
88.9
64.4+m.
65.8e02
43.3+0.2
53.2+05
83.2+um
73.0eoz
71.3+0.
43.7s02
49.5+02
CAR-FTwaR
88. 920.1
65,7an2
67. 1.03
44.5.0.2
55 35.2.03
83.2.00
73.2.0.1
74,4
49.2.0.
51.1
Lipsun-FTKIn LLD
89.0=0.
66.0a0.1
67.7=03
45.8.0.2
55.7+03
83.4=0.0
73.7
75.6=0.1
48.4
51.2+0.1
Lipsun-FT
89.0.0.0
66.3,022
68. 68.0z00
46.0un.2 56.2.02
83.3,0.0
73.6u0.1
75.9.0.1
49.910.3
51.4.0.
changes after fine-tuning, as long as it does not compromise the pre-trained vision-language connec- tion. Given that the linear probing baseline, which freezes features entirely, falls short of delivering satisfactory downstream performance, it becomes evident that introducing degree of feature dis- tortion is necessary for improved results downstream tasks. This poses a key challenge in robust fine-tuning, specifically in deciding how to control the level of distortion within an acceptable range. In response to this challenge, our L,i kipssumFFT approach maintains a suitable 'certain level of fea- ture distortion' by retaining the pre-trained vision-language connection, effectively addressing both reference and distribution shift performance in downstream tasks in the context of robust fine-tuning. Conducting an empirical investigation, we tested the aforementioned Feat method in the Ima- geNet scenario, yielding classification accuracy of 1 20.2 72.0a0.1 43.4203, 49.9 9.0.2) on ImageNet (IN, V2, R, A, S}. While these results are commendable, they fall short when com- pared to Li ps This underscores the effectiveness of the Lipsum-F approach, especially terms of its flexibility to accommodate feature distortion.
DomainNet results with different reference data. We extend our main experiments for the Do- mainNet scenario by using different domain as reference data for fine-tuning and employing other domains as distribution shift data. It might provide additional insights into the adaptability and robustness of the proposed method. Table 10 summarizes the classification accuracy results for B/16 in four different distribution shift scenarios DomainNet. It is important to note that applied the same hyperparameters as the innmainNett. - Doma i nNeen C, I, S} scenario presented in the main text, without any adjustments. The results distinctly indicate that the proposed Lipsum-Fr FT approach consistently outperforms other baseline methods in these scenarios.
Visualizing energy vectors. We further depict visualizations of energy vectors an M-dimensional space before and after the fine-tuning process, This aims to provide an intuitive grasp of the distinc- tions between initial zero-shot and fine-tuned models. Given the high dimensions, such as M 80, we project these vectors into a -io-dimesiiona space using t-SNE (van der Maaten & Hinton, 2008). Specifically, we generate energy vectors for the first 100 data points from each split and visualize them in two-dimensional space using sklearm ea mani fo ld. TSNE, with default pa- rameters. Figure 15 clearly demonstrates that L. psum-FT displays the least distinction in compar- ison with other baseline methods. It implies that Lipsum-FT effectively alleviate the decline in pre-trained connections between vision and language models.
Analysis on training costs. We provide the wall-clock times for the training runs of the fine-tuning methods to offer a clearer understanding of the computational complexity of each method. All training runs occurred in a consistent TPU VM environment with eight TPU-v3 cores. It would be crucial to acknowledge that, even when running the same script on the same VM within a TPU development environment, there may be variations in execution times. Hence, we supply average
29

---

Published as a conference paper at ICLR 2024
Table 10: Results with varying reference data on DomainNet. summarizes the accuracy of various methods for B/1 6 in distribution shift scenarios on DomainNet. The split denoted by an as- terisk(*) serves as the reference dataset. An underline highlights the top two values in each column, where all values are averaged from the three measurements.
(a) PomainNctt22 + DomainNct -3, C, 8,
Method
R
p*
C
I
Average
FT
74.2
79.8a0.0
60.9a0.4
38. 7.0.4
48.5
60.4
EMA
78,4
80.8+0.1
65.2+0.1
4
54.4+0.0
64.2
L2SP
74.2
79. 79.8+0.3
60.4+0.
38.4:0.3
48.0+0.3
60.2
XD
77.2au.1
79.720.2
62.220.3
39.420.2
50.3a0.4
61.8
CAR-FT
76.0av.1
80.1
61.620.4
40.020.5
50.
61.6
Lipsum- FT (ours)
78.9
80.610.1
64.820.3
23.1
53.8u0.1
64.2
(b) DomainNet- -C DomainNet - (R, P, I,
Method
R
C*
I
S
Average
FT
76.2=n.2
58. 1
79.9.0.1
39.8.0.2
53.5.0.2
61.5
EMA
79, 79.9+0.1
63.2+0.2
81.4.0.1
44.1
58.940.
65.5
L2SP
76.0
58.
79.7=0.1
40.1
53.5+0.5
61.5
XD
78.7
11.1
79.6=0.0
41.9a0.1
55.2+0.4
63.3
CAR-FT
78. 20.1
60.5a0.2
80.720.2
43.0am
55.420.2
63.5
Lipsum-FF FT (ours)
80.8
63.8a0.2
81.020.1
45.8.0.1
58.720.
66.0
(c) DomainNet -I -I DomainNet: -R, C,
Method
R
C
I
S
Average
FT
77.3+04
60.7+1.3
61.0+1.0
57.5+0.5
50.2+1.6
61.3
EMA
79.320.2
62.3a0.1
63.5e0.1
58.7.0.1
52.620.2
63.3
L2SP
77, 27.4
60.240.9
61.22166
57.520.6
49.9+1.
61.2
XD
78.0=1.0
60.5+18
60.6=1.9
57.1
48.7+2.4
61.0
CAR-FT
76. 20.
59. 59.240.2
59.6.0.7
57.4
48.8
60.4
Lipsum-FT FT (ours)
19.9
62.9s0.
63.920.3
58.8.0.2
53.420.3
63.8
(d) DomainNet- S DomainNet (R, [R,
Method
R
C
I
S*
Average
FT
71.5+04
56.0+0.2
65.4+0.7
32.2+1.0
71.6H0.2
59.3
EMA
76,4+1.
61.4+1.5
68.7+1.0
37.4+1.1
73.5+0.4
63.5
L2SP
21.5
55.810.1
65.720.3
32.820.5
71.720.1
59.5
XD
74.920.1
59.710.2
66.720.4
34.720.4
71.240.1
61.4
CAR-FT
73.6an.2
57.340.2
66.2400
35.2+0.3
72.040.2
60.9
Li Lipsum-FT (ours)
78.0
62.440.2
68.3+0.1
40.1
72.4x0.
64.2
and standard deviation values from five runs, effectively fulfilling the purpose of comparing overall training costs. The following summarizes the wall-clock times for training runs on the ImageNet scenario with 50000 training steps: FT 175 minutes, EMA took 175 + 11 minutes, 1.2.PP took 177 8 minutes, XD took 209 + 11 minutes, CAR-FT took 207 14 minutes, and Lipsum-F7 took 211 + 18 minutes, Overall, KD, CAR- FT. and Lipsum FT approaches, requiring an additional forward pass compared to FT, lead to approximately 20% increase in training time. It would be worth mentioning that Lipsum- requires virtually no cost over KD and CAR-FT.
Results for generalized zero-shot learning. The focus of the paper is addressing distribution shift scenarios under fixed classes, as opposed to evaluating the zero-shot capability across both seen and unseen classes. However, may contemplate situations related generalized zero-shot learning.
30

---

Published as a conference paper at ICLR 2024
Table 11: Generalized zero-shot learning results on CUB. It summarizes the classification ac- curacy for seen classes (S), unseen classes (U), and their harmonic mean (H). Our outcomes are averaged from three measurements, and the corresponding standard deviation values are provided. "These are the baseline results adopted from Wang et al. (2023).
Method Zero-shot
S
U
H
55.3
52.5
53,9
FT
83. +0.3
33.6=0.8
47.820.5
CAR-FT
85.2a0.1
44.6s06
58.5a0.5
Lipsum-FT
82.6a0.2
48.5
61.120.3
ZZero-shot
54.8
55.2
55.0
CCoOp (Zhou et al., 2022b)
63.8
49.2
55.6
1CoOp+ + SHIP (Wang et al., 2023)
58.9
55.3
57.1
To address this, further present outcomes for generalized zero-shot learning, using the Caltech- UCSD Birds 200 (CUB) dataset'. In accordance with the official codebase of Xian et al. (2018)', partitioned the original set of 200 classes into 150 classes designated as seen and 50 classes as unseen, the latter being excluded from the training set. Due to the incapacity of the conventional softmax classifier handle unseen classes, we opted for fixed zero-shot head weights derived from the pre-trained text model of CLIP. To elaborate, we commenced by fine-tuning the vision model, incorporating the fixed zero-shot head weights, using the training data for the 150 seen classes. Subsequently, we evaluated the performance of the fine-tuned vision model, equipped with the fixed zero-shot head weights, on the test data comprising 150 seen classes and 50 unseen classes. Table summarizes the results averaged from three trials; S represents the accuracy for 150 seen classes, represents the accuracy for 50 unseen classes, and H denotes the harmonic mean of these values. In comparison to FT, Lipsum-FT demonstrates enhanced proficiency in maintaining the original superior performance of the zero-shot model for unseen classes (specifically, L. psum-FF drops from 52.5 to 48.5, while FT drops 52.5 to 33.6). This serves clear evidence of the effectiveness of the Lipsum-3 FT regularization in preserving the pre-trained vision-language connection. Notably, despite not originally targeting generalized zero-shot learning scenario, Lipsum achieves performance level that is on par with generalized zero-shot learning methodologies.
Pittpss//ww.vviiio.. wcl.techh.cu/atttssees//uu. 200 2011/
hhttp: com
wwm//m////aaoo Abhi http://www....aooo///////////eea
31

---

Published as a conference paper at ICLR 2024
Initial (zero-shot)
-uned
L25P
CAR-FT
Lipsum-
(a) DomainNet R > Domain NomainNet-(2 (2, C, I, s}
Initial (zero-shot)
Fine-tuned
FT
EMA
L2SP
KD
CAR-FT
Lipsum- FT
IN
V2
(b) ImageNet ImageNet (V2, R, S)
Figure 15: Two-dimensional I-SNE visualizations of energy vectors. It depicts two-dimensional t-SNE representations of M --dimensionnl energy vectors for the initial 100 data points in each data split, both before (represented as Vo.. with blue markers) and after fine-tuning (represented as vo. with orange markers).
32

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL DETAILS
The code for the main experiments is built on JAX (Bradbury et al., 2018), Flax (Babuschkin et al., 2020), Optax (Babuschkin al., 2020), TensorFlow Fow Datasets (Abadi al., 2015), and Transformers (Wolf et al., 2020) libraries, which are available under the Apache-2.0 licence." All experiments are carried using eight TPUv2 or TPUv3 cores, supported by TPU Research Cloud. Code is available at https: githab...b.... com/ cs www.g/giissuuurrr..eee..
Pre-trained weights. Throughout the experiments, we employed the pre-trained weights provided by OpenAI. These weights can be accessed publicly through the following URLs:
3/32: http:///ww...hgggha..///////////
: httpps://hhggggggaaac.co.o
L/14: http:///wwaaggggg......///////aa
Datasets. We employed the following datasets in this work, where Figure 16 visually depicts exam- ple images from each dataset: (1) ImageNet (IN; Russakovsky et al., 2015) consists of images from 1,000 categories. Due its significance in the field of computer vision, several related datasets have been introduced: ImageNet-Ve V2 (V2; Recht et al., 2019), ImageNet-Renditien 2021a), ImageNet-A (A; Hendrycks et al., 2021b), and ImageNet-Sketchh (S; Wang al., 2019). These datasets are considered as natural distribution shifts of the ImageNet dataset (Taori et al., 2020), and it is common practice evaluate the robustness of models fine-tuned on ImageNet using these datasets (Taori et al., 2020; Radford et al., 2021; Wortsman et al., 2022a;b; Mao et al., 2023: Tian al., 2023). (2) DomainNet (Peng et al., 2019) comprises images categorized into 345 classes from six distinct domains, which include photos (R), paintings (P), clipart (C), infographics (I), sketches (S), and quickdraw (Q). We excluded the quickdraw data from experiments since obtaining meaningful results, either before after fine-tuning, is difficult due to its extreme sim- plicity (to be more precise, the classification accuracy remains around 5% both cases). To address duitt--distribution generalization rather than domain generalization, fine-tunee models solely data and then evaluated them on all domains.
(R; Hendrycks et al.,
33

---

Published as a conference paper at ICLR 2024
Reat
Painting
Clipart
Infograph
W CLEAN WORK SPARE
Sketch
Qui Quickdran raw
(a) Doma CommainNir wnweinNettttt R and DomainNet- (p,, C, I, 8, Q) datasets.
IN
V2
OPIM
GNATUP
CIIDMM
(b) ImageNet and ImageNet- [v2,, S) datasets.
Figure 16: Example images from datasets. shows randomly chosen images from (a) Do- mainNet datasets and (b) ImageNet datasets, respectively.
34


Reviews:
[{"review_id": "NLhgk4EMVM", "reviewer": "Reviewer_MuzV", "summary": "The paper points out that fine-tuning zero-shot model like CLIP can improve downstream performance, however, the accuracy of the fine-tuned model falls short of the original zero-shot model across distribution shifts. To address the chanllenge of robust fine-tuning, authors first delve into the problem by employing feature distortion theory and joint energy-based models. Subsequently, they introduce a novel robust fine-tuning algorithm called Lipsum-FT. This approach leverages random text guidance as a regularization technique during the fine-tuning process to minimize the change in energy. Authors conduct extensive experiments on two datasets to demonstrate the effectiveness of the proposed approach in addressing distribution shift scenarios.", "questions": "1. It remains unclear whether there exists a weight for incorporating the regularization term proposed in Eq. (7) into the loss function. In Appendix B.2, authors have discussed existing methods that involve weights related to the regularization term, such as $\\lambda_{L2SP}$ in Eq. (10), $\\lambda_{KD}$ in Eq. (12), and $\\lambda_{CAR-FT}$ in Eq. (12). Authors have also detailed how to select these hyperparameters. However, there seems no mention of how the weight for the proposed method is determined.\n2. The precision of the standard deviation values in Table 2(b) should be improved. The values of NLL are presented accurately to two decimal places, whereas the standard deviation values are limited to only one decimal place, with many of them showing as 0.0. Ensuring consistent precision in reporting these values would enhance the clarity and reliability of the results.\n3. There may be a typo in the gradient descent update rule presented in Eq. (9). It should be as follows: $\\theta_t = \\theta_{t-1} - \\eta \\nabla_\\theta L_{CE}(\\theta)$. It's advisable for the authors to thoroughly review other equations to ensure they are accurately represented.\n4.It would be interesting to know if the proposed method is applicable to various fine-tuning techniques, such as adapters, LoRA, and prompt tuning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigate the trade-off between the reference and distribution shift data when fine-tuning the zero-shot CLIP model, utilizing feature distortion theory and joint energy-based models as analytical tools.\n2. This paper proposes a simple and effective regularization term based on the correlation between the similarity of vision features and text features derived from the fine-tuned model and original model respectively. Specifically, the text tokens are generated randomly.\n3. The proposed method outperforms the original zero-shot model and exisiting robust fine-tuning methods in both reference and shift domains, demonstrating its superior performance in handling distribution shifts.", "weaknesses": "1. It would be benefical if authors could include visualizations of $v_{\\theta, \\phi}(x)$ and $v_{\\theta_0,\\phi}$ to provide an inituitive understanding of the distinctions between the original zero-shot model, the fine-tuned model with exisiting methods, and the fine-tuned model with the proposed emthod.\n2. Expanding the experiments to involve various domains as reference data for fine-tuning and using other domains for evaluation would enhance the comprehensiveness of the study. This approach can shed light on the adaptability and robustness of the proposed method in different real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Er2xAgqeXX", "reviewer": "Reviewer_2e7P", "summary": "This paper proposed a fine-tuning model, Lipsum-FT, by utilizing the language modeling aspect of the vision-language pre-trained models for zero-shot classification.", "questions": "What means of four distribution shifts DomainNet-{P, C, I, S}. Figure 1 fails to represent the distribution shifts.\n\nThere are too many contents that need to be referred to Appendix. Please reduce some important Appendix contents and put them into the text.\n\nIt looks like capturing correlations between images and each language by calculating their inner product vectors in Equation 8. The meaning of Equation 8 wants to express is to match all the language information one by one or to match the current m-th information? If it matches M language information, how much does the algorithm complexity increase? What is the significance of matching with alonely m-th?\n\nLack of complexity analysis on the Lipsum-FT algorithm and its impact on the experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper proposed a fine-tuning model, Lipsum-FT, by utilizing the language modeling aspect of the vision-language pre-trained models for zero-shot classification. Lipsum-FT as an incremental algorithm enhances the robustness of zero-shot models by combining language and visual information.", "weaknesses": "The motivation is unclear. The authors mention a problem with cross-distribution shifts, but Lipsum-FT is a fine-tuning model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6cDrKfqNb", "reviewer": "Reviewer_26HN", "summary": "This paper focuses on the problem of robust fine-tuning a large-scale vision-language pre-trained model, which is expected to obtain enhanced downstream performance while maintaining its accuracy across distribution shifts on zero-shot tasks. By investigating the behavior under the perspectives of feature distortion theory and joint energy-based models, the authors propose a robust fine-tuning algorithm Lipsum-FT. Experimental results on DomainNet and ImageNet proves effectiveness of their proposed method.", "questions": "1. As the main contribution of this paper is to utilize the regularization term in (7) to minimize the energy gap in (6), please explain why the energy gap is chosen for improving the robustness of zero-shot model fine-tuning. Why is it defined to be the squared difference of two inner products as in (6)?\n2. Could the authors give more details about how the tokens $\\mathbf{t}$ are generated? The authors just assert that they are generated randomly from the vocabulary. I also want to know what types of texts are used for constructing such regularization and to what extent it covers the common used semantic information.\n3. How much extra computation are introduced by such a regularization term?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on the problem of how to maintain the performance of a zero-shot model on distribution shift data while improving its performance on reference data during fine-tuning, which is an important and valuable topic.\n2. The proposed method is easy to realize since it simply introduces an extra regularization term.\n3. The idea of utilize the language model to construct regularization on the vision model is interesting.\n4. The English writing is good and I do not find obvious grammar errors or typos.", "weaknesses": "1. I am not sure whether the novelty of this paper can be regarded as significant. It just introduces a regularization item to make the vision model keep the original inner products with features generated by the language model after fine-tuning.\n2. The illustration organization of this paper is not clear enough. Therefore, even though the key idea is not so complicated, I find it difficult to understand the viewpoint quickly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZiiRXdCzU", "reviewer": "Reviewer_Q7od", "summary": "The paper introduces a method named LIPSUM-FT for zero-shot robust fine-tuning. Specifically, the authors, by observing the relationship between accuracy and energy gap under distribution shift, propose enhancing the accuracy of fine-tuned pre-trained models under distribution shifts by reducing the energy gap. The experimental results demonstrate that the method proposed in this paper can effectively improve the performance of CLIP during zero-shot fine-tuning.", "questions": "The authors claim that the text tokens are randomly generated. What are the specific rules for generation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors explain the performance degradation of fine-tuned pre-trained models under distribution shifts from the perspective of energy models.\n\n2. The proposed LIPSUM-FT effectively enhances the robustness of fine-tuned pre-training.\n\n3. Ablation studies indicate that LIPSUM-FT is insensitive to random token lengths and token quantities.", "weaknesses": "1. Utilizing energy models to explain the fine-tuning of pre-trained models seems not to be essential. As per my understanding, the objective of the method in this paper as well as related methods ([1,2,3], etc.) is to reduce the difference in features extracted by the models before and after fine-tuning.\n\n2. The authors claim that the text used is randomly generated, but it appears from the code in the supplementary material that tokens are sampled from the openai_imagenet_template. According to CAR-FT, using all templates as text input also yields good performance. What then is the significance of random token sampling in this scenario?\n\n3. It is suggested that the authors provide a brief introduction to energy models in the related work section.\nIn Figure 1, it is not mentioned which points different learning rates in the left graph and different steps in the right graph correspond to.\n\n[1] Context-aware robust fine-tuning. \n\n[2] Fine-tuning can cripple your foundation model; preserving features may be the solution.\n\n[3] Robust fine-tuning of zero-shot models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)