Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
Authors: Xiyuan Wei
Ming Lin
Fanjiang Ye
Fengguang Song
Liangliang Cao
My T. Thai
Tianbao Yang
Venue: ICML 2025 spotlightposter (2025)
Keywords: Contrastive Learning
Multimodal Learning
Distributionally Robust Optimization
CLIP
Foundation Model
Primary area: general_machine_learning->representation_learning

Abstract: This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named **model steering**. While ad-hoc methods have been used in various contexts, including the training of large foundation models,  its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called **DRRho risk minimization**, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering.  Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches. Code is released at [github.com/Optimization-AI/DRRho-CLIP](https://github.com/Optimization-AI/DRRho-CLIP)

Model Steering: Generalization Bounds and Scaling Laws
Learning with a Reference Model Improves
Xiyuan Wei
Ming Lin Fanjiang Ye
Fengguang Song²
Liangliang Cao
My T. Thai
Tianbao Yang'
Abstract
Reference
Target
70
This paper formalizes an emerging learning paradigm that uses a trained model as a refer- ence to guide and enhance the training of a target model through strategic data selection or weight- ing, named model steering. While ad-hoc meth- ods have been used in various contexts, including the training of large foundation models, its under- lying principles remain insufficiently understood. leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called DRRho risk minimization, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, pro- vide theoretical insights into why this approach improves generalization and data efficiency com- pared to training without a reference model. To the best of our knowledge, this the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building these insights and the connection be- tween contrastive learning and DRO, we intro- duce a novel method for Contrastive Language- Image Pretraining (CLIP) with a reference model, termed BRRhO-CLIPP Extensive experiments val- idate the theoretical insights, reveal a superior scaling law compared CLIP without a refer- ence model, and demonstrate its strength over existing heuristic approaches. Code is released at giihub.comOOpmmizziiooAADDD
ImageNat-1h Tap- Accuracy (%)
68
Ralentrcee
66
64-
62
60
OpenAl CLIP
DRR
Figure Comparison between target model (ViT-B/16) trained by the proposed DRRho- CiRhoCLIIP and the reference model it leverages. OpenAl CLIP (ViT-B/32) was trained on a private 400M dataset with 12.8B samples seen and 32768 batch size. DRRho-CLIP model was trained on DFN- 192M with 1.28B samples seen and 5120 batch size, and using OpenAl CLIP as a reference model
state-of oftte-of-the-att models. While the data and code are often proprietary, the resulting models are sometimes released publicly, such as the CLIP models from OpenAl (Radford et al., 2021) and the Llama models from Meta (Dubey et al., 2024). This raises an intriguing question:
"How can we leverage public models to improve training of a target model on custom datasets?"
To study this question, we explore an emerging learning paradigm that leverages a trained model as reference to guide and enhance the training through strategic data se- lection or weighting. We refer to this paradigm model steering. Unlike the widely adopted knowledge distillation framework. model steering does not assume that the ref- erence model is a stronger teacher; in fact, it can lead to the training of a model that ultimately surpasses the refer- ence model in performance, i.e., enabling weak to strong generalization (cf. Figure 1).
1. Introduction
With the success of large foundation models, numerous panies and research groups have entered the race to develop
'Texas A&M University "Oracle Indiana University 'Google Unniversity of Florida. Correspondence Tianbao Yang tiianbao-yang tam
A few works have studied learning with a reference model (LEAR) in different contexts and demonstrated its effective-
'Our training took 376 GPU hours on H100 (2 days), OpenAl CLIP (ViT-L/14) model was trained on 256 V V100 with 12 days, which gives estimate of 256 12*24/1 11122/111.6e. 1.6=6356 GPU hours for training ViT-B/32 as FL.OPs 11.6 smaller.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
ness in accelerating the training of various models, including
VIT-B-22 OpenCLiP
VIT-8/16 DRRPh-CLIP
VIT-U/14
classification models (Mindermann et al., 2022; Evans et al., 2025), large language models (Lin et al., 2024; Xie et al., 2023), CLIP models (Evans et al., 2024a:b). An interest- ing idea emerging from these studies is to perform online data selection, sampling, or data weighting using the ref- erence model through a shifted loss called the RHO loss E(0, =) E(Oret, z), where 0 denotes the target model to be learned, and Per denotes the reference model, z denotes data sample, and f(:-)) denotes loss of interest. This approach is intuitive in the sense that data with high loss of the current model {(O,, z) and a low loss of the ref- erence model f(Gret, =) means that it has high learnability and should be used for training (Mindermann et al., 2022).
40
ImageNes- Emor
20-
E-5.77 C-
E-7.55-C-032
10"
10"
Compute (BFLOPS Samples Seen)
Figure 2: Scaling performance of OpenCLIP (Cherti et al.,
2023) and the proposed DRRho-CLIP, which uses the Ope- nAI CLIP model (Radford et al., 2021) as the reference model. We conduct experiments of the two methods under different settings to fit scaling laws, as shown in the bottom left corner (c.f. Section for more detail).
However, the theory behind this approach is quite limited, especially on improving generalization, which hinders our understanding of its effectiveness and derivation of best practice for model steering. Mindermann et al. (2022) tried to motivate this approach through maximizing the likelihood of true labels of classification for a hold-out dataset based on the selection of informative data in the current mini- batch. Through Bayes' theorem and the approximation of negative log-likeli log-likelihoods by losses, their analysis yields data selection strategy by selecting data in the mini-batch that have the top values of E(0,, z) - (Ore, z). which is termed RHO loss selection. While their analysis offers some intuition into why the RHO loss is a reasonable choice for data selection, it falls short of providing guarantees on how and why this approach improves generalization with reduced training data. Moreover, there is no theoretical framework to guide the optimal selection or weighting of data to accelerate training effectively. The heuristic approach that selects data with top RHO losses in the mini-batch is not necessarily the best.
Our framework not only provides a theoretical foundation for existing heuristic approaches data selection, sam- pling, or weighting but also offers improved practices for model steering. To illustrate its practical utility, we focus on contrastive language-image pretraining (CLIP) with a reference model. By leveraging the connection between contrastive loss and DRO (Qiu et al., 2023), we introduce a novel method for CLIP with a reference model, termed DRRho-CLIP, which utilizes the DRRho risk for each an- chor data point.
Our experiments demonstrate the effectiveness of DRRho- CLIP. For example, when using OpenAl's CLIP (ViT- B/32) model as a reference, DRRho-CLIP with ViT-B/16 as the backbone trained from scratch on the 1FN-192M dataset (Fang et al., 2024) achieves 68.84% zero-shot clas- sification accuracy on ImageNet- 1K data. outperforms OpenAl's CLIP performance of 63.32% trained on differ- ent 400M dataset, and OpenCLIP's performance of 67.8% trained on the same data (cf. Figure 1). In addition, our extensive experiments using various reference models and large-scale datasets reveal the following: (1) DRRho-CL.I IIPIP achieves comparable performance to the standard CLIP training method (without a reference model) while using significantly less training data (e.g., a 50% reduction). This offers great potential in reducing the burden on collecting high-quality data. (2) DRRbo-CLIP CLIP outperforms existing heuristic data sampling or selection methods applied to the standard CLIP training framework. (3) When paired with a strong reference model, DRRho-CLIP integrates seamlessly with knowledge distillation, surpassing existing knowledge distillation methods for CLIP training (Vasu al., 2024). (4) In terms of scaling performance, DRRho-CLIN has better scaling law than OpenCLIP (Cherti et al., 2023) (cf. Figure 2).
To address this gap, this paper introduces a novel learning framework for enhancing the understanding and practice of model steering based on the RHO loss ((0, z) ((1re, 2). Here, we use the term RHO loss in broader sense, which is not necessarily restricted to classification as derived in Min- dermann et al. (2022). Our framework builds on Distri- butionally Robust Optimization (DRO). Traditional DRO seeks to minimize the worst-case risk over perturbed data distributions within an uncertainty set derived from the em- pirical distribution. We extend this idea by applying DRO to the RHO loss ((0,2) x) ((Oret 2) across the training data, yielding risk function for model steering, which we term DRRho risk. Leveraging the generalization theory of DRO, we derive theoretical generalization bounds of DRRho risk minimization, offering insights into how enhances general- ization. To the best of our knowledge, this work presents the first generalization theory for model steering, significantly advancing understanding of its effeetiivnness
Our contributions are summarized as follows:
2

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
We propose a novel framework for model steering based on DRO and th RHO loss. Theoretical gen- eralization bounds are derived and analyzed, which provide insights into why it improves generalization.
the key differences lie in: (1) a sample that may be dis- carded in later stage of training could be useful in the early stage of training; (2) both the model to be trained and the reference model are used together for data selection, sampling, or weighting. The RHO loss has been used in mul- tiple papers (Mindermann et al., 2022; Evans et al., 2025; Lin et al., 2024; Xie et al., 2023; Evans et al., 2024abb) for training different models. Different from these prior works, our work aims to provide theoretical foundation for data selection, sampling or weighting with the RHO loss, and use it to derive better practical approaches for training CLIP models. To the best of our knowledge, this is the first work that provides a generalization analysis for learning with a reference model.
We introduce DRRho-CLIP, a method for training CLIP models with a reference model. Extensive exper- iments on large-scale datasets and various reference models demonstrate the effectiveness of DRRho-CLIP, including better scaling laws.
2. Related Work
Using pretrained model to improve the training of target model is not a new idea, which has been long studied in deep learning. A typical approach is transfer learning (Donahue et al., 2013), which uses a pretrained encoder as initializa- tion for finetuning a target model on a different supervised dataset. However, transfer learning restricts the target model to have the same structure (at least in the encoder part) the pretrained model. There have been some studies trying to expand the architecture of the pretrained model to that of the target model so that the target model can inherit the knowledge of the pretrained model (Chen et al., 2016; Wang et al., 2023; Du et al., 2024). Another category of related works is knowledge distillation, which distills knowledge from stronger, usually larger, teacher model to a smaller student model (Hinton, 2015). Knowledge distillation has been extensively studied for learning various models (Hin- ton, 2015; Vasu et al.. 2024; Qin et al., 2022). However, these approaches do not consider using the pretrained model for data selection or weighting to facilitate the training of target model. Hence, the studied technique in this paper is complementary knowledge distillation.
Our algorithm for optimizing the proposed contrastive loss informed by DRRho risk is based on recent advances for optimizing global contrastive losses (Yuan et al., 2022; Qiu et al., 2023; Wei et al., 2024). In particular, Yuan et al. (2022) proposed an optimization algorithm termed SogCLR for optimizing a global contrastive loss, which does not suffer from the limitation of using a small mini-batch size. Qiu et al. (2023) has established the connection between DRO and the global contrastive losses, which enables opti- mization of individual temperature parameters in contrastive learning. Wei et al. (2024) proposed techniques for training CLIP models in a distributed system.
Our theory of DRRho risk minimization is built on existing generalization error bounds for distributionally robust opti- mization (DRO) (Duchi & Namkoong, 2016; Duchi et al., 2021), which will be reviewed in next section. We notice that a group DRO formulation was considered in (Xie et al., 2023) for using a reference model to learn the weight of a dataset when training language models. However, their for- mulation does not consider the regularization on the weight variables in DRO, which cannot benefit from the theoretical guarantee as derived in this paper.
Leveraging a pretrained model for offline data selection or pruning has been studied. For example, Schuhmann et al. (2021; 2022); Gadre et al. (2023); Fang et al. (2024) leveraged a pretrained model to curate a training dataset to accelerate the training of CLIP models. For each image- text pair in the original dataset, they use the pretrained model (e.g., OpenAl's CLIP model) to compute score (e.g.. similarity score between the image and text) for evaluating its quality. A new subset is then constructed by selecting samples whose score exceeds a certain threshold. Ankner et al. (2024); Marion et al. (2023) have studied a similar idea for pruning the training data for language modeling.
3. Preliminaries: DRO
Notations: Let = ~ P denote a random data, where P represents the distribution of data. We denote by E θ the model parameters, where θ is the space of model param- eters. Let ((O, denote a loss of interest. The expected risk of 0 is defined as R(0)- = EE-P(E(O, =)]. Denote by o, arg millage R(0) the optimal solution that minimizes the expected risk. For any model 0 E O the excess risk R(0) R(0.) is measure of generalization performance of 0. Let = (E(O,-), 0 E 9). To derive the generaliza- tion error bounds, we need some complexity measure of the function class F. We follow Duchi & Namkoong (2016) by using the VC-dimension of F denoted by d, VC(F), which is defined VC-dimension of the set of subgraphs of functions in F (van der Vaart& & Wellner, 1996).
This paper falls into the category of exploiting pretrained model for online data selection, sampling or weighting. We refer to this learning paradigm as model steering, as the pretrained model is used as a reference to guide training. Mindermann et al. (2022) proposed the RHO loss for se- lecting samples in a mini- for training a classification model. Though similar to offline data curation approaches,
3

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Given n samples 21...., Sn drawn i.i.d. from P. DRO solves the following problem:
We refer F(0) as the DRRho risk. Building upon this, we formulate the DRRho risk minimization problem:
0. E arg min DEA
sup Da(p.//)So// 1m1
Σ pre(O, zi),
(I)
0, E arg min F(0)- AEe
(3)
Next, present the generalization error bounds of DRRho risk minimization. We consider the function class F, {E(O,-) -) E(Oret. -), 0 E } and abuse the notation de = VC(F,). The proofs of the following results are presented in Appendix A.
where A is a simplex, 1p/p. = - - the ø-divergence between and uniform probabilities, (e.g., Kullback-Leibler divergence (Kullback, 1997), CVaR- divergence (Levy et al., 2020), >=-divergence (Duchi & Namkoong, 2016)), and p 2 0 a hyperparameter. DRO minimizes worst-case risk over all possible perturbed dis- tributions from the empirical distribution within an uncer- tainty set. The supremum over the weights p indicates that if a sample is hard (i.e., has large loss value), then it will have large weight so that the algorithm will pay more attention to optimizing it.
0, Σ 1 Rd P
}} p(pin) denotes
Theorem 4.1. Under the same setting of Theorem 3.1, let C = log - log c + du log 16e + 2d, log n. for a given
> 0 and some constant e < cc. Cf n 2 32M and
P Z 9C1, with probability at least 1-8. 8,
R((0) + V Var(e(0..) E((Ond,))) + C2
2p
R(O.) I inf AEB
The generalization error bounds of DRO have been devel- oped (Duchi & Namkoong, 2016; Duchi al., 2021). We present corollary from Duchi & Namkoong, Theorem 3.
where C2- = (50p/3 + 4)
We present two corollaries to understand how DRRho risk minimization improves the generalization over DRO/ / ERM.
Theorem 3.1. Let D& be x² diverrgence with d(t) = (t- /22/2. Assume that ((0,)) E [Mo, Mi] with M = M 1M Mo- Let C = =log 1/4 + loge di log 16e + 2d, log n fofor a given
Corollary 4.2. Under the same setting of Theorem 4.1, we have
8 > 0 and some constant e < oc. If C1 n 2 8M2 and
p 2 9C1, then with probability at least 1- 8,
R(O.) S R(0.)+ + V 22 n
Var Var( -(((,,)))(((eet))) + C2
n
2p Var(e(O. )) + C2
R(O.) ) S inf AEE
R(0) +2
n
n
Remark:
Comparing the excess risk bound of R((-)
R(0.) for DRRho with that of R(O.)= - R(0.) for DRO, we can see that the variance term changes Var(e(O...) ((Oret, -)) from Var(e(O..,.) It is reasonable assume that the reference model 0:et is sufficiently trained such that ((Oret, -) has a similar distribution to E -); hence we expect that Var(E(O.,; ((Oref))))))) would be much smaller than (ar(e(e..)))
where Var(E(0,.)) .)) denotes the variance of E(0, 2) for 2 P and C2- = (25p/3 + 2).
The above theorem indicates that DRO may have a better excess risk R(O.)- - R(O.) than that of ERM when the variance term Var(e(O...) at the optimal solution O. is small (Duchi & Namkoong, 2016). Although the above re- sult is derived for the -dvverrgeee,,, one can follow Duchi et al. (2021) to derive similar results for other divergence, e.g., KL-divergence, Nevertheless, we focus on the theoreti- cal insights that DRO can bring to our framework.
The following corollary provides insights into the reduced sample complexity of DRRho risk minimization in achiev- ing the same level of generalization the reference model.
Corollary 4.3. Under the same setting of Theorem 4.1, and assume re E O we have
4. DRRho Risk Minimization
C2 R(Ō.) - R(O.) S R((0et) R(00) + n
A central argument for the improved generalization bound of DRO is when the variance Var(E(O...)) is small. However, achieving a low variance is not always guaranteed. To ad- dress this, we propose to leverage a reference model Oref to reduce this variance. Specifically, our framework of model steering involves replacing the standard loss function ((0.-) with the RHO loss, defined as ((0,)) :) = (( - E(Oret,)) In particular, we define the following risk:
The above result allows us to compare the ex- cess risk of DRRho risk minimizer 0, with that of refer- ence model 0,e E Ô that from the same family. Sup- pose that reference model 0 is learned using ERM on dataset of m samples. A standard generalization error analysis (Boucheron et al., 2005) would yield an excess risk bound on the level of O (1/m),), (mmmm.i.. i..., R(Orer) - R(O.)= = O (m) In order to reach the same level of general- ization error of the reference model, DRRho needs only
Remark:
F(0)= = sup De(D.1/m)Su/n 1=1
Σ Pe(E(0.2)) - E(Orot, zi)). (2)

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
n=O(ym (m))) samples, which dramatically reduces the sam- ple complexity (m) of ERM without a reference model.
Finally, we would like mention that efficient stochastic algorithms have been developed to optimize the objectives in Equations (4) to (6) (Qi et al., 2023b;a; Wang & Yang. 2023; Hu et al., 2023). Hence, solving the proposed DRRho risk minimization with these advanced optimization algorithms could yield better guarantees than heuristic approaches in existing studies. We illustrate this with an application of CLIP training in next section.
Before ending this section, we discuss how DRRho risk minimization provides foundation for existing heuristic ap- proaches for data selection, weighting, and sampling based on the RHO loss, and for inducing better practices.
Data Selection: When we use the CVaR divergence ((t) = 1 t S n/k: and ((t) oo otherwise, the DRRho risk becomes the average of top-k: RHO losses:
5. DRRho-CLIP with a Reference Model
In this section, we explore the application of the DRRho risk to CLIP training (Radford et al., 2021) with a reference model. Although the proposed DRRho risk minimization framework is general and can be applied to training vari- ous models, we focus on CLIP training for several reasons: (i) CLIP involves more complex data structures, including anchor data and their corresponding negative samples for contrastive learning. Without guidance from theory, heuris- tic methods that sample data based on the RHO loss may not give optimal performance (Evans et al., 2024a). (ii) The publicly available OpenAl's CLIP model has already been utilized to enhance CLIP training by filtering high-quality data based the CLIP score (Gadre et al., 2023). This raises an intriguing question: can our framework deliver additional improvements when using OpenAl's CLIP model as a reference on the filtered data?
F(0) K
((. E(Oret: 214)-
(4)
where 2 denotes the data whose RHO loss is ranked at the i-th position in descending order. Existing studies have applied RHO-loss-based data selection to the mini-batch for simplicity (Mindermann et al., 2022; Lin et al., 2024), which do not necessarily optimize the average of top-k RHO loss among the whole dataset as in our framework.
Data Weighting/ / Sampling: If we use the KL-divergence (L( 1/n) = >i= Pi log(pan), where (() = L log (
++ L, then from the Lagrange dual theory we can derive:
n
exp (e(O, 2.) [Oroff Zi)
F(0)= := min T log +20
T
+ Tp/n..
(5)
For CLIP, the training dataset consists of images and their corresponding text descriptions. We use æ to denote an image and y to denote a text, and use (æx, y) to denote pair. We use S ((x1, yr),..., (E.. yn)) to denote training set of size n. Given an image 2;, let ei,i ei (01 xx) E Rd denote image representation by an image encoder with parameter 0,. Similarly, 62 = e2 y:) E Rd denotes the embedding of text y; by the text encoder with parameter 0g. Let 0 = (0), 02) denote the parameters of the image encoder and the text encoder jointly. Let S._ S (i) denote the dataset without i-th pair. Let s(x,, yj) denote the cosine similarity between i-th image embedding e and j-th text embedding e2.j-
If we compute the gradient of the above objective in terms of 0 given 7, we obtain E pt VoE(O, 2;), where
exp
Pi
(00,2,)-((0mrff
Σj=1 exp
Hence, the above DRRho risk acts like assigning different data different weights such that data with a larger RHO loss has a higher weight in the gradient calculation. Heuristic approaches have implemented this idea by sampling data in large batch following p, calculated based on the mini- batch data to create a smaller batch (Evans et al., 2024a;b).
In order to apply DRRho risk to CLIP, we leverage the connection between a contrastive loss and DRO (Qiu et al., 2023). For simplicity of our presentation, we consider the KL-regularized DRO formulation (6), which yields an ob- jective with tunable temperature parameter Similar extensions can be made to KL--constrained DRO (5) for learnable temperature (cf. Appendix C.3). First we define the contrastive loss function for each anchor image data æ; without using a reference model. To this end, we define a pairwise loss E(0,, x;, yj) = s(x; yj) - s(x, y:), which measures the gap of similarities between negative pair (æ:; yj) and the positive pair (æ;, y:) of the anchor data æ.. Then we use DRO to aggregate these individual pairwise
To simplify the complexity of optimization, one can turn into hyperparameter, which is equivalent to using a KL divergence as regularization in defining the DRRho risk:
F(0)= sup Σ
pe Pi(E(O, 2,) E(Orer, 21)) 7 (L. 1/n)
DEA i=1
n
(e(0, 21) -((Oref: (0)))
=loo
exp(
5

---

Model Stecring: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
losses for yj E S into the following loss of ay;:
Algorithm 1 DRRho-CLIP
1: Input: Model 0 ,7", sequences (uf,)) { {ug,)). J. number
12
Fao F0. (), == max PEA
pj pj Tt, F,,y TKL(p, 1/n),
of iterations T.
for 1 O,..., T do
==1
3: Sample batch B C S and compute features
1 10, vi)
4: Update "1 (+1 1+1 using (10) for all E B
= T
exp
(7)
2,1
5: Set u = u1, "2 = u²,, for i & 1
j=1
6: Compute Gi G3 using Equation (11)
To apply DRRho risk for integrating a reference model, we plug the following shifted loss into the above formulation:
7: Update 0-1 using G + G32y as gradient estimator
with an optimizer (e.g.. AdamW)
end for
00, Peet, Ls yj) == 0 T. yj) E(Oret, Z;; Uy)-
Efficient Implementation: It is notable that, like other LEAR methods, DRRho-CLIP requires computing the loss of the reference model on the training data. It is expensive to compute these losses on the fly. To reduce this cost, we compute the embedding vectors of training data by the ref- erence model in an offline manner. During training, we load the pre-computed features of the current mini-batch from the disk and compute the loss based on the pre-computed features. Similar approaches have been used in (Vasu et al., 2024; Evans et al., 2024a).
As result, we obtain the following DRRho contrastive loss for each image an;
i(0, Ojet, 2t, 2i,3y), My)
F(O, x., S) log
exp
(8)
Similarly, we define a DRRho contrastive loss for text yı:
_ Σ
(00 O Peef yi, D.y)
F(O, yi, S) = -oo
exp(
==1
Then, we solve the following optimization problem:
6. Experiments
1 n (F(O, S) + F(0.y. S)).
In this section, we conduct experiments to demonstrate the superiority of our proposed framework, where we focus on training CLIP models. First, we empirically verify claims of our theory: (1) our framework is more data-efficient than learning without reference model, and (2) the variance of the RHO loss in the excess risk bound of our framework is lower than that of a regular loss. Next, we compare DRRho-CLIP with other baselines, and we also show that DRRho-CLID can be seamlessly integrated with distillation methods. Finally, study the scaling law of DRRho-CL.N and show that it has a better scaling trend than OpenCLIP.
min
0 n2
To optimize the above objective, we use the SogCLR algo- rithm (Yuan et al., 2022) that has a provable convergence guarantee without using a large batch size. In particu- lar, the algorithm maintains two sequences of estimators uf. u2,,0t t 1....,77 T to track the inner average in the log function of F(0, S) and F(O, y, S). At t-th iteration, the following update with mini-batch B are executed:
4-1 = (1 - ui) + BeB exp( (O,, Orof; Di Us)
The training datasets we use consist of CC12M (9M samples) (Changpinyo et al., 2021), DFN-192M, (192M samples) (Fang et al., 2024) and DFN-12M (a 12M subset selected from 192M)
Data:
le Ozff yi, 0z,)
u/ = (1 - -1) + y Be exp(
(10)
where B B'y, \ {i) and E,eøy_ denotes the average over
The reference model is either ViT-B/32, ViT- B/16 or ViT-L/14, which are either pretrained open- open-weight models or models we trained from scratch. For ease of presentation, we use "model (data)" to denote a model and the data on which it is pretrained. The following reference models are pretrained open-weight models: ViT- B/32 (WIT-400M) (Radford et al., 2021), ViT-B/16 (DFN- 2B) and ViT-L/14 (DFN-2B) (Fang al., 2024). The target model trained from scratch could be VIT-B/32 or ViT-B/16.
Models:
data in B and % is regarded as an inner learning rate.
Then we compute the gradient estimators by
& (0,, O.of vi, vz,U,) Un)
G Eieg
1I VoEyEB exp(
E u
(2, Orot: ye vr
G = EieB 14I E +
T
VolEgER;
exp(
u2,1
(11)
where E is treated as a hyperparameter in practice (Wei et al., 2024). We present the details of our algorithm in Algorithm 1, which referred to as DRRho-CLIP.
Metrics: We leverage the Datacomp benchmark (Gadre et al., 2023) to evaluate the performance of target models, which comprises 38 zero-shot classification and retrieval
6

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
FesCCLIP (100%
DRRho-CLIP
DRRho-CLIP
(50%
45
ImagreNet Molt-1KA Accuracy (%)
36
30-
35
35
20-
30
16
25
100
200
300
100
200
300
Samples Seen (million)
Samples Seen (million)
Samples Seen (million)
42.5-
40
Datacomp Airerage (56)
40.0
30
36-
35.0
25-
32.5
32
30.0
30
27.5-
100
200
300
100
200
300
100
200
300
Samples Seen (million)
Samples Sean (millio))
Samples Seen (million)
(a) Target: ViT-B/32 (CC12M)
(b) Target: ViT-B/16 (DFN-12M)
(c)
Target: ViT-B/16 (DFN--1M)
Reference: ViT-B/32 (WTT-400M)
Reference:
VIT-B/32 (WTT-400M)
Reference: ViT-B/16 (DFN-9M)
Figure 3: Performance curves of FasICLIP and DRRho-CLIP with different target and reference models (with each column representing one combination). Top row: ImageNet Top-1 accuracy, bottom row: Datacomp average performance.
tasks. The numbers report the ImageNet-IK 1K Top accuracy and the average performance on the 38 tasks.
The performance curves of different methods different metrics are plotted in Figure 3. From the results we can observe that (i) with OpenAl's ViT-B/32 (WIT-400M) as the reference model, DRRho-CLIP with 50% of training data can achieve comparable performance to FastCLIP on the whole training data (Figure 3a, b): (ii) with 100% train- ing data, DRRho-CLIP outperforms FastCLIP by large margin; (iii) even with weaker reference model ViT-B/16 (DFN-9M) (Figure 3c) trained by using FastCLIP 9M subset of DFN- 12M, DRRho-CLA still benefits from it, achieving better performance with 75% data (9M) than FastCLIP trained on DFN-12M without reference model.
Hardware and Training framework: Unless otherwise specified, we use a batch size of 5120 on H100 GPUs for training ViT-B/16, and use batch size of 4096 on 8 A100 GPUs for training ViT-B/32. We implement method using the codebase of FastCLIP distributed train- ing framework (Wei et al., 2024). For hyperparameter tuning, we refer readers to Appendix C.2.
6.1. Empirical Verification of Theoretical Results
ORRho-CLIP is more data-efficient. Corollary 4.3 im- plies that DRRho requires less amount of training data than learning without reference model to achieve the same level of excess risk. To empirically verify this theory, we run DRRho-CLIP on different portions of the training dataset (100%, 75% and 50%) and compare with the baseline FastCLIP (Wei et al., 2024) on the whole training dataset (100%). Multiple target reference model combination are tested, as specified in the following table.
DRRho-CLIP has lower variance of RHO loss. From Corollary 4.2 and the remark below know that the main difference between the excess risk bound of DRO and that of DRRho lies in the variance term (Var(E(O,..)) -)) for DRO and Var(f(e.,,) ter understanding of the two terms, we train a ViT-B/16 target model on DFN-12M with 320M samples seen using FastCLIP and DRRho-CLIP with a reference model ViT- B/32 (WIT-400M). Then we select a 200K subset from the training data and compute the variance of the original pair- wise loss in FastCLIP and the RHO loss in DRRho-CLIP w.r.t. each image and text. We use the trained model for computing the variance of the RHO loss and the original.
((Orett-)) for DRRho). To have bet-
Target Model (Data)
Reference Model (Data)
VYT-B/32 (CC12M)
ViT-B/32 (WTT-400M)
VIT-B/16 (DFN-12M)
ViT-B/32 (WTT-400M)
VIT- B/16 (2FN-12M)
VIT-B/16 (DFN-9M)
7

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Table I: Comparison of different methods on DFN-192M with 1.28B samples seen. Reference denotes the performance of the reference model. OpenCLIP and FastCLIP does not leverage a reference model. For distillation-aatl methods, the reference model is also the teacher model for distillation.
Reference Model ViT-B/16 (DFN-2B)
Metric
Method
ViT-B/32 (WTT-400M)
ViT-L/14 (DFN-2B)
Reference
63.32
76,23
81.41
OpenCLIP
66.94
66.94
66.94
FastCLIP
67.37
67.37
67.37
JEST
56.40
56.56
55.96
ImageNet Top 1
JEST (Top-k)
57.75
57.34
56.96
DRRho-CLIP CLIP
68.84
69.19
68.69
MobileCLIP (w/ Distillation)
66.94
68.67
68.47
FastCLIP (w/ Distillation)
67.33
69.15
68.85
DRRho-CLID (w/ Distillation)
68.84
69.50
69.25
Reference
52.27
60.75
66.65
OpenCLIP
54.58
54.58
54.58
FastCLIP
54.69
54.69
54.69
JEST
48.25
48.97
48.27
Datacomp
JEST (Top-k)
48.26
49.22
48.67
DRRho-CLIN
55.20
55.48
54.74
MobileCLiP (w/ Distillation)
54.58
55.21
55.31
FastCLIP (w/ Distillation)
54.69
55.60
55.91
DRRho-CLIP (w/ Distillation)
55.20
57.17
56.29
Loss Variance (x10 (x10-9)
times larger super-batch.
Ref. Model
DC
Image
Text
For knowledge distillation, compare with Mobile-
No Ref.
7.26 (0.58)
7,02 (0.91)
38.57
CLIP (Vasu et al., 2024), which optimizes:
ViT-B/32 (WTT-400M)
4.49 (0.54)
4.09 (0.60)
43.02
BE min (1 - Con (0) Xl (0, Oret, B). (12)
We report the mean and standard deviation of the variance above, along with the Datacomp (DC) average performance of both methods. The results show that with a reference model, the variance of the RHO loss lower than that of the original loss.
hyperparameter, Lcon denotes mini- batch contrastive loss and Caist the distillation loss be- tween the target model and reference model (cf. Equa- tion (16) in Appendix C). To demonstrate the benefit of our approach integrated with knowledge distillation, we re- place the contrastive loss above with our DRRho contrastive loss, which referred to as DRRho-CLIP (w/Distillation) For comparison, we also implement another baseline Fast- CLIP (w/Distillation) which uses the global contrastive loss instead of the mini-batch contrastive loss in MobileCLIP.
where [0,
6.2. Comparison with Baselines
In this section, we compare our method with other baselines of learning with reference models and knowledge distilla- tion. For the former, we compare with JEST (Evans al., 2024a), which is heuristic approach of applying the RHO loss for data sampling. In particular, they first sample image- text pairs from a large batch according to their similarities, and then compute an averaged RHO loss for each remaining data using the selected data in previous step as negative data and then perform sampling. We also implement another vari- ant that chooses the top pairs in the remaining data based on their averaged RHO loss in the second step, which is referred to as JEST (Top-k). Then a mini-batch contrastive loss is computed based on the selected data for updating the model parameter. These methods are implemented to select the same size of mini-batch samples as our method from a
We present the results on DFN-192M in Table (results on DFN- are presented in Table 3 in Appendix C due to space limit). All methods reported train a target model of ViT-B/16 from scratch. From the results we arrive at following conclusions: (1) Compared with JEST and JEST (Top-k), our approach achieves much better performance. This shows that theory-driven approach is better than the heuristic approaches. (2) When the reference model is relatively weak, e.g., ViT-B/32 (WIT-400M), DRRho- CLIP achieves better performance (68. 66% ImageNet Top-1 accuracy) than MobileCLIP (66.94% and the reference model itself (cf. Figure 4); (3) when the reference models

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Refarence
Reference
Referance
2022) these features filter low-quality data). Nev- ertheless, present the results of taking the compute of the reference model into consideration in Appendix C.5, where we observe similar trend.
70
ImagaNet-11 Accuney
68
66
7. Conclusion
64
In this paper, we have presented novel learning paradigm of model steering, which uses reference model to guide the training. Different from other heuristic approaches for learning with a reference model, our framework is grounded on distributionally robust optimization theory. We provided generalization analysis for our framework to analyze the improved generalization and data efficiency of our method. We applied our approach to CLIP training and proposed DRRho-CLIP, Experiments of DRRho-CLIP ified the theory but also demonstrated its superiority over heuristic approaches and better scaling law than that of the state-of-the-ar CLIP training method.
62
60
OpenAi CLIP
DFN
DRR
Figure 4: Zero-shot Top Accuracy on ImageNet-IK: of different models. DFN model was trained on DFN-192M dataset with 1.28B samples seen with batch size 8192 (Fang al., 2024), DRRho- CLIP model was trained under the same setting with batch size 5120, and using OpenAl CLIP as reference model.
only ver-
are strong, e.g., ViT-B/16 (DFN-2B), ViT-L/14 (DFN-2B), DRRho-CLIP (w/ Distillation) achieves the best result.
Acknowledgments
6.3. Scaling Law
We thank anonymous reviewers for constructive comments. This work used GPU resources at TAMU ACES and NCSA Delta through allocation CIS230245 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #21 #8138307, #2 13763,,, and #2138296. XW and TY were partially supported by National Science Foundation Award #2306572 and #2147253, National Institutes of Health FY and FS were partially sup-
Finally, we study the scaling law of ORRho-CLIB running with a batch size 5120 on 8 H100 GPUs and using ViT- B/32 (WTT-400M) as the reference model. Similar to Cherti al. (2023), we aim at uncovering the scaling law in the following form: E = S where E denotes the ImageNet error rate, C' denotes the amount of compute (GFLOPs), a,3 3 are real numbers that need to be determined. We choose number of samples seen 7' = 0.32, 0.64, 1.28, 2.56B for our experiments, and use ViT-B/32, ViT-B/16 and ViT-L/14 as the target model. We follow the open_elip repository and use its calculated amount of compute for each model?.
Award #RO1HL1681 ported by National Science Foundation Award #2326495 and #2247060, and Lilly Endowment, Inc., through its sup- port for the Indiana University Pervasive Technology Insti- tute.
To estimate E for each C, run method with ViT-B/16 on datasets of varying sizes from 144M to 624M, which are subsets selected from DFN-2B (Fang et al., 2024), and use the lowest error rate among different dataset sizes to estimate
Impact Statement
E. Next, we run our method with ViT-B/32 and ViT-L/14 on datasets of the same sizes as in ViT-B/16 training. Then fit the power law with different C' and corresponding E. We repeat the same procedure for OpenCLIP. We plot the relationship log E' log 3 log C' in Figure 2 for both DRRho CLIP and OpenCLIP. We observe that DRRho- CLIP has a better scaling law than OpenCIIP with smaller 3. This also demonstrates that our theory has practical im- plications. We would like to note that Figure 2 we do account for the compute of the reference model (mostly for obtaining image and text features), since these features are usually already computed and stored for other purposes and can thus be leveraged free of charge (e.g., Common- Pool (Gadre et al., 2023) and LAION-SB (Schuhmann et al.,
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity based data pruning with small reference mod- els. arXiv preprint arXiv:2405.20541, 2024.
Boucheron, S., Bousquet, O., and Lugosi, G. Theory of classification: survey of some recent advances. ESAIM: probability and statistics, 9:323-375, 2005.

---

Model Stecring: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3558-3568, June 2021.
European Conference, Milan, Italy, September 29-0c-
tober 2024, Proceedings, Part XVII, pp. 264-280, Berlin, Heidelberg, 2024b. Springer- ISBN 978- 3-031-72642-22 doi: 1011077/778---011172655...1 URL htttt::://dci..o/1101.000000 966439916..
Chen, T. Goodfellow, I., and Shlens, J. Net2net: Accel- crating learning via knowledge transfer. In The Fourth International Conference on Learning Representations, 2016.
Evans, T., Pathak, S., Merzic, H., Schwarz, J., Tanno, R., and Hénaff, O. J. Bad students make great teachers: Active learning accelerates large-scale visual understand- ing. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.). Computer Vision ECCV 2024, pp. 264-280, Cham, 2025. Springer Nature Switzerland. ISBN 978-9-031--2244399
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for con- trastive language-image learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. (818-2829, June 2023.
Fang, A., Jose, A. M., Jain, A., Schmidt, L.., Toshev, A. T., and Shankar, V. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024. URL https:: hetps///pperevveeee http///pppeeeeeeeeeeeeeeeeeeeeee net/ net.o//wwee//tttt forum? Add-Akknng (AAk6n.Z
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang. N., Tzeng, F., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1533 2013. URL http://arxiv.co abs/1310. 1531.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smymis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., Orgad, E., Entezari, R., Daras, G., Pratt. S., Ramanujan, V., Bitton, Y., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishna, R., Koh, P. W. W., Saukh, O., Ratner, A. J., Song, S., Hajishirzi, H., Farhadi, A., Beaumont, R., Oh, S., Dimakis, A., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, L. Datacomp: In search of the next generation of multimodal datasets. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27092-27112. Curran Associates, Inc., 2023. URL https: /ttp:///ecccccc...///o/. .ew.eeeeedddnnnn..nn......... neurips. cc/paper_ httppeerfiileespppee//eeee hiapper__iiess/pppe//////////
Du, W., Luo, T., Qiu, Z., Huang, Z. Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: closer look at model growth for efficient LLM pre- training. The thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: // /openreview net http://ooommmiaxXdeeeedeade
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Duchi, J. and Namkoong, H. Variance-based regularization with convex objectives. arXiv preprint arXiv:1610.02581, 2016.
563224id55.77ac60444666633
Paper Dat---Daaassssee Battasets_and_Bencnneett .df.
Duchi, J. C., Glynn, P. W., and Namkoong, H. Statistics of robust optimization: A generalized empirical likeli- hood approach. Mathematics of Operations Research, 46(3):946-66, 2021. doi: 10. 10.1287/moor.2020 1085. URL https: htttps://doi.org/ 10. 128. /moor........ .00.orr..00............... 1085.
Hinton, G. Distilling the knowledge in a neural network.
arXiv preprint 1033:115
Hu, Q.. Zhu, D., and Yang. T. Non-smooth weakly-convex finite-sum coupled compositional optimization. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http: httt:://pppprs... ccpps.ccppapp wwwer httpppparssaeeeppprreeeaaaa /hash/ 116799eeab11e222aeffeeeeee Abstract-Confereneen html.
Evans, T., Parthasarathy, N., Merzic, H., and Henaff, O. J. Data curation via joint example selection further accelerates multimodal learning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL https: // //openreview net/ / formm ww=mmmmdddxxxaxpx.p0000
Evans, T., Pathak, S., Merzic, H., Schwarz, J., Tanno, R., and Hénaff, O. J. Bad students make great teach- ers: Active learning accelerates large-scale visual un- derstanding. In Computer Vision ECCV 2024: 18th
Kullback, S. Information theory and statistics. Courier Corporation, 1997.
10

---

Model Stecring: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Levy, D., Carmon, Y., Duchi, J. C., and Sidford, A. Large-scale methods for distributionally robust optimization. In Larochelle, H., Ranzato, M., Had- sell, R.. Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 8847-8860. Curran Associates, Inc., 2020. URL https::/prooceddingg meerr neurips.
Qiu, Z.-H., Hu, Q.. Yuan, Z., Zhou, D., Zhang, L.,, and Yang, T. Not all semantics created equal: Contrastive self-supervised learning with automatic temperature in- dividualization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 23389-28421, PMLR, 23-29 Jul 2023. URL httt:/tt:://// wtt.https./pproeeednii /ttp///ecccceegg..ee/eeee/. ml press/ v202 v202/qiu23a. html.
paf.r.pdf.
Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: h //openrevien http:/neee//rrrrreeee/// 1d=
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language su- pervision. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18-24 Jul 2021.
Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadace, M., and Hooker, S. When less is more: Investigating data pruning for pretraining iims scale. arXiv preprint arXiv:2309. 2409.0464, 04564,
mlr / 1339/radoord2la.. html..
2023.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Mindermann. S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., and Gal, Y. Prioritized train- ing on points that are learnable, worth learning, and not yet learnt. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1 15630-1564) PMLR, 17-23 Jul 2022. URL https:::: //tp:::///rreeeeee ppwooeeeiingg..... wwweeeiinnnsmmmmm m....p.es////. v1 62 /mi nnderann2222 html.
Schuhmann, C., Beaumont, R. Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informatinn Processing Systems, volume 35, pp. 25278-25294, Curran Associates, Inc., 2022. URL cc/ hc//ppppe_fiileeepppprr///2//// neurips. a a1 a.3560ss133330004434430000330303 pdf.
Qi Q Lyu, J., Chan, K., Bai, E., and Yang, T. Stochas- tic constrained DRO with a complexity independent of sample size. Trans. Mach. Learn. Res., 2023, 2023a. URL hnttps://oope http://eereveeee.eeeooommm ner/forum? id id=VpaXrBFY29
Qi Q. Xu, Y., Yin, W., Jin, R., and Yang, T. Attentional- biased stochastic gradient descent. Trans. Mach. Learn. Res., 2023, 2023b. URL https:/ rew.eeeeee...eeeeeeeeeee..e..w.e net/ fott/foru?iin d-d= =033000VV222...
Doper-DDttastts__dd_Beeedddi
van der Vaart, A. W. and Wellner, J. A. Weak Conver- gence, pp. 16-28. Springer New York, New York, NY, 1996. ISBN 978-14455-2255-- 978- 178-1-4757-2545-222 URL https: org/1..000.....
doi: 10.1007/
Qin, Y., Lin. Y. Yi, J., Zhang, J., Han, X., Zhang, Z., Su, Y., Liu, Z., Li, P., Sun, M., and Zhou, J. Knowl- edge inheritance for pre-trained language models. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, pp. 3921 3937, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. (0.18653/v1/2022. naacl- 288. URL https: / aalaannnhhhaa anlahhohhooyyy /rgg 2022. wwa.caaal........ 1-m in 288/.
Vasu, P. K. A., Pouransari, H., Faghri, F, Vemulapalli, R., and Tuzel, O. Mobileclip: Fast image-text models through multi modal reinforced training. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15963-15974, June 2024.
Wang. B. and Yang, T. ALEXR: optimal single-loop al- gorithms for convex finite-sum coupled compositional

---

Model Steering:
Learning with Reference Model Improves Generalization Bounds and Scaling Laws
stochastic optimization. CoRR, abs/2312.022777, 2023. doi: 1048550APPXIV251200077 URL https:// doi wwg/10.gg1..oo 48550 www.////.....3333. 02277.
Wang, P., Panda, R., Hennigen, L. T., Greengard, P., Kar- linsky, L., Feris, R., Cox, D. D., Wang. Z., and Kim, Y. Learning grow pretrained models for efficient trans- former training. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.neeeeeeeeee www.eeeeieeeeeeeooorire.. id=iiddssciiiii 6f.
Wei, X., Ye, F., Yonay, O., Chen, X., Sun, B., Tao, D., and Yang. T. Fastclip: A suite of optimization techniques to accelerate clip training with limited resources. arXiv preprint arXiv:2407.0144: 2024.
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang. P. S., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pretraining. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.
Curran Associates, Inc.,
2023. URL https://proceeddnngg neurips. cc cc/paper fft.ee//ppaper//2////ee/e///// ccbbcbee11333562255c222233333 Paper-Coofeerence.po
Yuan, Z., Wu, Y., Qiu, Z.H.,, Du X., Zhang, L., Zhou, D., and Yang. T. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25760-25782. PMLR, 17-23 Jul 2022. URL https::p/// httt::p://prroeeee /ttp///eoeeeiiecoooooc ngs. p.pess/////// vl 2162/yuan225 html.
12

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
A. Detailed Theoretical Analysis
In this section, we provide generalization analysis of our DRRho framework. Given a function class F and set X with samples, define
(o) sup N(F(x), Iloe); (lll))) =EX"
where F(x) = ((. : f F for x E 2", and
NV, = [NV E N there is an e-cover of size N of V w.r.t || 113.
Theorem A.1 (Theorem (Duchi & Namkoong, 2016)). Assume that f(.) E [Mo, Mi] with M M1 Mo for all
F. Let 8M2 /t. 2 log 12, and p 2 9t. Then with probability least 233 (F, (m) + 1)e
11
Mp +
2t
E[f(x)]
sup P.D&(PIPESE
Erlf(X)) +
+ 4
(13)
3 n
n
for all f/E. F. Defining the empirical minimizer
Ep Ep]f(X)) : - +
f E arg min fEF
sup
P
we have with the same probability that
2p E[fl + 2 Var(f) n
19Mp
21
Eifl
< inf JEF
+
+
2 + 41
c.
(14)
3n
Corollary A.2 (Corollary 3.1 in (Duchi Namkoong, 2016)). In addition the conditions of Theorem A. let F have finite VC-dimension VC(F). Then for numerical constant oc, the bounds in Theorem A.l hold with probability at least
1 -
(VC(F)
(6MMe))
VC(F)-1
+2 e-
Theorem A.3. Under the conditions of Theorem A.I, let dy denote the VC-dimension of O and 3o denote a constant.
1. For any 0 E θ, let t = log & + log c + d, log 16e + 2d, log and p 2 96 for d > 0, then with probability at least
8,
R(O) + 2/ 2p Var(e(0,)))) +
25p
M
R(0) < inf BEe
+
3
Ep(e(0, (PD(()
sup
min oE
sup (:D,(PIP))
Er(e(0,)))
2. For the solution 0
min 6-0
sup P.D.(P
Ep((((,)))
(15)
let t = log & + +ogg c d, log 16e + 2d, log n and 2 9t for > 0, then with probability at least - 8,
2p Var(e(0.)))
25p 3
M
R(00) inf
R(0) + 2/
+
+
2
13

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Proof. We first prove the first part of the theorem, which follows the proof of Theorem 3 in (Duchi & Namkoong, 2016).
From Theorem A.1 we know for any F, with probability least 1 2(3N (F, 2n) 1)e
[r(s(x)] 11 Mp
21
[[((x)
sup P.D,(PIIP)SE
2 + 4V
7
Ep[f(X)] 11 Mp 3 n
2f
-min
sup
2 2 4
fEF P.Do(PIP))S
+ sup
Ep[s(X)] min P.Dx(PIIP)EH
sup jeF P:D,
Ep Ep[f(X)].
Following the proof of Theorem 3 in (Duchi & Namkoong, 2016), setting M , log $ +log c de log 16c 2d, log we know for any f E O, with p 2 9/ and probability at least 2(3N2o (F, 2n) + +1e-
12 Var(E(O,,)) + (5pp + M
R(0)= S inf BEB
R(0)) + 2
3
sup (PD.(PI)))
Ep Ep(e(0,
min P.Dx(PIP)sE
sup
Ep(e(0,-))
From Corollary A.2 we know the probability is least
16Mme
VC(F)-1
1 cV((F)
for some < cc. Let d, denote the VC-dimension of F, then we know the probability is at least
1 (edo (16n²e) ) exp(-t)
=1 exp (log c log du + (dg 1)(l 16e + 2 llg exp(-t)
21 exp (log du log 16e + 2d, log - t)
Since / log + log e + d, log 16e + 2d, log n, we know that with p 2 9t and probability at least 8,
2p
25p
M
R(0) < inf AEB
R(0) + 2 Var(((e,-)) +
+
3
+
sup PPDI(S
Ep(((O, )]
min GEO
sup P.D.(PI))
Erl((0,))
This completes the proof for the first part. The second part then follows from the fact that 0* is the solution of Equation (15). This completes the proof.
[
Proof of Theorem 4.1. Since Ô, minimizes Equation (3), applying Theorem A.3 with f()) E(0,-) - -E(O,or .) (where the range becomes 2M), letting log 1 + log e + dy log 16e 2d, log and p 2 9t for given å > 0 and constant oo, then with probability at least 1 - 8,
2p R(0) R(Oret) + V Var(e(0..) - e(Oret:))
50p 3
M
R(O.) R(Orat) < inf aee,
+
+4
which is
2p R(0) + V Var(E(0,) - E(Orer,)))
55pp
R(O.) < inf o
+
+
This completes the proof.
14

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Proof of Corollary 4.2. From the definition we have 0_ E + Thus know
2p Var(e(0,-) ((Orot: < R((0)) +
2p Var Var(e((..)) E(Oret: -))
inf
R(0)+
n
Plugging the above inequality into Theorem 4.1, get
2p 2 Var(e((...) E(Orof, .)) C2
R(OO)= S R(O.)+ + V
This completes the proof.
Proof of Corollary 4.3. Since Oref O, we know
/ 12p 2 Var(e(O,,)) E(Orot :)) < R(Orot) V
2p Varle(Oret: E(Oref, -)) R(Orot).
inf OEe.
R(()-
V
Plugging the above inequality into Theorem 4.1, and subtracting R(O.) on both sides, we get
R(O.) R(O.) S R((ret)) R((0)= C2
n
This completes the proof.
B. Connection between DRRho and Existing Methods
In Section 4 we show that our proposed framework can be applied to data selection, weighting and sampling. Here we provide a comparison between our framework and existing methods.
Data Selection: With the CVaR divergence, the DRRho objective becomes the average of top- RHO losses (Equation (4)). This is connected to RHO (Mindermann et al., 2022) and RHO-1 (Lin et al., 2024). The key idea of both RHO and RHO-1 both to first sample a large batch of data points, then leverage reference model to compute the RHO loss for each sample in the large batch, and in the end only samples with the largest RHO loss values are kept for back-propagation. The major difference between framework and the two existing works is that our objective will select top samples in the whole dataser for back propaaropag,tion, while existing works operate in batch level.
Data Weighting/ / Sampling: With the KL.-divergence, our framework becomes a data weighting method that assigns different weights to different samples. In the CLIP training setting, data weighting is applied to every combination of image and text. Evans et al. (2025) applied the idea for data sampling and proposed ActiveCLIP. Specifically, given a large batch of data points, they first compute the RHO loss using a reference model for each sample in the batch, then they pass the loss scores through softmax function and view Evans et al. (2024a) builds upon the idea of ActiveCLIP and proposed JEST, which leverages the RHO loss as well but use fine-graine nne-grained sampling approach due to the special structure of the contrastive loss. Note that both ActiveCLIP and JEST are proposed for training CLIP models, which is the same as our DRRho-CLIP. The core difference between our DRRho-CLIP and ActiveCLIP and JEST that their losses are used for data selection of anchor data while ours has an effect of data re-weighting of the negative data for each anchor data. Moreover, their methods use mini-batch contrastive loss define RHO loss. While we define the loss using all negative data in the training dataset (instead of the mini-batch) and use rigorous optimization approach (SogCLR) to optimize the objective.
probability distribution sample small batch for back- back-propagation.
C. More Experiment Results
C.1. Implementation Details
Our implementation is based on FastCLIP (Wei et al., 2024), which includes the implementation of FastCLIP and OpenCLIP. We leveraged the code provided by Evans et al. (2024a) with minor modification for our implementation JEST and JEST (Top-k). We merge the code released by Vasu et al. (2024) into our code base for the implementation of MobileCI.IP.
15

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
C.2. Hyperparameters
DRRhO-CLIP. We set the learning rate to 3. 3.125e-4 for training ViT-B/16, and set the learning rate 8e-4 for training ViT-B/32. In most experiments in Section 6, set the temperature to 0.01. The only exception is the with reference model trained on DFN-9M where the temperature is learnable (ef. Appendix C.3) and the initial temperature is set 0.07.
Baselines: OpenCLIP and FastCLIP We use the FastCLIP-v3 from Wei et al. (2024) as the FastCLIP baseline. For both OpenCLIP and FastCLIP, we set the learning rate to 3. 155e-- for training ViT-B/16, and set the learning rate to 8e-4 for training ViT-B/32. We set the initial temperature to 0.07. For FastCLIP, we set the learning rate of be 1/4 the learning rate of the model, we set p to 11.0 for training ViT-B/16 and 8.5 for training ViT-B/32.
JEST and JEST (Top-k) We set the learning rate 3.1 125e-4 for training ViT-B/16. Following Evans et al. (2024a), the selection ratio is set to 0.2, which means only 20% of the super-bates batch will be used for training. The size of the super-batch at each iteration is 25600 and the size of the mini- batch for training is 5120, which is the same as other methods. We set the number of chunks to 2 so that the chunk size of 2560, which is close to the value 2048 suggested by Evans et al. (2024a). At selection ratio 0.2, JEST and JEST (Top-k) spends times more compute on forward propagation (and the same amount of compute on backward propagation) compared with other approaches. Since the amount of compute of forward propagation approximately 1/3 of that of backward propagation (Evans al., 2024a), we increase the number of iterations to 1.87 times of that of other approaches so that these methods consume the amount of compute.
MobileCLIP, FastCLIP with Distillation, DRRho-CLIP with Distillation We set the learning rate to 3. 3.125e-4 for training ViT-B/16. We set the value of À of DRRho-CLIP with Distillation to be the same as that of FastCLIP with Distillation. On DFN-12M, for MobileCLIP, we set the value of À to 0.4; for FastCLIP with Distillation, we set the value of À to 0.25, On 192M, for MobileCLIP, we set the value of À to 0.75; for FastCLIP with Distillation, we set the value of À to 0.25. On DFN-192M with reference model ViT-B/32 (WIT-400M), since the performance of the reference model low, the value set to 0.0 (i.e., no distillation) for all three methods.
C.3. Additinnal Ablation Study
Comparison between Fixed and Learnable Temperature т. most our experiments DRRho-CLIP, we set the temperature 7 to 0.01 and fix it throughout training. The only exception is the one with reference model trained on DFN-9M where set to a learnable hyperparameter in FastCLIP(-v33), which gives better performance than using fixed temperature. To further study this, we conduct experiments to compare the performance of learnable temperature and fixed temperature. Mathematically, the loss formulation with learnable temperature is similar to Equations (8) and (9) except that the temperature now becomes parameter that can be optimized:
((0) Orof, Di, wil) + log exp(
( Pref, Vi. ;;)) + 2TP, exp(
min
T log
mm1
where > O is a hyperparameter. We conduct experiments of DRRho-CLIP with fixed and learnable temperature on different target model and reference model combinations. We present the results in Table 2 and Figure 5. We find that, for a reference model with high performance, using fixed temperature leads to slightly better performance than using learnable temperature. While for reference models with low performance, learnable temperature yields higher performance.
Table 2: ImageNet Top accuracy of DRRho-CLID with fixed and learnable temperature different target models and reference models.
ImageNet Top 1 Accuracy
Target Model (Data)
Reference Model (Data)
FastCLIP
DRRho-CLIP
DRRho-CLIP
Reference
(Learnable r)
(Fixed r)
ViT-B/16 12M)
VIT-B/32 (DFN-12M)
43.49
46.31
37.65
36.27
VIT-B/16 (DFN-12M)
VIT-B/32 (WIT-400M)
43.49
49.81
49.91
63.32
ViT-B/16 (DFN-1922)
VIT-B/32 (WIT-400M)
67.37
68.17
68.84
63.32
16

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
DRRho-CLIP
46
Imagehee 1K Aocuraay (N)
65
ImageN 1K Aocuraay
Imagelie- 10--1AAAn Acnumaay
40-
60
30
55
26-
20
25-
60
20
45
100
150
200
250
300
50
100
150
200
250
300
0.2
0.4
0.6
0.8
1.0
12
Samples Seen (iillio))
Samples Seen (millio)
Samples Seen (aillion)
(a) Target: VIT-B/16 (DFN-12M)
(b) Target:
VIT-B/16 IDFN-12M
(c) Target: ViT-B/16 (DFN-192M)
Reference: VIT-B/32 DFN-12M)
Reference:
VIT-B/33 (WIT-400M)
Reference: VIT-B/32 (WIT-400M)
Figure 5: ImageNet Top accuracy curves of DRRho- CLIP with fixed and learnable temperature different target models and reference models.
C.4. Comparison with Baselines
The distillation loss we consider is the same as in Vasu et al. (2024), and defined as follows
exp(s(x;, Uj)/f)
exp(s(x;.U;)1
Ldistill (0, Oret,
log
B
KEB exp @ 2 //7/F)
kEB exp(s ((pps 2x.,,,3)/77 3x)/T)
iEB JEB
(16)
exp ( v y1 /1)
exp(s(2), 9:)/T)
log
Eke exp ( //F
Σ exp ( y 7T)
iEB
JEB
where sn,; Uj) (8(2;; Uj). resp.) denotes the cosine similarity between i-th image and j-th text output by the target model (reference model, resp.).
In Table 3, we present the results of different baselines on DFN-12M with 320M samples seen.
In the following figure, we plot the ImageNet-11K Top accuracy of different models trained using OpenCLIP DRRho- CLIP.
Reference
w/o Reference
w/ Reference
I Top-1 Accuracy (%)
80
ImageNet-1K Top-1 Accurscy (%)
72.5
75
70.0
70
67.5
65.0
65
62.5
60.0
60
Reference VIT-B/16
OpenCLIP ViT-8/16 DFN-182M
DRRho-CLIP VIT-8/16 DFN-192M
Reference VIT-U144
OpenCLIP VIT-B/16 DFN-192M
DRRho-CLL CLIP VT-B/16 DFN-192M
DFN-2B
DFN-2B
Figure 6: Zero-shot Top Accuracy on ImageNet-Iee IK of different models. Reference model is ViT-B/16 (left figure) or ViT-L/14 (right figure) pretrained on DFN-2B dataset with 12.8B samples seen and a batch size of 901 12 (Fang et al., 2024); OpenCLIP model a ViT-B/16 trained on DFN-199 dataset with 1.28B samples seen and a batch size of 8192 (Fang et al., 2024), DRRho-CLIP model was trained on DFN-192M, dataset with 1.28B samples seen and using the reference model with batch size of 5120. The latter use the same model structure.
17

---

Model Steering: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
Table 3: Comparison of different methods on DFN- with 320M samples seen. Reference denotes the performance of the reference model. OpenCLIP and FastCLIP does not leverage a reference model. For distillation- methods, the reference model is also the teacher model for distillation.
Reference Model ViT-B/I6 (DFN-2B)
Metric
Method
ViT-B/32 (WIT-400M)
ViT-L/14 (DFN-2B)
Reference
63.32
76.23
81.41
OpenCI.IP
41.10
41.10
41.10
FastCLIP
43,49
43,49
43.49
JEST
36.78
34.63
33.46
ImageNet Top
JEST (Top-k)
36.30
34.75
33.38
DRRho-CLIP
49.95
47.57
46.70
MobileCLIP (w/ Distillation)
52.77
48.01
45.80
FastCLIP (w/ Distillation)
53.21
50.38
47.86
DRRho-CLIP (w/ Distillation)
53.33
52.58
49.84
Reference
52.27
60.75
66.65
OpenCLIP
37.67
37.67
37.67
FastCLIP
38.57
38.57
38.57
JEST
35,59
35.42
34.36
Datacomp
JEST (Top-k)
34.60
34.56
34.88
DRRho-CLIP
43.02
42.15
41.47
MobileCLIP
44.57
41.68
39.81
FastCLIP (w/ Distillation)
44.57
43.72
42.04
DRRho-CLAN (w/ Distillation)
46.29
43.88
42.49
C.5. Scaling Law
In Figure 7 we plot the relationship log E = log + log C' for DRRho-CLIP and OpenCLIP, where E' denotes the ImageNet error rate and C' denotes the combined amount of compute used by the target model and the reference model. The main difference between Figure and Figure the latter does not. Similar Figure 2, we also observer that DRRho-CLIP has a better scaling law than OpenCLIP with smaller 3.
that the former considers the compute used by the reference model while
18

---

Model Stecring: Learning with Reference Model Improves Generalization Bounds and Scaling Laws
VIT-B/32
VIT-B/16
VT--/14
OpenCLIP
DRRho-CLIP
ImageNet-tn 1K Error
30
-
E E 8.8.78- 8-788---018
10
Compute (GFLOPSX Samples Seen)
Figure 7: Scaling performance of OpenCLIP (Cherti et al., 2023) and the proposed DRRho-CLIE which uses the OpenAI CLIP model (Radford al., 2021) as the reference model. We conduct experiments of the two methods under different settings fit scaling laws, shown in the bottom left corner (c.f. Section for more detail).
19


Reviews:
[{"review_id": "XcMdYLc1gc", "reviewer": "Reviewer_uJqj", "summary": "Authors present a framework for using available open weights model to improve model training on given dataset (learning with a reference model - LEAR). The framework is based on distributionally robust optimization (DRO). DRO makes use of available data empirical distribution to create perturbed data distributions and use those for worst case risk minimization. Author further employ RHO loss, a generic loss aiming on identifying data points worth learning, to obtain a risk function via applying DRO (DRRHO risk)  They derive theoretical generalization bounds using the risk, aiming to explain how DRRHO improves generalization. They study their method on example of CLIP training, using pre-trained CLIP models as reference. Obtained DRRho CLIP (using various reference models) is compared to various baselines, stating its advantages in downstream task performance and data efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestion on experiments. We believe that the comments raised by the reviewer are not critical drawbacks of this paper. We request the reviewer to consider our contribution in terms of theoretical framework and analysis, and our experiments comparison with multiple baselines. \n\n> **Q1**:  Why was the derivation of full scaling laws (e.g. following Cherti et al. 2023) was omitted in the work?\n\n**A**: Thank you for raising this concern. We would like to note that Cherti et al. (2023) focused on reproducing CLIP, which is empirical only. In contrast, we have rigorous theoretical analysis of the generalization error, which is not restricted to any model or data scale. Our experiments include the comparison with multiple baselines in the context of learning with reference models and our scaling law experiment serve to corroborate the presented theory.\n\nFollowing the reviewer's suggestion, we have added scaling law experiments for two more model scales (ViT-B/32 and ViT-L/14). The experiments have not completely finished since they take many days to run on large scales, but we can already observe that DRRho-CLIP has a better scaling trend across different model scales than OpenCLIP.\n- FLOPs vs. performance, without accounting for reference model cost: [anonymous link](https://github.com/icml2025drrhoclip/icml2025drrhoclip/blob/main/scaling_law_flops.pdf). The fitted scaling law for OpenCLIP is $E=5.77\\cdot C^{-0.116}$, where $E$ is the ImageNet-1K Error and $C$ is the compute (GFLOPS). The fitted scaling law for DRRho-CLIP is $E=7.15\\cdot C^{-0.127}$.\n- FLOPs vs. performance, accounting for reference model cost: [anonymous link](https://github.com/icml2025drrhoclip/icml2025drrhoclip/blob/main/scaling_law_flops_reference.pdf). The fitted scaling law for OpenCLIP is $E=5.77\\cdot C^{-0.116}$, while for DRRho-CLIP it is $E=8.78\\cdot C^{-0.135}$.\n\n> **Q2**: It would be also good to plot FLOPs vs. performance, accounting for FLOPs used by reference model.\n\n**A**: Links to the plots are provided in the answer to the above question. We want to highlight that the cost of leveraging a reference model can be amortized. In particular, since the reference model is frozen during training, we store the reference model features before training so that they can be reused multiple times across epochs and across different runs. Indeed, these features of the reference model have been already computed in the data filtering stage for creating existing datasets, e.g., LAION-2B (Schuhmann et al., 2022), DFN-2B (Fang et al., 2024) and Datacomp-1B (Gadre et al., 2023). In this case, we can directly leverage the features of the reference model without spending any resources computing them.", "claims_and_evidence": "To test their claims of enhancing generalization via training with reference model, authors perform CLIP training guided by various reference models on various scales and measure model performance via various well-established benchmarks. I think authors' approach is valid and evidence they gather is sufficient to argue for their procedure being useful. Scaling law derivation claim seems overblown. Authors present for single fixed model scale (ViT B/16) measurements across 4 different samples seen scales, through which they fit a power law. This is not what is known as full scaling law. Eg cited work by Cherti et al performs measurements on combinations of 5 different model scales, 3 different samples seen scales (3B, 12B and 34B) and 3 different data scales (80M, 400M and 2B), each of which is tuned to construct a Pareto front corresponding to minimum error on downstream tasks. This full scaling law derivation is not what authors perform and thus only limited (if any) conclusions can be drawn from the Fig 3.", "methods_and_evaluation_criteria": "Authors derived theoretical generalization bounds which they use to backup various heuristics for data filtering / selection that lead to training of models with stronger generalization. They enhance standard contrastive loss of CLIP with DRO loss component that includes reference model, conducting training of DRRHO-CLIP. Datasets used are well established CC12M and DFN subsets (DFN-9M and DFN-128M). Evaluation used well established CLIP benchmarks, eg those used in DataComp, which are used to compare DRRHO-CLIP with other strong reference baselines. Methods and evaluation make sense.", "theoretical_claims": "Theoretical generalization bounds obtained by authors seem correct, as well as introduction of the DRO loss into CLIP training.", "experimental_designs_or_analyses": "Experimental design of CLIP training with reference model seems sound. The derivation of scaling law is not executed properly, or stated differently, there is no scaling law presented in the study although the title strongly suggests so.", "relation_to_broader_scientific_literature": "The work fits well into landscape of language-vision learning research and distillation methods.", "essential_references_not_discussed": "Authors cite relevant works properly.", "other_strengths_and_weaknesses": "Strength of the paper is in establishing theoretical grounds for learning with reference model and testing the approach on important scenario of CLIP training, which often uses distillation as technique to boost model performance. Weakness is the missing scaling law that is announced in the paper title. Authors do not derive proper full scaling law, picking only one fixed model scale (ViT B/16). Based on this plot, authors attempt to make a conclusion how DRRHO CLIP compares to openCLIP, which does not work, as for such comparison full scaling law would have been necessary. Authors can only conclude that for ViT B/16 and rather small span of samples seen scales, DRRHO CLIP has advantage over openCLIP without reference model. It is not clear what happens across scales and on larger scales , eg L/14 , H/14, from this examination (full scaling law would allow prediction for those larger scales). Moreover, the compute necessary for using reference model (which has also to compute loss on the training data during training) is not incorporated into considerations for model comparison. In general thus, using total compute on x-axis for learning procedure comparison via scaling law derivation should have been the correct approach here if aiming for strong conclusions about advantage of the learning procedure over other baselines.", "other_comments_or_suggestions": "Text is well written and is easy to follow.\n\nUPDATE: raising score to 3 after scaling law extension experiments by the authors.", "questions_for_authors": "Scaling law derivation on smaller scales used in the work should have been possible, as those experiments are not expensive. Why was the derivation of full scaling laws (eg following Cherti et all 2023) was ommitted in the work? It would be also good to plot FLOPs vs performance, accounting for FLOPs used by reference model - can authors provide such a plot?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLvmpaQUoa", "reviewer": "Reviewer_DaNa", "summary": "The paper proposed DRRho risk minimization with a reference model and provided a theoretical analysis of it. It also applied this approach to training the CLIP model. Experiments show that the proposed method achieves better performance than the baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and positive evaluation of our work. We are happy to address any concerns the reviewer may have in later stage.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The datasets and benchmarks are appropriate and relevant to the problem. However, since I am not an expert on this topic, I’ll leave it to other reviewers to judge whether the baselines are comprehensive and the comparisons are sufficient.", "theoretical_claims": "I didn’t go through all the proofs in detail, but the theoretical results seem convincing to me.", "experimental_designs_or_analyses": "The experimental design is valid and demonstrates the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "The paper is related to the literature on distributionally robust optimization. Although the technique itself is well-established, combining it with a reference model and providing a theoretical analysis appears to be a novel contribution. Furthermore, applying it to training CLIP, a relatively large-scale problem with practical significance, seems like a solid contribution.", "essential_references_not_discussed": "I didn’t notice any.", "other_strengths_and_weaknesses": "Overall, I find the contribution solid, with both theoretical insights and experiments showing improvement over the baseline. However, I am not entirely sure about the comprehensiveness of the evaluation and comparison, so I would like to hear other reviewers’ opinions.", "other_comments_or_suggestions": "I don’t have other comments.", "questions_for_authors": "I don’t have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMDCwHFO3Q", "reviewer": "Reviewer_Mms9", "summary": "The paper establishes a theoretical framework for RHO-based learning with a reference model using DRO as the perspective and introduces a novel DRRho risk. It further applies DRRho-based LEAR to CLIP, achieving good and data-efficient performance.\n\n## update after rebuttal\nMy overall evaluation remains unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and suggestions.\n\n> **Q1**: More powerful reference model does not always yield superior target model performance. But Corollaries 4.2 and 4.3 show that a better reference model should lead to greater improvements in target model training.\n\n**A**: There is misunderstanding of results in Corollaries 4.2 and 4.3 regarding that more powerful reference model yields superior target model performance. Corollary 4.2 shows that the generalization depends on the variance of the RHO loss for a reference model. However, a powerful reference model does not necessarily have a small variance of the RHO loss, i.e., $\\mathrm{Var}(\\ell(\\theta_*, \\cdot) - \\ell(\\theta_\\mathrm{ref}, \\cdot))$, where $\\theta_*$ is the optimal solution in the considered model space. Corollary 4.3 only compares a reference model in **the same space** of the target model. However, in Table 1 except for ViT-B/16, other two reference models (ViT-B/32, ViT-L/14) are not in the same model space of the target model ViT-B/32. Hence, we cannot apply Corollary 4.3.\n\nWe would like to point out that this phenomenon is also empirically observed in other works that leverage a reference model, e.g. DFN (Fang et al., 2024), JEST (Evans et al. 2024a) and MobileCLIP (Vasu et al., 2024), where stronger reference models did not necessarily lead to more powerful target models.\n\nTo better verify Corollary 4.3,  we have conducted the following experiments with ViT-B/16 as target model and ViT-B/16 trained on different subsets of DFN-12M as reference models. We list the ImageNet-1K Top 1 Accuracy of both the target model and the reference model in the following table, from which we can observe that the performance of the target model and the reference model is positively correlated.\n\n| Target Model (Data, Samples Seen) | Reference Model | Target Model Performance | Reference Model Performance |\n| -- | -- | -- | -- |\n| ViT-B/16 (DFN-12M, 320M) | ViT-B/16 (DFN-6M, 320M) | 42.50 | 30.19 |\n| ViT-B/16 (DFN-12M, 320M) | ViT-B/16 (DFN-9M, 320M) | 46.80 | 39.09 |\n| ViT-B/16 (DFN-12M, 320M) | ViT-B/16 (DFN-12M, 320M) | 48.88 | 43.49 |\n\n\n> **Q2**: Regarding interpretation of Corollary 4.3 and do Theorem 4.1 or Corollary 4.2 still indicate $n=O(m)$ sample complexity? \n\n**A**: The interpretation of Corollary 4.3 about the reduced sample complexity is to guarantee that the generalization error of the learned model $R(\\tilde\\theta_*) - R(\\theta_*)$ by our framework is on par with that of the reference model $R(\\theta_{\\text{ref}}) - R(\\theta_*)$, where $m$ is the data size for training the reference model. If we want to use Corollary 4.2 for deriving $R(\\tilde\\theta_*) - R(\\theta_*)$ in the same order $1/\\sqrt{m}$ of $R(\\theta_{\\text{ref}}) - R(\\theta_*)$ , it will imply that $n = \\max(\\sqrt{m}, m\\mathrm{Var}(\\ell(\\theta_*, \\cdot) - \\ell(\\theta_\\mathrm{ref}, \\cdot)))$. It could be still much better than $O(m)$ as $\\mathrm{Var}(\\ell(\\theta_*, \\cdot) - \\ell(\\theta_\\mathrm{ref}, \\cdot))$ could be very small. In this context, we cannot ignore $\\mathrm{Var}(\\ell(\\theta_*, \\cdot) - \\ell(\\theta_\\mathrm{ref}, \\cdot))$. Hence, it is more convenient to use Corollary 4.3 to make this comparison argument. Thus the two results do not contradict to each other.", "claims_and_evidence": "The authors make several key claims:\n- The DRRho framework improves generalization via variance reduction. \n- DRRho-CLIP outperforms heuristic methods.\n- DRRho-CLIP is more data-efficient than vanilla ERM.\n\nOverall, the claims in the paper are well supported.", "methods_and_evaluation_criteria": "The DRO-based DRRho risk and its application to CLIP are well-motivated.", "theoretical_claims": "I didn’t check the detailed proofs in the Appendix.", "experimental_designs_or_analyses": "The experiments are well-designed and carefully analyzed to support their theoretical claims and the efficiency of their method.", "relation_to_broader_scientific_literature": "The paper builds on DRO (Duchi & Namkoong, 2016), contrastive learning (Qiu et al., 2023), and LEAR (Mindermann et al., 2022).", "essential_references_not_discussed": "I didn't notice any missing key references.", "other_strengths_and_weaknesses": "**Strengths**:\n- The paper is well-written and easy to follow.\n- The theoretical justification of RHO-based LEAR using DRO is creative and reasonable. \n- The analysis is comprehensive, supported by sufficient theoretical and experimental evidence.\n\n**Weaknesses**:  \nThe paper offers a potential theoretical explanation via generalization bounds for why LEAR is more data-efficient than ERM, and it is expected that a better reference model should lead to greater improvements in target model training (as shown in Corollaries 4.2 and 4.3). However, we can observe in Table 1 that a more powerful reference model does not always yield superior target model performance. \nIt would be beneficial if the authors could clarify and explain this discrepancy.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "I have a question regarding the interpretation of Corollary 4.3. The paper claims that \"DRRho needs only $n = O( \\sqrt{ m })$ samples, which dramatically reduces the sample complexity $O(m)$ of ERM without a reference model.\" However, when viewed through the lens of Theorem 4.1 or Corollary 4.2, it seems that the sample complexity would still be $O(m)$. Could the authors clarify this apparent contradiction in the sample complexity analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpiqS9UE4N", "reviewer": "Reviewer_26QA", "summary": "This paper theoretically studies the mechanism behind a learning paradigm, called Learning with a Reference Model (LEAR) and proposes a new learning algorithm that achieves better scaling than the naive approach. They first relate the RHO loss with DRO and show how using the RHO loss can improve the generalizability of DRO. They then show that when a certain divergence function is used, RHO+DRO is equivalent to weighting samples with RHO loss, which theoretically explains why LEAR is useful. To optimize RHO+DRO, they adopt the SogCLR algorithm. They show that their approach has better sample efficiency than the naive method and the distillation methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the technical merits of this paper. We will follow the reviewer's suggestions to improve the paper, which should be an easy task. \n\n> **Q1**: Would this method work for other tasks? Is there a reason to focus only on CLIP?\n\n**A**: Yes! We have discussed that the existing works of employing the RHO loss for data selection or sampling for different tasks can be considered as heuristic implementation of the proposed framework (Section 4). These different tasks include classification (Mindermann et al., 2022) and training LLMs (Lin et al. 2024). We have discussed the reason for our focus on CLIP at the beginning of Section 5.\n\n> **Q2**: How $C_2$ on the right hand side at line 733 is derived in the proof of Corollary 4.3.\n\n**A**: The constant $C_2$ in Corollary 4.3 is the same as that in Theorem 4.1. In Theorem 4.1, we derived $$R(\\tilde{\\theta}_*)\\leq \\underbrace{\\inf _{\\theta\\in \\Theta} \\left(R(\\theta) + \\sqrt{\\frac{2\\rho}{n} \\mathrm{Var}(\\ell(\\theta, \\cdot)- \\ell(\\theta _{\\mathrm{ref}}, \\cdot))}\\right)} _{\\mathrm{term1}}+ \\frac{C _{2}}{n},$$ where $C _2=(50\\rho/3+4)M$. In the proof of Corollary 4.3, we showed that term1 in the brace is upper bounded by $R(\\theta _\\mathrm{ref})$. Hence, $C_2$ is the same as in Theorem 4.1.\n\n> **Q3**: Does the proposed method optimize the same objective as (Evans et al., 2024a;b)? Comparing with their approach seems important.\n\n**A**: We would like to point out that we indeed compared with Evans et al., 2024a (JEST). We omitted the comparison with Evans et al., 2024b (ActiveCLIP) since JEST is a later work and improved over it by the same groups of authors. Our method does not optimize the same objective as their methods:\n- their methods use mini-batch contrastive loss to define a RHO loss. While we define the loss using all negative data in the training dataset (instead of the mini-batch) and use a rigorous optimization approach (SogCLR) to optimize the objective.\n- their loss is used for data selection of anchor data, i.e., selecting subset for training. In contrast, our DRRho contrastive loss, defined by leveraging the relationship between global contrastive loss and DRO, has an effect of data re-weighting of the negative data for each anchor data.\n\nMoreover, their methods consume more resources for data selection due to sampling from a larger batch. We compared the performance of JEST and DRRho-CLIP on DFN-12M and DFN-192M with fixed amount of compute. The results (c.f. Tables 1 and 3 in the paper) showed that DRRho-CLIP significantly outperforms JEST.\n\n> **Q4**: This paper doesn't seem to be able to explain all LEAR methods. It should be more specific about which LEAR method(s) this paper is for.\n\n**A**: LEAR refers to, as stated at the beginning of the abstract, *leveraging a pretrained model ... through strategic data selection or weighting*. Thus we focus on methods for data selection and weighting. We categorize existing works of leveraging a pretrained model into different families in Section 2, where in Lines 120-140 we provide background about the specific LEAR methods that motivates this work, such as RHO (Mindermann et al., 2022), RHO-1 (Lin et al., 2024), ActiveCLIP (Evans et al., 2024b) and JEST (Evans et al., 2024a) to make Section 4 clearer.\n\n> **Q5**: I would suggest that the authors provide some background about the specific LEAR methods this paper can provide insights into. This would make the purpose of Section 4 clearer.\n\n**A**: Thank you for the suggestion. We will give more background about the related LEAR methods mentioned in the above question to make Section 4 clearer.\n\n> **Q6**: While it's the convention to call scaling law scaling law, the word \"law\" is actually misleading, as it is not a real law. If possible, I would suggest that you just say you scale better.\n\n**A**: While we agree with the reviewer that the term \"law\" is misleading or sometimes overclaims, however, we hesitate to invent a new term as it has been widely accepted in the literature. Cherti et al. (2023) also used the term \"scaling law\" for CLIP training. \n\n> **Q7**: Does the x-axis of Figure 2 represent the training time? If not, why did you use different line to represent different ratio of data used?\n\n**A**: The x-axis in Figure 2 represents the number of samples seen, which is proportional to training time. Different ratio of data means the training dataset size is different. For example, 100% data means the whole dataset is used during training, while 50% data means only half of the dataset is used during training. For different datasets, we set the total number of samples seen to be the same, which means on smaller datasets there will be more epochs.\n\n> **Q8**: Why is the trend of Figure 2 different from Figure 3?\n\n**A**: In Figure 2, we use ImageNet-1K Top 1 Accuracy as the y-axis. While in Figure 3, we use ImageNet-1K Error as the y-axis, which is equal to 100% - ImageNet-1K Top 1 Accuracy. Thus the two figures have different trends.", "claims_and_evidence": "1. The author claims that RHO can achieve a smaller variance, and verify it empirically.\n2. The author claims that RHO+DRO can achieve a better sample complexity/scaling law. It is also verified.", "methods_and_evaluation_criteria": "They evaluate the algorithm by training CLIP models and evaluate ImageNet, which seems reasonable. However, a caveat is that they only experiment with this single task.", "theoretical_claims": "They don't seem to be incorrect.\n\nA caveat is that there are too many constants and it's hard to understand their meaning. As a result, I am not sure whether Corollary 4.3 is correct or non-trivial.", "experimental_designs_or_analyses": "No crucial problems found.", "relation_to_broader_scientific_literature": "Based on the paper, this work is related to DRO papers and provides the theoretical foundation of [1]\n\n[1] Data curation via joint example selection further accelerates multimodal learning", "essential_references_not_discussed": "Not aware of", "other_strengths_and_weaknesses": "Strength\n\n- The authors show that their approach indeed has great sample efficiency compared with the naive and the distillation approaches.\n\nWeakness\n\n- Based on what I understood, the proposed method optimizes the same objective as (Evans et al., 2024a;b). Comparing with their approach seems important.\n- Though in the introduction, the authors motivate this paper by the lack of theoretical understanding of LEAR, this paper doesn't seem to be able to explain ALL LEAR methods. It should be more specific about *which* LEAR method(s) this paper is for.\n- The flow of this paper could be improved. See suggestions.\n- Please also see questions.\n\nIn sum, while I acknowledge this paper has its technical merits, I suppose this paper was written in a rush. Thus, while technically I suggest this paper is probably acceptable, I also suggest that another round of revision will make it better and more up to the level of ICML. \n\nPlease let me know if I misunderstood anything.", "other_comments_or_suggestions": "If I understood correctly, this paper has two goals, one is to provide the theoretical foundation of LEAR, which, in this paper, seem to be specific to (Evans et al., 2024a;b). The other one is to propose a better algorithm for LEAR. While I don't see apparent problems for the second goal, the one seems to be done very implicitly. \n\nI would suggest that the authors provide some background about the specific LEAR methods this paper can provide insights into. This would make the purpose of Section 4 clearer.\n\nBeside, while it's the convention to call scaling law scaling law, the word \"law\" is actually misleading, as it is not a real law. If possible, I would suggest that you just say you scale better.", "questions_for_authors": "1. One thing I didn't understand is what the x-axis of figure 1 (Samples Seen (million)) means. Does it represent the training time? If not, why did you use different line to represent different ratio of data used? And why is the trend of Figure 2 different from Figure 3? It is confusing because these two figures have the same x and y axes. \n2. Would this method works for other tasks? Is there a reason to focus only on CLIP?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a rigorous theoretical foundation for Learning with a Reference Model (LEAR), a paradigm where a pre-trained model guides the training of a target model. By using LEAR with DRO, the authors offer a formal generalization analysis explaining this method's effectiveness. They further exploit the DRO-contrastive learning connection to develop DRRho-CLIP, for contrastive language-image pretraining with a reference model. Extensive experiments validate their theory and demonstrate superior performance over heuristic baselines.

The reviewers unanimously recommended acceptance of this paper, primarily due to the strength of its comprehensive theoretical and empirical contributions. I fully support this decision and believe the work will be of broad interest to both the ICML audience and the wider machine learning community.
Decision: Accept (spotlight poster)