Adaptive Training Distributions with Scalable Online Bilevel Optimization
Authors: David Grangier
Pierre Ablin
Awni Hannun
Venue: Accepted by TMLR ()

Abstract: Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely matches that of the application domain. This work considers modifying the pretraining distribution in the case where one has a small sample of data reflecting the targeted test conditions. We propose an algorithm motivated by a recent formulation of this setting as an online, bilevel optimization problem. With scalability in mind, our algorithm prioritizes computing gradients at training points which are likely to most improve the loss on the targeted distribution. Empirically, we show that in some cases this approach is beneficial over existing strategies from the domain adaptation literature but may not succeed in other cases. We propose a simple test to evaluate when our approach can be expected to work well and point towards further research to address current limitations.

Published in Transactions on Machine Learning Research (10/2024)
Adaptive Training Distributions with Scalable Online Bilevel Optimization
David Grangier, Pierre Ablin, Awni Hannun Apple
fgrangier, p_ 2balin, awni)
Reviewed on OpenReview:
https: // openreview. net/ forun? id= JJ1GV1555
Abstract
Large neural networks pretrained on web-scale corpora are central to modern machine learn- ing, In this paradigm, the distribution of the large, heterogeneous pretraining data rarely matches that of the application domain. This work considers modifying the pretraining distribution in the case where one has small sample of data reflecting the targeted test conditions. We propose an algorithm motivated by a recent formulation of this setting as an online, bilevel optimization problem. With scalability in mind, our algorithm prioritizes computing gradients at training points which are likely to most improve the loss on the targeted distribution. Empirically, we show that in some cases this approach is beneficial over existing strategies from the domain adaptation literature but may succeed in other cases. We propose simple test to evaluate when approach can be expected to work well and point towards further research to address current limitations.
1 Introduction
Large models pretrained on massive, heterogencous datasets have impacted various application do- mains [Bommasani et al., 2021), ; including natural language processing (Devlin et al., 2019), computer vision [Mahajan et al.J 2018), and audio processing (Schneider et al.) 2019p. These models are typically trained on two different distributions: a generic distribution for pretraining and specific distribution for fine tuning. Only the specific distribution matches the test conditions while the generic distribution offers abundant source of data with some similarities the specific data. This novel paradigm builds upon earlier work in multitask learning (Caruana 1997). transfer learning (Bennett et al. 2003), and domain adaptation (Moore & Lewis 2010p. For all of these methods, the accuracy of a model on the specific task heavily depends on selecting an appropriate distribution over the generic auxiliary tasks and data.
This work proposes a scalable online strategy for data selection along with a comprehensive and realistic empirical s We build upon a bilevel formulation of the generic re-weighting problem which allows for gradient-bases optimization (Franceschi al. 2018).
Contributions First, unify several gradient-based data selection methods into common framework in which their similarities and distinctions are more easily understood. Second, we introduce a scalable, online algorithm. This algorithm can train large model while updating an inexpensive auxiliary data selection model which tracks the distribution required to make fast progress on the targeted task. Our algorithm leverages the asymmetry in computational cost between the selection model and the large model by filtering examples on the fly, ensuring that the majority of examples are not examined by the large model. This allows for much faster training on the specific distribution than pre-training on the generic distribution only.
Third, we perform comprehensive and realistic empirical comparison of data selection strategies. We compare several alternative strategies across different tasks and modalities including large scale language
"odee reproduce our experiments is http:://ggiaaa..co/mpllllaaaiiit -aaaivii--trii-iitt

---

Published in Transactions on Machine Learning Research (10/2024)
modeling, machine translation, and image classification. Finally, we propose simple metric based gradient alignment that correlates with the success and failure of gradient-based data selection methods.
2
Related Work
Prior work has proposed aulomatic methods to adjust the generic training distribution in order to improve model generalization on the specific task. The domain adaptation literature has explored variants of impor- tance sampling, which uses importance weights to emphasize or select some generic examples. These weights have been determined via domain classifiers (Aharoni & Goldberg 2020 Gururangan et al.) 2020), via the estimation of the label distribution (Ngiam et al. 2018), or via gradient alignment and fine-tuning.
Contrastive Data Selection, CDS MMoore& & Lewis 2010; van der Wees et al., 2017 Wang al. 2018) falls into
this later category. This method has four phases: (i) an imitial model is pre-trained on the generic dataset,
(īi) this model is fine tuned on the specific data, (iii) the generic set is restricted to the generic data whose loss improvement between the pre-trained model (i) and the fine tuned model (ii) is the greatest. Finally, (iv) the training of the pre-trained model (i) is resumed on the selected data from stage (iii). Although CDS is a generic method applicable to any training objective, it enjoys additional properties when applied to generative models trained to maximize the (conditional) training likelihood. It can both be considered an importance sampling method and an influence function based selection method (Grangier & Iter 2022).
Huang et al. 2006) cast weights estimation as a quadratic problem with kernel function. Related to domain adaptation, the removal of label noise in the generic distribution has reccived attention with methods based on influence functions KKhh & Liang 2017 Pruthi et al. 2020 Schioppa et al. 2022p, data models (lyas
et al. 2022 Jain et al., 2022 and data Shapley values (Ghorbani & Zou 2019 Karlas et al. 2022).
As an alternative to static weighting, the literature also explored dynamic weighting where the distribution over generic examples is adapted during training. The two primary strategies are reinforcement learning and direct optimization. Reinforcement learning does not assume that the specific task loss can be differentiated with respect to the weighting parameters. Instead, parameterized model of the generic distribution is adjusted through reinforcement learning: the current. model proposes generic distributions, and their reward is measured as the specific loss after a few steps of generic training over proposal distribution (Kunar et al. 2019 Yoon et al. 2020 Zhu et al. 2020). On the other hand, direct optimization assumes a differentiable lunctional dependency belween the weighting parameters and the specific training loss. This dependency can be derived through meta learning by unfolding the generic update (Ren al., 2018: Hu et al. 2019 Shu et al. 2019 Zhang& & Pfister, 2021p: one gradient update step minimizing the weighted generic loss depends the weighting parameters. The impact of this update can be evaluated by computing the post-update specific loss which can then be differentiated with respect to the weighting parameters. As an alternative to update unfolding, bilevel formulation of the reweighting problem also allows for direct optimization Franceschi et al. 2018). Our work builds upon this bilevel formulation.
Other research areas intersect with sample reweighting. Ho et al. (2019); Lim et al. 2019 Zoph et al. (2000) considered learning a distribution over training data augmentations. Curriculum learning visits successive
training distributions based on training instance difficulty (Bengio et. al.] 2009 Kumar al., 2010 Jiang
et al. 2018 Saxena et al. 2019). Multi-task learning research has considered gradient projection to minimize negative interactions between tasks (Yu et al. 2020 Dery et al. 2020 Liu et al. 2021). Importance sampling for accelerated stochastic training (Zhao & Zhang
2015
Katharopoulos & F'luuret 2018) is also relevant.
3 Problem Setting
Classical machinc learning assumes that the model is trained on data drawn from the distribution from which the lest. data will also be sampled from [Vapnik 1999). Our setting is different and belongs to the field of transfer learning (Caruana, 1993 Thrun & Pratt 1998). We are given two training sets, a large generic training set D Renerii and small specific training set D Depecific- Only the latter set is representative of the test conditions. The large generic set can be leveraged as it might contain information related to the targeted specific. Its large scale allows more reliable statistical estimation and allows training higher capacity models.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Domain adaptation (Farahani et al. 2021 multi-task learning (Caruana 1993), fine-tuning (Denevi et al, 2018) are transfer learning setups related to our setting, see Appendix for a discussion on the relevant terminology. Importantly, we assume that the targeted task is known at the beginning of learning (which is not a necessity for fine tuning) and that only the specific task accuracy matters (unlike multi-task learning, which might also larget high generic accuracy). While our main focus is largeting a single specific task, we discuss the case where the specific loss is a mixture over multiple targeted specific tasks in Appendix
Optimization-based Filtering. Large language and vision models have been shown to perform well wide range of test distributions. This stems from two combined causes: (t) diverse training set which is mixture over many domains and (ii large capacity to fit. cach domain in the mixture. Therefore, there is an inherent tradeolf wetween capacity and generality. When one knows the application domain in advance, fitting a large web corpus without data selection will waste capacity on train examples far from the targeted domain. Ad-hoc selection techniques are commonplace for that purpose (filtering of captions by confidence in LAION, books are all you need for QA (Gunasckar ct al.) 2023), etc..... Our goal is explore optimization-baseed techniques to complement widely used application-specific filtering techniques.
4 Methods
We aim to identify the parameters 0 of a model that achieves good generalization performance (held-out likelihood) over the specific distribution. We are given large generic training set peeneric and small specific training set Papecifie. We cast the generic training problem as the minimization of the weighted loss
C (0,
Σ
w(z;a))((;;
DED
where w((;; a)) denotes smaller, secondary weighting neural network which defines a distribution over and ExeD yeneric a(x;a) 1. The weighting network is parameterized by
D gennerie, i.e. Vx, w(z;a) the parameters We propose to use weighting network that is much smaller than the main model, so the dimension of a is typically smaller than that of f. We denote the solution to generic training problem as
0" (a) E arg min Cgeneric (0, ar)
(1)
which depends on the weighting networks' parameters a. Our goal is to find the parameter of the weighting network such that the loss on the specific training set is minimal, i.e. minimizing,
Eapecific (0" (a))
Σ (a);
(2)
d'EDapactie
with respect to a, where the samples x' come from the specific training set.
4.1 Data Selection as a Bilevel Optimization Problem
The previous equations make it clear that finding the optimal weighting network is a bilevel optimization problem: with a fixed weighting network, the optimal parameters for the main model are found by minimizing the weighted loss over the generic dataset, fegnneric (Equation [ The optimal main model parameters 0" depends explicitly on the weighting network parameters a: changing a changes the optimization problem in Equation 1 and its solution. The selection of is driven by the specific set loss, Lapecitic (Equation 2
Equation and Equation 2 form bilevel optimization problem (Franceschi et al. 2018): the outer problem (Equation depends implicitly on a through the solution to the inner problem [ One of the strengths of such a bilevel formulation is that the weighting network must adapt. to the main model: the question is to learn a weighting network such that the main model trained with that network leads to good specific performance. This has the potential to go beyond a simple model-agnostic scheme that would, for instance, build w(x) based on the similarity between x and the specific set. While large body of the literature is devoted to solving bilevel problems where the inner problem Equation convex in 0 (hadiii & Wang
3

---

Published in Transactions on Machine Learning Research (10/2024)
2018 Arbel & Mairal 2021), in case, Equation |
corresponds to the training problem of neural network This leads to several difficulties:
which is non-convex.
1) The arg min in Equation is not a single element since there are multiple minimizers. Therefore, the function 8* (a) is not properly defined.
ii) In order use gradient-based methods to find the optimal a, we have compute the approximate Jacobian of (a). This is usually done using the implicit function theorem, which only applies when the loss function in equation D is locally convex and such property is hard to check in practice.
Furthermore, we want a method with computational cost similar to the standard training of the main model. In other words, we have enough budget to solve Equation [ only once: learning a and must be carried out synchronously. This has an important consequence: the bilevel methods that we study update a basod on the current main model state and not on the optimal solution 6* (a)).. Hence, this deviation from the bilevel formalism. This also means that the weighting network adapts to the current state of the main model and, ideally, tries to up-weight generic data that is useful al the current state of learning. We explore online algorithms to solve the bilevel problem when the main model is large. These algorithms alternate I and updates and leverage the asymmetry in computation cost between evaluating the large main model and the small auxiliary weighting network.
slight
Algorithm 1 Scalable, Online Bilevel Data Selection
Require: D Pgeneric; Depeccitte Osmall, barge Training datasets, batch sizes. 0o + main model initializer()
ag t weight weight_modell model_ inneoiiiiitiiizr()) for / - 1....,T do
b Sample generic and specific batch.
B Bxeneric f- sample(D gonerios barge))
Sspecific < sample (Dspeiifeee bemall)
Sample generic sub-batches.
Buitered filter( B generii: Ot bamall) B' geeneric + sample B Bgenerio bamall)
b Inner and outer updates.
a + _aan [atue__ooe(((_1111 Britered, 0-11)
0. + uuddate__
(B (geuerier Bspecitics Oc az-1)
end for return 0T
b Trained main model.
4.2 Updating the main model
To update the main model, we fix a and do a step to minimize Equation [ A first, natural idea would be to take a mini-bated of generic data Bgeneric size b, compute the corresponding gradient g = Exe Bgeerrs w(x;; ax) Vol(;; and then use it to update 0, either implementing SGD by doing 0 0 17 9 with 17 0 learning rate, or by using it into more involved optimizer like Adam. However, the computation of g with the previous equation can be wasteful when significant fraction of the examples of Bgenerie are assigned small weights w(x; a). These examples do not contribute much to g while still requiring the expensive computation of their gradient Vol(z;
To accelerate the optimization of 0, we leverage the asymmetry between the cost of evaluating the weighting network and the main model: computing w(æ;; a) only requires inference of a small network while computing Vi(x; f) requires inference and back-propagation through a large network. We start by sampling a large generic from the generic dataset and compute w(x; a) for each x generiic From there we can take smaller batch Bgmallic from Biig Bgineries either by sampling from the distribution defined by w(x; a) or by taking the examples with the highest o();a). The first option is unbiased solution corresponding to
batch B Bbig
4

---

Published in Transactions on Machine Learning Research (10/2024)
importance sampling, while the second option is biased but observed to work better in practice. In both cases, we compute the gradient to update 0 with uniform weights, using 9 = Exe BrauBl Vol(x; 0).
B This proposed training method sparse: the samples in gennrii have a sparse weight, only fraction of them is non-zero. In the experiments, we prefix all methods that use this stralegy with the sparse adjective.
4.3 Updating the weighting model
With scalability mind, we only consider slochaslic methods, Le., that update the weighting network parameters a using only a mini-batch of specific data Especific and a mini-batch of generic data Bgeneric. We consider three alternatives update the weighting model.
Before describing alternative methods to update a, summarize our approach in Algorithm ! We denote sample(D, ( n) the set resulting from sampling n times uniformly from a set D. We denote filter( a, n) the result from either (a) sampling n times from D i.idd relying on the distribution induced by the weighting model at a, or (b) selecting the top-n highest weighted examples from D. The batch sizes bsmall, blarge are hyper-parameters selected through validation.
4.3.1 One gradient step unrolling differentiable data selection (DDS)
This method updates the weighting network by doing descent step on the loss
((a)=
Σ (ld': a)) with u(0: a) = 0 - p Σ v(x;()Vol(x;O),
(3)
o' E' Bapecifie
thE Bgansric
which corresponds to the value of the specific loss on the mini-batch Bspecitic after a gradient descent step for on the generic mini-batch Bgeneric using the current weights. It is similar to (Wang al. 2020). The idea behind this method is that w(0,a) is reasonable approximation to (a)).. This method requires back- propagating through a gradient descent step, which requires only a little overhead compared to a standard gradient computation. In the limit where the step size p in the gradient update u(0, a:) goes to O we see that C(a:) = plgspecificr Igeuerie); with Ispecifie = -Ex' -Ex'EBaente BypennRi (z((z;; 0) and Igeneric = -ExE w(x; V 0). Hence, the loss L approximately measures the aligmment between specilic and generic gradients. Taking derivatives gives VL(a) E Bxonerie (gspectfic: Vl(zz 0)) (w)
4.3.2 Stochastic Bilevel Algorithm (SOBA)
We also implement the SOBA method of (Dagréou al. 2022), which is a scalable method to solve the bilevel problem, developed in setting where the inner function (Equation [ is convex. This algorithm approx- imates a gradient descent on h(a)= (0 The chain rule gives Vh(a) de
ae' Lappccitio (a))).
The optimum 8* (a) satisfies the first order condition that the Hessian V2 Eggnriicr (a)) is invertible, the implicit function theorem applied to the previ- a) [V Cgeneiii (0 (a). which overall yields Vh(a)
(f"
= 0. Under the assumption
ae ous equation gives da Ceooeeeei -Va (f a) [2 (8 aa) Lespectii (a)). SOBA approximates this quantity in two ways: first, (a(a) is replaced by the current iterate 0 in the above gradient. Second, in addition to 0 and SOBA has an additional variable u of the same size as 0 that keeps track of the quantity 1) de with
(0). This is done using the stochastic iterations [z) a P(; 0)v + Ed'EB.mane V (z': The first part in de is Hessian-vectorr prod- a cost. similar to that a gradient (Pearlmutter 1994). Then, the parameters are moved in the direction da Exe Bremeric (Vi(2; 0), v) Vw(z; which is a stochastic approximation of V2 Cgeneeni a)v, which is itself an approximation of Vh(a).
V2 Vegngeneric (0) VoL VoLspocitic(())
du = -x uct that can be computed efficiently
4.3.3 Aligned NOrmalized GRADient (Anograd)
We derive Anograd (Aligned NOrmalized GRADient) as variant of DDS which relies on steepest de- scent. [Boyd & Vandenberghe 2004 We apply the steepest descent. algorithm to the specific loss
5

---

Published in Transactions on Machine Learning Research (10/2024)
0 > (8)) We recall that the steepest normalised descent direction according to the Euclidean norm || for the specific loss is
Addd arg min Vo VoCspecifie (0) ||||| = 1}
(4)
This direction aligns with the opposite gradient when e not further constrained. In our case, 0 updates should correspond to gradient descent updates on the weighted generic loss. We therefore constraints 0 updates to decompose as an afine combination of individual generic example gradients, i.e. Σ a V.Ve,, {
where D& Dgeneric is denoted as {xft Ji-l and ag > 0, Vi. Therefore, we need to solve Equation with the
constraint V, with V {EE a, Vl(x A), Vo, 2 0}
This amounts to solving
F a VE( 1 a Ve(27. 0))1
E
min
V Capecciii (8))
which itself is equivalent to solve min, cosine (E a, Vi(az. VoLapecate(()) We param- eterize as the output of the weighting network and introduce the loss, G0nograa(O.0)) a) = cosine (Volgeneric(O,og) a), VoLapecitc(O)). The anograd method performs gradient descent on that loss to up- date a. Like for DDS and Soba, we perform a single step before updating 0. For scalability we rely on stochastic (batch) estimates for both both terms in the cosine. Compared to DDS, the normalization in anograd reduces the benefit of weighttngng generic examples with high gradient norm.
5 Experiments & Results
Our experiments focus on three application domains: language modeling, machine translation and image classification. Before introducing our experimental setup and discussing our results on each domain, we describe the baselines we considered.
5.1 Evaluated Alternative Methods
For our empirical comparison, we first consider two common, simple methods which do not rely on data selection. We call baseline pretraining on the generic training set followed by fine tuning on the specific set. We call mizing pretraining on mix of generic and specific data. Each training batch contains fixed fraction of specific data. This fraction is selected by validation.
Our first baseline is contrastive dala selection, CDS, as described in infsection 2 As we do for all data selection method, we consider further fine tuning the final CDS model on the specific training set.
We also consider a domain classifier. In that case, a simple model is protrained on a binary classification problem to distinguish between generic and specific training examples. The model has the same architecture as the weighting model we use with bilevel methods and minimizes the binary cross entropy on batches with the same proportion of specific and generic data. This model can estimate the probability that an example belongs to the specific set and is applied to restrict the generic set to the data with the highest estimates We can train model on this restricted set and later fine tuning the specific data.
Closer to our bilevel selection methods, we evaluate learning to re-weight, LTR (Ren et al. 2018) and meta-weight net (Shu et al.] 2019)). Learning to re-weight is similar to the DDS approach we presented in Section $ exceppt it does not maintain weighting model. Instead, at each step, the model considers uniform distribution over the generic batch. It then computes the gradient of this flat weighting as free paramoters with the outer update, Equation 2 This single step updates from uniform is then used to reweight the generic loss and update the main model, Equation D Compared to work, this method does not persist weighting model across steps and does not allow learning complex distributions. The lack of weighting model is also less efficient since a generic example x cannot be discarded without evaluating the main model and its gradient at x.
Meta-weight net is a particular, simple case of DDS which the weight model takes as input a single scalar for each example: the example loss, i.c. w(2; a) = mlp({(x;;)) This parameterization sensible for some applications, e.g. loss based up- weighinht is common approach for active learning in classification problems
6

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Model architectures
Language Model
Main model: Transformer decoder with 12 layers, 8 attention heads, residual dimension of 256, feed- forward latent dimension of 1,024.
Weight model:
Convolutional network with layers followed by mean pooling, latent dimension of 128.
Translation Model
Main model: Transformer with 6 encoder layers and 6 decoder layers, 16 attention heads, residual dimension of 1,024, fera-forward latent dimension of 4,096,
Weight model:
Emnbediing layer of dimension 32 followed by an MLP with latent dimension of 128.
Image Classifier
Main model: Dual encoder clip model with Res.et 50 for images (224x224) and an multi-layer pereep- tron (2 latent layers with dim. 768) lop of sentence BERT for texi.
Weight model:
Convolutional network over 32x32 images with 4 layers of dimension 32, 32, 32 and 64.
with little intrinsic uncertainty (Settles 2009 Loss values can be indicative of an example difficulty and loss based up-weighting might accelerate learning. However, an example loss seems be orthogonal to the example value for domain transfer.
5.2 Language Modeling
Our language modeling (LM) experiments relies two datasets, the C4 dataset RRffell et al. 2019) is used as the generic set and the RCV1 (Lewis et al.] 2004) dataset is used as the specific set. C4 is dataset of English language web pages from common crawl (Patel 2020), while RCV1 consists of newswire stories from Reuters. This setup is represenlative of a generic large corpus spanning different types of examples (c4) while the specific task contains an homogeneous set of examples from the same domain and from the same source (RCVI). In our setup, we use 30m examples from C4 and 10k examples from RCVI.
Our language model is byte-level language model based on the transformer decoder archi- tecture (Vaswani et al. 2017). Although sub- word language models are more common than byte-level ones (Semnrich et al. 2016 Af-Rfou et al. 2019), we rely on bytes to avoid our out- of-domain generalization results to be contam- inated by the mismatch between the evalua- tion data and the tokenizer training data (Rust) et al. 2021 The weighting network is small convolutions network. Table [ gives archi- tectural details. We also use the same archi- tecture for the domain classifier baseline. We report performance terms of log- pegpeeelyxi,t, i.c. negative log likelihood. Our implementa- tion is derived from the language model of the Flax library (Heek al. 2020
Method
Pre-train
Fine-tune
Baseline
1,198 = 0.003
0.864 = 0.002
Mixing
0,861 = 0.002
0,847 = 0.002
CDS
1.067 0.005
0.867 0.002
Domain classif.
1.098 + 0.002
0.894 + 0.003
Meta Neightttt
1.212 = 0.004
0.868 = 0.003
LTR.
1.156 + 0.002
0.879 + 0.002
Sparse DDS
1.039 + 0.004
0.824 + 0.003
Sparse Anograd
1.034 + 0.002
2
0.824 + 0.002
Sparse SOBA
1.019 + 0.003
0.820 + 0.002
Table 2: Language modeling: Log-perplexity (negative log- likelihood per byte) specific (Reuters). Circled num- bers indicate the best results. Numbers after the + are the standard deviation between 5 runs. The adjective sparse refers to methods that use the sparse batch trick described
Table reports the results of our language modeling experiments. In general, domain adaptation is beneficial our setting. The only method trained exclusively on c4 (baseline without fine- tuning) much worse than all alternatives except for Meta WeitWteigt,Net.. Before protraining, mixing is the only method which directly applies updates from the specific training data and performs best. The other methods only emphasize part of the generic set without applying specifie updates during pretraining. This
in Sec. 4.2
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Language modeling: Log- Log-perples perppeeeyeyyty on specific for Pile domains.
Method
arxiv
europarl
freelaw
gutenb.
opensub.
openweb,
pined abs
stackex.
wikipedia.
Base
1.438
2.219
1.555
1.365
1.277
1.220
1.088
1.313
1.103
+ ft.
0.898
0.993
0.603
0.488
1.058
1.180
0.870
1.039
0.877
Mixing
0.909
1.019
0.606
0.487
1.067
1.153
0.874
1.049
0.874
+ ft.
0.899
1.081
0.600
0.478
1.059
1.156
0.860
1.042
0.865
CDS
1.216
1.981
1.284
1.253
1.193
1.154
0.829
1.100
0.944
ft.
1
0.861
0.977
0.614
0.482
1.039
3
1.131
0.791
0.971
0.823
Classifier
1.293
1.608
1.133
1.202
1.266
1.139
0.787
1.159
0.914
+ N.
0.920
0.892
0.582
0.481
1.066
1.125
0.765
0.998
0.807
DDS
1.231
1.868
1.184
1.285
1.288
1.290
0.828
1.112
0.988
ft.
2
0.867
0.948
0.580
0.477
1.104
1.262
0.792
2
0.977
0.839
S.Anograd
1.219
1.659
1.237
1.274
1.193
1.150
0.814
1.132
0.989
ft..
0.871
3 0.895
2 0.580
! 0.477
1.042
1.133
0.784
0.988
0.838
S. SOBA
1.210
1.582
1.124
1.296
1.184
1.149
0.803
1,134
0.908
ft.
0.872
T 0.883
① 0.579
2 0.480
L 1.035
2 1.128
0.779
3
0.989
T 0.803
emphasis already show benefit at pretraining time. More importantly, this benefit is complementary fine tuning (Tter & Grangier 2021) and these methods yield better results that mixing finettinng.. Among them, bilevel methods perform best, with SOBA giving the highest held-out specific likelihood.
We perform additional language modeling experiments with different domains. We take 9 domains from Gao ct al. 2021 and rely on 10k specific document for each domain. The generic set (c4 dataset) and the model architectures are the same as in the previous experiments. Results are given in Table 3 Data selection methods show that it is helpful to emphasize part of the generic set and that this emphasis complementary the benefit of fine tuning. The benefit varies across domains. For instance openweb is similar to the generic set c4 and only modest gains are observed, while freelaw contains legal procoedings whose domain is surcly relatively rare in c4. Among methods, CDS and classifier provides strong benefit. for some dalasets, but only SOBA consistently ranks among the best methods.
5.3 Machine Translation
Our machine translation (MT) experiments learn a translation model from English into German. They rely on two datasets: our generic set the Paracrawl dataset (Release 1 for WMT 2018) with 36m sen- tence pairs (Bañón et al 2020). Our specific set con- catenates the WMT newstest sets (2009-2019) with
Method
Pre-train BLEU
Fine-tune BLEU
loss
loss
Bascline
27.63
2.56
34.06
2.53
Mixing
31.34
2.60
33.11
2.69
CDS
34.14
2.53
34.25
2.53
source original English sentences, which amounts to
Domain classif.
35.56
2.37
38.03
2.35
10,015 sentence pairs (Akhbardeh et al.) 2021). We use the 2020 newstest data (1,997 sentences) as our validation set and Ieave the 2021 newstest data (1,418 sentences) as our test set. Our generic set. there- fore large crawled set with different types of text and varying translation quality while the specific set is a small set from a single domain with high quality translation.
Meta Nei
26.81
2.59
33.34
2.53
LTR
28.60
2.73
31.15
2.71
Sparse DDS
33.53
2.46
35.83
2.44
Sparse Anograd
36.06
2.41
37.28
2.40
Sparse SOBA
34.23
2.39
37.16
2.38
Table 4: Machine translation: BLEU and loss on spe- cific (ncwstest2020).
Our translation system a sub-word model based on the transformer encoder-docoder architecture. For the weighting network and the domain classifier we compose a shared embedding layer for source and target and apply multi-layered perceptron the contatenated averaged embeddings of the source and target sentences.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table gives architectural details. Our evaluation relies BLEU scores (Papineni et al. 2002) for beam-4 decodes. We also reports some results in terms of loss (i.e. negative log likelihood with label smoothing strength of 0.1). Our implementation is derived from the translation model of the Flax library (Heek et al. 2020
Table reports the results of our machine translation experiments. In that case, SOBA and Anograd provide a strong improvement over the baseline, both before (more than +7 BLEU) and after fine tuning (more than +3 BLEU). However in this setting, the domain classifier is even more effective. Both for language modeling and for machine translation, we remark that Meta Wei performs poorly. Meta WeightNet predicts the selection weight from the loss on the example. This is a common strategy in active learning for classification (Settles) 2009). This notably assumnss that the loss per example is indicative of how well the model perform on them. However, in the case of language modeling and translation, the loss per example also reflects the intrinsic entropy of the examples which varies greatly across examples. It. therefore seems difficult to use the loss the sole feature for data selection for these tasks. Comparing the results of LTR and DDS is also interesting as the methods are similar. DDS maintains weighting model across steps, while LTR just adapt the distribution for each batch and does not persist cross-steps selection parameters. The benefit of DDS tells that the stability across steps and the lesser dependency on the batch size are important.
5.4 Image Classification
Our vision setup performs contrastive training over image and captions CLIP (Radford et al. 2021) for generic training and image classification for spe- cific training. Specifically, contrastive learning should select the correct caption within a large set of ran- dom captions. This approach also allows to perform classification by representing classes captions of the form 'a photo of <class name>" and letting the model infer the most appropriate caption within that set. As datasets, we rely on yfec15m [Rad-
Method
Pre-train
Fine-tune
Acc.
Loss
Acc.
Loss
Baseline
41.1
2.694
54.9
1.902
Mixing
55.1
1.928
3
55.1
1.928
CDS
42.3
2.512
2
55.2
1.957
Domain classif.
44.1
2.571
①
57.5
1.949
Meta Net
35.5
2.743
43.9
2.351
LTR
36.4
2.712
44.9
2.364
Sparse DDS
40.5
2.609
53.2
2.067
ford et al. 2021) for generic training (14.9m im- age/caption pairs) and ImageNet67 (Eshed 2020) dataset for the specific task. Imagenet 67 consists in 67 high level classes over Imagenet (Deng et all, 2009), e.g. building, person, fish, dog... Like for other experiments, we consider a setup with limited specific data and take 2,010 specific examples, 30 per class, for training. Held-out evaluation is performed with 50 images per class.
Sparse Anograd
41.4
2.563
53.6
2.055
Sparse SOBA
41.1
2.622
53.9
2.057
Table 5: Image Classification: Accuracy specific (ImageNel67).
For our CLIP model, the image branch is a Resnet 50 (He et al. 2016) while the text branch applics an MLP over precomputed sentence embeddings from Sentence BERT (Reimers & Gurevych] 2001)). Training applies contrastive learning only over alternative captions: for the generic loss, we consider a buffer of past captions as negatives; for the specific loss, we consider all the other class captions as negatives. Our weighting network is a small convolutional network over low resolution images (32x32). Table ! gives architectural details.
Table 5 reports the results of our image classification experiments. Unlike for our text experiments, the benefit of data selection is limited for this task. After fine-tuning, only the CDS and domain classifier methods outperform the baseline model. The bilevel data selection methods do not outperform the bascline method. We shed light the cause of this poor performance in further analysis in Section 6.3
9

---

Published in Transactions on Machine Learning Research (10/2024)
Does the weight model's trajectory corre- spond to a curriculum? Pre-train LM log perplex- ity on Reuters for different trajectories.
Table 6:
Table Alternative sampling strategies filter the generic batch. Pre-train LM log perplexity Reuters.
Weight model trajectory
log-pen perppeelyxytty
Sampling strategy
log- log-perplexity
SOBA curriculum
1.047
Importance Sampling
2.038
Final weighting
1.044
Sampling without replacement Selecting the highest weights
1.040
Shuffled weighting
1.055
1.227
Table 8: Specific & Generic Acceleration Rates SAR
Task
GAR
Language modeling
(c4, reuters) (paracrawl, newstest)
86.2%
69.4%
Machine translation
78.1%
68.2%
Image classification
(yfee15m, imagenet67)
50.3%
49.8%
6 Analysis
6.1 Learning Distribution vs Learning a Curriculum
A Algorithm | produces a sequence of main model's parameters 0, and weighting model's parameters ot, that go towards the solution of the bilevel problem (2). We investigate whether the weighting model's parameters correspond to a curriculum: does the evolution of the weighting parameters ar adapt to the partieular data needed at each step, helping the model perform better than fixed weighting scheme?
Wc are in the LM task setup described in Section 5.2 except that we use smaller "large batch size" B B (see Section
generic 1.2 We run Algorithm 1 with SOBA to obtain sequence 0,, azz, We then compare this set ting with two new training runs with standard ERM using different data weighting:
Final weighting: a new main model is trained with fixed weighting from the weighting model or-
Shuffled weighting: new main model trained with a random permutation a of the weights ao(t))- ( shows that. SOBA's 's curriculum is not beneficial compared to the fixed final weighting scheme on this task. The lesser performance of shuffled weighting certainly highlight poor weighting from early ar Results reported in this section do not match Section 5.2 because of smaller B was used in this ablation.
Table
6.2 Big Batches: Importance Sampling vs Filtering
In Algorithm we denote filter(B, a, n) the operation resulting a smaller sub-batch of size n starting from the generic batch B using the weighting network paraneterized by a. To get an unbiased estimate of the ro-weighted generic loss, can apply importance sampling and sample (with replacement) from the weight distribution induced by
B. Allernatively one can instead sample withoul replacement from that distribution or restrict the batch B to its highest weighted elements. The last two alternative are biased. Nevertheless, our results Section 5 uses sampling without replacement.
Table 7 justtiies this choice. Basically, we observe that the learned weighted distribution is concentrated along few examples which yield importance sampling batches to contain less diverse sets than when sampling with replacement. Similarly, cutting the tail of the distribution (highest weights selection) drop lower weighted but still helpful examples. These experiments illustrate that gradient-basee estimates fail to account for the long term benefit of a more diverse training set. Although sampling with replacement alleviates this issue, more principled solutions should be investigated in future work.
6.3 On the Discriminative Power of Gradient Aligments
Our experiments highlight that bilevel optimization for data selection performs differently across tasks. We explore if simple diagnostic could help understand these differences. Our method considers base model 0,
10

---

Published in Transactions on Machine Learning Research (10/2024)
trained the generic distribution for t steps. We take diagnostic batch Buix which blends unseen generic and specific data in equal proportion. We want to verify how the weighting model on the mixed data would move away from a uniform weighting scheme in an outer update. We want to observe whether the weighting model would increase the weights of specific examples some of these were "hidden' within the generic set.
For DDS and Anograd, increase or decrease weights depends on the alignment between individual example gradients from x Bmix and the expected gradient on the training specific batch Bspecifie-
a(z, Bapecitie)
Vol( V) Caeppeeciiiie
between individual example gradients from E Biix and the expected gradient on the training specific batch EreBBaecttt ((x, 0). We then normalize the batch gradient and Bepecttic
Especifie, denoted as Bpspeiife: 8) == [apecafic Exe
define,
Vap(Bapecifie 0)
Gporm (I, Beporitic) = Vol(z, 0)T
Vol ( Rspecifie, 0)1
This normalization allows to take example and verify whether its gradient aligns better with the specific batch gradient than with the generic batch gradient, i.e.
Gnorm (x, Bepocitie) > anorm (I, B Beenerie).
We report the rate at which this inequality is true for specific examples,
SAR
{anorm (2, Bsprecftee > Qnorm (x, B Bgenerie)
aow D Spoccific B Bapacific BreedBe--Bachht Batchi Papecisi) Bgenerie C Batch1 DD geuertt))
We call this measure the Specific Acceleration Rate, SAR. We would like this rate to be high, meaning that, according to the Taylor approximation the loss, updates collected from a batch of specific examples should improve the loss on a given specific examples faster than updates collected from a generic batch. Symmetrically, we define the Generic Acceleration Rate,
GAR
E generio
1 {Anorm (I, Bgeneric) > anorm (Ir, Bepocifie))
B Bepocilic Ceoocitie--Btee Batch speciile)
Beenerto - Batch(D Batchh(Pgnentt
It is also desirable that this rate is high. When SAR, GAR are close chance (50%), there are two possible explanations, (i) either generic and specific batches have the same effect the model, meaning that data selection is unlikely to be helpful since training generic is already as good as training specific for the purpose of minimizing the specific loss; (ii) alternatively, the linear approximation (order Taylor expansion) does not help discriminating between the effect of generic and specific examples on the specific loss. In that later case, such a lcarning problem will be a challenge for bilevel optimization methods where gradicnt alignments indicates which part of the dataset to upweight.
Table 8 reports SAR and GAR for our results. These results are indicative of the empirical benefit of bilevel optimization methods for data selection. Language modcling where DDS, Anograd and SOBA are advantageous, has the highest SAR. Conversely, our image classification problem shows near random SAR, GAR in line with the poor performance on bilevel methods on this problem. We therefore consider that measuring SAR/GA BAR/GAR can be simple but informative diagnostic to assess the potential benefit of bilevel methods on a new problem.
6.4 Re-using Weighting Strategies with Larger Scale Models
The weighting network is trained by solving the bilevel problem (2, where the loss function é depends on the model's architecture. We investigate whether a weighting network learned with a small model can be re-used out-of-tho-box train a large model and get good performances on the specific set. The weighting network is frozen: the large model is trained by solving ming EeeP Panaeecc w(x; (){ f) with a fixed to the final parameters of the weighting network trained with the small model. We perform this experiment
11

---

Published in Transactions on Machine Learning Research (10/2024)
1.5
Pre-train Spocific Loss
Classifier
Sparse SOBA
1.4
Weights from Small
Bascline
1.3
1.2
1.1
1.0
2
Time (hours)
Figure 1: Specific loss as a function of time for the scaling experiment. Sparse-SOBA which learns weights on the fly accelerates learning. Using frozen weights learned with a small model leads to even faster training because this removes the cost of learning the weights. The training acceleration is significant. The loss value of 1.3 is reached 8.5 (resp. 3.9) times faster by the frozen weights (resp. Sparse SOBA) method than the baseline. The loss value of 1.25 is reached 21 (resp. 5) times faster by the frozen weights (resp. Sparse SOBA) method than the baseline. Note that the Sparse-SOBA method eventually gets better than the frozen weights method, as reported in Table 10 but this happens after longer training time.
Table 9: Small and base model architectures for scaling up the language modeling task.
Table 10: Results of the scaling experiment
Model
Method
Pre-train
Model
Layers
Res. dim.
ff dim.
# Params
Small
Sparse SOBA
1. 1.292
Small
4
128
512
824.064
Large
Baseline
1.197
Large
12
256
1024
9.530.880
Large
Sparse SOBA
1.018
Large
Weights from Small
1.034
the language modeling task (Section 5.2 where the small model and large model architectures are specified in Table g the large model's architecture is the same as in Scction 5.2 and it has about ten times more parameters. We observe that the weighting werk learned with the small model transfers the large architecture and leads to large decrease in the loss on the specific set, which is only slightly worse than using the Sparse SOBA method on the large model itself. The weighting network learned at a small scale seamlessly be used at larger scale and lead to significant performance improvement on the specific set.
In order to have different perspective on the improvements provided by the reweighting methods, we display the training curves for that experiment in Figure 1. We see that the proposed methods lead to significantly faster training on the specific set.
7 Conclusions
This work studies bilevel optimization for learning training distributions. We consider the setup where a model is trained from two training sets, a large generic set and small specific set, where only the later is representative of the test conditions. We propose scalable algorithm that learns a training distribution over the generic data such that the loss on the specific set is minimized. We showed that our formulation gathers independently-propsss introduced an algorithm that enables streaming through the generic datasel quickly by examining most of the generic samples with only an inexpensive small auxiliary model. This work reported a comprehensive and realistic empirical comparison of data selection strategics across language modeling, machine translation and computer vision. We studied the conditions in which gradient- basee data selection is effective and propose diagnostic based gradient alignment to efficiently assess these conditions.
gradient- based methods for data selection under a common framework. We
12

---

Published in Transactions on Machine Learning Research (10/2024)
Our work also delineates interesting questions for future work. Conceptually, we observe that gradient- based selection methods fail to reward properly the diversity of the selected samples (Section 6.2p, which deserves further theoretical study, Empirically, the complementarity between fine-tuning and generic data selection highlights that the updates collected from re-weighted generic training and from specific training are different. The existence of complementary updates and their exploitation might also be possible even when one is presented with a single monolithic training distribution.
The experiments in this paper have bigger scale than most experiments found in the data reweighting litterature (Ren et al. 2018 Shu et al., 2019). However, our methods are not applied to what is nowadays considered "large" models. The scaling ablation (ubsection 6.4) suggests that the proposed methods would be well-suited for large-scale models, and we plan to conduct experiments on such models in the future.
References
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7747-7763, Online, July 2020. Association for Computational Linguistics. doi: (0.18653//112222..accmm https://acllntthol.og www.gy.org//20000 www.-mmmmmm...m
URL
Farhad Akhbardeh, Arkady Arkhangorodssy, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España- Bonat, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewiez, Barry Haddow, Lconie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsab-Kaakyinn Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). Tn Proceedings of the Sirth Conference on Machine Translation, pp. 18, Online, November 2021. Asso- ciation for Computational Linguistics. URL lttps://aclaanthologg/ org/20201 2ttp/2222.mmt11..11.....
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3159-33 3159-3166, 2019.
Michael Arbel and Julien Mairal. Amortized implicit differentiation for stochastic bilevel optimizatiion arXiv preprint Xix::22111. 14580, 2021.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis. Mikel L Forcada, Amir Kamran, Fahcem Kirefu, Philipp Kochn, et al. Paracrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Anreual Meeting of the Associalion for Compulational Linguislics, pp. 4555 4567, 2020.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In An- drea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebee, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pp. 41-48. ACM, 2009. doi: 1111145/15533741553333 URL atttps://doi. org/10. 11t.//11.11151153337133 1553380
Paul N Bennett, Susan T Dumais, and Eric Horvitz. Inductive transfer for text classification using generalized reliability indicators. In Proceedings of the ICML-2009 Workshop on The Continuum from Labeled to Unlabeled Data, 2003.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research, 10(9), 2009.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dal- las Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora
13

---

Published in Transactions on Machine Learning Research (10/2024)
Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karam- cheti, Geoff Keeling, Fereshle Khani, Omar Khaltab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Mamning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Malei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2021.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
R Caruana. Multitask learning: A knowledge-based source of inductive biasl. In Proceedings of the Tenth International Conference on Machine Learning, pp. 41 48. Citeseer, 1993.
Rich Caruana. Multitask learning. Machine learning, 28(1): 1997.
Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. In NeurIPS, 2022.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning learn around a common mcan. Advances in Neural Information Processing Systems, 31, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Icee, 2009.
Lucio M Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. Tn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (80.18653/vl w1/l/11///1111/ 1423. URL
Noam Eshed. Novelty detection and analysis in convolutional neural networks. Master's thesis, Cornell University, 2020.
Abolfazl Farahani, Sahar Voghoci, Khaled Rasheod, and Hamid R Arabnia. A brief review of domain adaplation. Advances in dala science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pp. 877-894, 2021.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel pro- gramming for hyperparameter optimization and meta-learning In International Conference on Machine Learning, pp.
PMLR, 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (ods.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 2015. PMLR. URL https://prooceddings mlr. www..rree//33... gtt.33//.nnnnn..hmmmm.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Comnor Leahy. The pile: An 800gb dataset of diverse text for language modeling, CoRR, abs/2101.00027, 2021. URL
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Newral Information Processing Systems, 33:3290 3300, 2020.
Saced Ghadimi and Mengdi Wang- Approximation methods for bilevel programming. arXiv preprint arXiv:1802. 02246, 2018.
Amirata Ghorbani and Jamos Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242 2251. PMLR, 2019.
David Grangier and Dan Iter. The trade-offs of domain adaptation for neural language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2082, pp. 3802-3813. Association for Computational Linguistics, 2022. URL https://aclaattooooog/// org/2022. acc/2oo2..cl-
Suriya Gunasckar, Yi Zhang, Jyoti Aneja, Caio César Toodoro Mondes, Allic Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kaulfmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint Xim:2000 1 1644,, 2023.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipla, Kyle Lo, Ix Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognilion (CVPR), June 2016.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for jax, 2020. URL hllp://gillubb. com/google/foo /oozlle/fazz 1, 2020.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based angmentation: Efficient learning of angmentation policy schedules. In International Conference on Machine Learning, pp. 2731 2741. PMLR, 2019.
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipu- lation for augmentation and weighting. Advances in Neural Information Processing Systems, 32, 2019.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. In ICML, 2022.
Dan Iter and David Grangier. On the complementarity of data selection and fine tuning for domain adap- tation. CoRR, abs/2109.07591, 2021. URL http::///arii..oorrb//////0////
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. A data-based perspective transfer learning. arXiv preprint arXiv:2207.05799 2022.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313. PMLR, 2018.
Bojan Karlaš, David Dao, Mattco Interlandi, Bo Li, Scbastian Schelter, Wentao Wu, and Ce Zhang, Data debugging with shapley importance over end-to-end machine learning pipelines. ar preprint 111111244.11311, 2022.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with impor- lance sampling. In International conference machine learning, pp.
PMI.R, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885 1894. PMLR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Tan Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distri- bution shifts. In Marina Meila and Tong Zhang (cds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR, 18-24 Jul 2021. URL httts::/
mlr. wwwpss/vve///ollie/eeeee
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning, pp. 5468 5479. PMLR, 2020.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. Reinforcement learning based curriculum optimization for neural machine translation. arXiv preprint arXi::1903.0000 2019.
M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In J. Lafferty,
C. Williams, J. Maylor, R. Zemel, and A. Culotta (eds.), Aduances in Newral Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL
2010 220////1188800066611100387887000
pdf
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume 1, pp. 3, 2008.
D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Rev1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5(Apr):361 397, 2004. URL http://www. jmlr org/ https////oeeee/eee//////eeeeee
pdf
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugmentt Advances in Newral Information Processing Systems, 32, 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradddnn learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021.
gradient descent for multi-task
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin
Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 11206 of Leclure Noles in Computer Science, pp. 185 201. Springer, 2018.
Nada Malic, Isabelle Guyon, J Denker, and Vladimir Vapnik. Writer-adaplation for on-line handwritten character recognition. In Proceedings of 2nd International Conference on Document Analysis and Recog- nition (ICDAR 193, pp. 187-191. IEEE, 1993.
Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Popers, pp. 220 224, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https:///cclaathoolggy. wrw.:looyy.or//1--00000000
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoe V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. ar preprint arXiv:1811.07055 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Jay M Patel. Introduction to common crawl datasets. In Getting Structured Data from the Internet, pp. 977-324. Springer, 2020.
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural compulation, (00:144-1660 1994.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advonces in Neural Information Processing Systems, 199199,0-19930, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Iyy Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: /ttpa:///axxvvoooo org. htt//ab/11900..000 10084
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334 4343. PMLR, 2018.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Tryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Chongqing Zong, Fci Xia, Wenjic Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeling of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3118 3135, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ vl/2021.ac--onng22 243. URL
243
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning differentiable curriculum. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceeddngg nwwrsss.eeiiiiccoeee /ctpp/s..c//aapp////////e/e 99611000a55556666666646966ee
Paper.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8179-8186, 2022.
Steffen Schncider, Alexci Bacvski, Ronan Collobert, and Michael Auli wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXic:1904.00 05866, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Lingwisties (Volume 1: Long Papers), pp. 1715 1125, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1 1162 URL https://aclanthooooy
org/P16-116 1162
Burr Settles. Active learning literature survey. 2009.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- Learning an explicit mapping for sample weighting. In NeurIPS, 2019.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoob, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adríà Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xiv preprint or arXiv:2206.04011 2022.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer, 1998.
Marlies van der Wees, Arianna Bisazza, and Christof Monz. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1400-1410, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.1 (0.18653/v1 /1/11/117/ URL htttp:://alaanthollla http:///ddaattooooooooor///11111 org/D17-1147
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.08762, 2017. URL http:
7arxivv. /erxiv.rrg/oooo abs/ 1706. 03762
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 133 143, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18633/v1/W18-66 6314. URL https://aclanthoology org/W18-631
Xinyi Wang, Hicu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Ncubig. Optimizing data usage via differentiable rewards. In International Conference on Machine Learning, pp- 9983 9995. PMLR, 2020.
Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp. 10842 10851. PMLR, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Newral Information Processing Systems, 33:5824 5836, 2020.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi Revisiting few-sample BERT fine-tuning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee net, 2021. URL
Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data. In 2021 IEEE/CVP International Conference on Computer Vision, 1CCV 2021, Montreal, QC, Canada, Oc- tober 10-17, 2021, pp. 705-714. IEEE, 2021. doi: 00.1199/CCC4488222222000007 URL https:
//doi.org/10.
htodloorg//..110//ccc/a833 1109/ICCV48922. 200.20222000
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss mini- mization. In international conference machine learning, pp. 9. PMLR, 2015.
Linchao Zhu, Sercan O Arik, Yi Yang, and Tomas Pfister. Learning to transfer learn: Reinforcement learning- based selection for adaptive transfer learning. In Furopean Conference on Computer Vision, pp. 342-358. Springer, 2020.
Barret Zoph, Ekin Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoc Le. Learning data augmentation strategies for object detection. In European conference on computer vision, pp. 566 583, Springer, 2020.
A Common Settings in Transfer Learning
The transfer learning literature defines various scttings leverage training data from different task and or distribution. Although not all papers use the same definitions, Table [I presents the most common settings with reference to the urterature supporting these definitions.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 11: Classical Transfer Learning Settings
Transfer Learning leverages source distribution in order to perform better target distri- bution (Trrun & Pratt 1998).
Multitask Learning improves generalization by loveraging the information contained in the training signals of related lasks (Caruana 1993 1997
Domain Adaptation aims to improve accuracy on target distribution with insufficient labeled data by leveraging a model trained on different but related source distribution (Farahani ct al. 2021
Unsupervised Domain Adaptation considers the setting where labeled source domain data (x, y) are available for training, while only unlaboled (x) data from the target domain are available (Ganin & Lempitsky 2015).
Distribution Shift considers that the lest distribution is different from the training distribution, usually in the context where the model cannot be retrained to adapt to the new test conditions (Koh et al. 2021h.
Gradual Distribution Shift is online setting where the training distribution progressively evolves (Kumar et al. 2020
Covariate shift corresponds p(x) is different at training and test time, while the posterior distribution p(y[x) does not change (Bickel et al. 2009
predictive setting where the distribution over the input features
Label shift corresponds to a predictive setting where the class prior p(y) between train and test changes but the conditional distribution p(ylx) assumed identical (Garg et al.] 2020
Fine-tuning is a specific domain adaptation technique which considers training a model on the target domain from an initial model trained on the source domain (Matic et al. 1993
Denevi et al. 2018 Zhang et al. 2021).
Zero-Shot Task Transfer addresses
tasks at test time without updating the model (Larochelle et al. 2008). It typically relies on way to represent novel tasks in order to condition the model, e.g. text prompts (Radford et al. 2019 Srivastava et al., 2022).
Few-Shot Task Transfer is similar to zero-shot transfer and does not update the model weights.
As difference, the task conditioning information provides few training instances along
with the description of the tasks (Radford et al.) 2019 Srivastava et al. 2022).
B Multi-Task Extension
We exlend the proposed framework to the multi-task setting, when there are several downstream specific tasks for which one seeks good model. We treat the outer loss as the average loss over the specific sets: DI DJ Dxpecific set of I' specific sets. In Equation 2 we simply use
let D1 D Uspecifir....
T
Cspecitic (0" (a))
Σ Σ
((z::0((a))
(5)
T=
F'ED!
specitic
so that the outer loss minimization leads to good performance on average over the specific sets. Algorithm is adapted to this case by sampling specific batch Bepocitic at random between each specific set; the rest
19

---

Published in Transactions on Machine Learning Research (10/2024)
Table 12: Multi-task experiment. "Dialogue" consists of the three sets opensubtitles, openwebtext2, and stackexchange, "Academic" consists of the sets arxiv, pubmed, and wikipedia.
Method
"Dialogue"
Avg.
opensubtitles
openwebtext2
stackexchange
Baseline
1.157
1.158
1.201
1.113
Classifier
1.148
1.167
1.154
1.122
CDS
1.140
1.180
1.163
1.077
SOBA
I 1.129
1.144
1.204
1.039
"Academic"
Avg.
arxiv
pubmed
wikipedia
Baseline
0.966
0.994
0.942
0.960
Classifier
0.896
0.980
0.815
0.892
CDS
0.894
0.947
0.807
0.929
SOBA 0.908
0.945
0.848
0.931
of the algorithm is identical. We conduct such an experiment in the Language Modelling setting described in (subsoction 5.2] where the specific sets consist of T 3 datasets from the Pile dataset. We consider two different subsets of 3 specific sets: "dialogue" sets (opensubtitles, openwebtext2, stackexchange and "academic" sets farxiv, pubmed, wikipedia). Tn Table 12 we report results after fine-tuning on the specific loss that is, models are fine-tuned on a mixture of the T = 3 datasets. We observe that the bilevel methods and the classifier both improve over the baseline. As expected, the resulting models are good on each dataset but are not as good as models pre-trained and fine-tuned on one dataset only, as shown subsection 5.2
C Seper-parameters
Table 13: Hyperparameters for the LM experiment
Hyperparameter
Value
batch size
128
dropout_ran _ate
0.1
abatch_sizae
16384
optimizer
adam
learning_res gateeeeareeeeee
0.002
meta_ Leta_learning_raat rate
0,001
meta_ oit__oppimizer
adam
num sam_steps
300000
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 14: Hyperparameters for the translation experiment
Hyperparameter
Value
batch_st
256
dropout_ran _ate big_batch_size
0.1
2048
optimizer
adam
learning_ rate
0.0002
meta_learning.c
0.001
deta_optimize
adam
num num_steps
500000
Table 15: Hyperparameters for the vision experiment
Hyperparameter
Value
batch_size size
256
ba_baahchsize
2048
optimizer
SGD + momentum
learning_ran mate
0.05
meta_
Lrra_learming_ree
0.0001
meta esta_optimizer
adam
num_steps
500000
21


Reviews:
[{"review_id": "c5AwdeIxR8", "reviewer": "Reviewer_YaMd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A unified theoretical framework and systematic evaluation of data selection methods for domain adaptation, a critical topic in language modeling.\n2. A systematic evaluation of existing optimization-based data selection methods, revealing a few important findings such as the complementary effects of pertaining data selection in conjunction with finetuning, the ineffectiveness of loss-based selection and model-free selection, the strong overall performance of SOBA (a second-order method), the critical impact of sampling method with weights, the ineffectiveness of curriculum learning, and a novel indicator for the effectiveness of bilevel optimization-based domain adaptation. \n\nWeaknesses:\n1. The writing clarity can be further improved. There are a few places where context organization is a bit messy. For example, in Section 2, what is the relation between contrastive data selection and sample reweighting? Why introducing it in the second paragraph of Section 2? In Section 5.1, it is recommended to introduce all baseline methods along with the bilevel optimization methods in a unified format and style, e.g., making method names bolded and leading the paragraph. Do all methods experience the same fine-tuning phase? If so, mention in Section 5.1. Why does DDS / Anograd / SOBA have a \"Sparse\" prefix in tables?\n\n2. More in-depth analysis would be great. For example, why are data selection methods generally inefficient in image classification domains? Is it due to a gradient misalignment or failure to discriminate different examples via gradient dot product? I would suggest looking into the numerical magnitude of $a_{norm}(x, B_{specific})$ and $a_{norm}(x, B_{generic})$.\n\nOverall, I feel the submission might be limited regarding novel research contributions, but it is a comprehensive study with a potentially profound impact, especially for practical use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission provides a systematic empirical study of bilevel optimization methods for matching pretraining distribution to specific distribution for large neural networks. Under a unified formulation, the submission summarizes a few data selection methods (DDS, SOBA, Anograd) by a bi-level optimization framework and a unified algorithm template. Experiments on three domains (mainly language modeling tasks, including multimodal alignment tasks) demonstrate the overall effectiveness of bilevel data selection methods and a few insightful findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the suggested changes in Weaknesses section, here are some minor points:\n1. In Table 1, it could be great to compute the number of parameters so we can make a direct comparison with LLM literature.\n2. In Page 7 last but one paragraph: \"better results that mixing+fine-tuning\" that -> than\n3. In Table 8, what model is used when computing the loss gradient?\n4. In secit5on 7, \"litterature\" -> \"literature\"", "broader_impact_concerns": "No statement-requiring ethical implication from my point of view.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ToLXP2JZf7", "reviewer": "Reviewer_1v3R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper conducts thorough experiments and interesting studies in the discussion part. Many of my concerns when reading the paper have already been addressed in the discussion.\n\n\nWeaknesses:\n1. Although the proposed setting could generalize many other learning schemes, the experiments are most relevant to transfer learning and domain adaptation. Experiments in different settings might be necessary to demonstrate the method's generalization to schemes such as continual learning and multi-task learning.\n\n2. I am a bit concerned with the setting and its relationship to meta learning. It seems the practical scenario is that a large generic dataset will be available. However, in the experiments, the paper does not start from a model that is pre-trained on the generic dataset. If a new specific dataset comes, the method will need to train on the new task together with the generic dataset. If there are many different specific datasets, the problem falls under meta learning.\n\n3. The proposed method essentially aims to identify samples in the generic dataset that share similar gradients as the specific dataset (in an online fashion during the training process). Moreover, as also pointed out in the paper, the sampling method performs worse than the top-k selection method. The intuition behind those is not well understood. It seems that the method is finding the most ''simple'' samples from the generic dataset to help the specific dataset. If there are really generic samples with very similar gradients, they probably do not give extra information in addition to the specific samples. It seems to me that samples that are similar but not the most similar are most beneficial. Moreover, the method is used as a pre-training step and a fine-tuning step on the specific task boosts the performance quite significantly. Then for a pre-training step, why is learning the ``simple'' samples beneficial?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of learning from a limited specific dataset with a large generic data set available, which is, however, of a different distribution. The paper models the learning as a bi-level optimization problem. Particularly, the paper uses a weight model to assign weights to the generic dataset samples so that the learned weights from the weighted generic dataset give the best performance on the specific dataset. The paper applies the proposed method to three tasks and shows some improved performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses part.\n\nMinor:\n1. Why are the proposed methods called ''sparse ***'' in the result tables? \n2. I appreciate the authors' efforts, but Table 3 is still very hard to parse.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VrQNwQsV6O", "reviewer": "Reviewer_NPug", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths.\n1. The studied problem is quite important. The authors have clearly described the motivations.\n2. The idea of bi-level optimization makes sense and is suitable for this problem.\n3. The authors provided solid theoretical analysis for their method.\n\n### Weaknesses.\nI have also carefully read the reviews in the last round, and I find the last-round reviewers have some common concerns with me.\n\n1. The scale of the experiments is limited. As mentioned above, the studied problem is very important because today's model size is very large (we must use large retained models due to their superior performance). Therefore, to validate the effectiveness of the proposed method, experiments with large-model and large-scale datasets are required.\n2. The proposed method is close to gradient alignment; in other words, it is based on gradient alignment without many new insights.\n3. The authors still do not provide codes for reproducibility.\n\nOverall speaking, the concerns proposed by the last-round reviewers are still unresolved. I observed two versions of manuscripts; there are not enough workloads or new high-quality changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about resolving the gap between pretrain tasks and the downstream task. This topic is very important, and the authors have selected a good research problem.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to above comments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a resubmission, and I served as the AE for the previous round. The paper presents a method for selecting specific data as a bi-level optimization, applied to large-scale datasets. Most reviewers have responded positively to the revisions, though some concerns remain.

> The scale of the experiments is limited. As mentioned above, the studied problem is very important because today's model size is very large (we must use large retained models due to their superior performance). Therefore, to validate the effectiveness of the proposed method, experiments with large-model and large-scale datasets are required.

I checked the experiment and think the authors addressed this issue by incorporating several large-scale datasets in their revised experiments.


> The proposed method is close to gradient alignment; in other words, it is based on gradient alignment without many new insights.

Insights are valuable, but the primary focus of TMLR is technical soundness.As this paper demonstrates, the proposed method is technically sound and practical. Further as others have noted, the paper is interesting and important, despite the relatively modest novelties.

> do not provide codes for reproducibility.

This is crucial, and I strongly suggest that the authors submit the complete code in the camera-ready version to address this issue.


**Decision.** After reviewing the resubmission and considering the feedback, I believe the authors have successfully addressed the major concerns. Therefore, I recommend a minor revision, with the request that the authors provide the full code for the camera-ready submission.
Decision: Accept (poster)