A Bayesian Approach to Data Point Selection
Authors: Xinnuo Xu
Minyoung Kim
Royson Lee
Brais Martinez
Timothy Hospedales
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian
LLM
Data selection
Data unbalancing
Data denoising
Domain adaptation
Primary area: optimization_for_deep_networks
TL;DR: BADS (Bayesian Data Point Selection)

Abstract: Data point selection (DPS) is becoming a critical topic in deep learning due to the ease of acquiring uncurated training data compared to the difficulty of obtaining curated or processed data. 
Existing approaches to DPS are predominantly based on a bi-level optimisation (BLO) formulation, which is demanding in terms of memory and computation, and exhibits some theoretical defects regarding minibatches.
Thus, we propose a novel Bayesian approach to DPS. We view the DPS problem as posterior inference in a novel Bayesian model where the posterior distributions of the instance-wise weights and the main neural network parameters are inferred under a reasonable prior and likelihood model.
We employ stochastic gradient Langevin MCMC sampling to learn the main network and instance-wise weights jointly, ensuring convergence even with minibatches. Our update equation is comparable to the widely used SGD and much more efficient than existing BLO-based methods. Through controlled experiments in both the vision and language domains, we present the proof-of-concept. Additionally, we demonstrate that our method scales effectively to large language models and facilitates automated per-task optimization for instruction fine-tuning datasets.

A Bayesian Approach to Data Point Selection
Xinnuo Xut* Microsoft Research Cambridge xinnuoxu@microoee www.oft.cccccoooooooooo.ooooo..o com
Minyoung Kim
Samsung Al Center Cambridge, UK mi kim21 @gmail.
Royson Lee
Brais Martinez
Samsung Al Center Cambridge, UK
Samsung Al Center Cambridge, UK brais wwwt mart@samsung
royson. www@seesungg...oom com
Timothy Hospedales
Samsung AI Center Cambridge, UK
University of Edinburgh, UK
www.ppdalleee.. es@ed. ac uk
Abstract
Data point selection (DPS) is becoming critical topic in deep learning due the ease of acquiring uncurated training data compared to the difficulty of obtaining curated processed data. Existing approaches DPS are predominantly based a bi-level optimisation (BLO) formulation, which is demanding in terms of memory and computation, and exhibits some theoretical defects regarding minibatches. Thus, we propose novel Bayesian approach to DPS. We view the DPS problem as posterior inference in novel Bayesian model where the posterior distributions of the instance-win wise weights and the main neural network parameters are inferred under reasonable prior and likelihood model. We employ stochastic gradient Langevin MCMC sampling to learn the main network and instance- wise weights jointly, ensuring convergence even with minibatches. Our update equation comparable to the widely used SGD and much more efficient than existing BLO- based methods. Through controlled experiments in both the vision and language domains, present the proof-of-coo sooof-of-coceppp Additionally, our method scales effectively to large language models and facilitates automated per-task optimization for instruction fine-tuning datasets.
demonstrate that
Introduction
Practical machine learning efficacy is heavily dependent on the choice, quality and quantity training data, especially so in the case of neural networks that can easily fit every detail of the training set. This leads to challenges from how to learn reliably with imbalanced data [22]. noisy data, noisy labels Similarly there is often key subset of data, which is most informative for given learning problem, but buried among a much larger set less relevant data. If the most salient data could be efficiently identified, learning could potentially be accelerated [15] All these challenges only growing in the era of large scale training on web-scraped data, where curation and gold-standard quality control are not feasible.
[66, and
Data Point Selection (DPS) algorithms aim address these challenges by filtering or re-ihinhting the training data reduce noise, imbalance, irrelevant background data and The most established
Part this work was completed while Xinnuo was affiliated with Samsung AI Center Cambridge, UK. Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

family of approaches umbrella, where one wraps the conventional learning problem with an outer loop that optimizes the dataset itself, so as to maximise performance on some validation set. These methods vary in the choice of their outer optimization variable (e.g., data point weights [4] 19 vs mini-batch [5]] 62) or reinforcement learning (1S)., and the customization of their losses and other design parameters for the different scenarios (e.g., label-noise [45] 41). etc). However, all the BLO approaches are quite expensive in computation and/or memory, which limits their applicability to the most salient use case
41 45 to DPS falls under the bi-level optimization or meta-learning
sampler [15), the method of computing meta-gradients (e.g.. reverse mode differentiation
of large models trained on large web data.
In this paper we revisit the DPS problem from the perspective of Bayesian learning. Rather than constructing expensive nested optimization problems whose convergence is hard to analyse, we treat it as a problem of inferring the joint posterior over the main neural network parameters and instance-wise weights induced by second weight-estimation neural network. This framework has several advantages in terms of being more efficient and scalable than typical BLO competitors and having a clear convergence guarantee. It is also able to address a variety of DPS-related problems- from noise and imbalance to data curation- - within a single framework.
Our empirical results present proof of concepts for all these capabilities on a variety of learning tasks in vision and language. We also show a use case of automating Large Language Models (LLMs) instruction fine-tuning (IFT) data curation for specific downstream tasks. The available IFT datasets are large, diverse, and of varying quality. This means that key activity for natural language processing (NLP) researchers and developers is often finding the right composition of IFT data sources to optimize particular use cases. Our framework can automatically resample and curate the wide array of available auxiliary IFT data to optimize performance for each NLP task of interest. To our knowledge, no BLO alternative has been demonstrated on aillio--parametr scale LLMs. We name our approach BADS (Bayesian Data Point Selection)
2 Our Approach
2.1 Problem Setup
For the DPS problem, assume that we given two datasets: the train set D, 12=12 and the meta set 2m {=""" Jäm Each data point z, can be an input-target pair 2; = (z,, yi) in the traditional (llbes-)l data scenarios. But in the autoregressive generative model scenarios (e.g.. LLM), 2; is simply a sequence of tokens in which the inputs and targets are rather implicitly defined (e.g.. all the tokens up until current time as input and the next token as target). We will use the notation [(2,; 0) for the loss of the model on the data point z,, which must be well-defined in both scenarios.
The meta dataset Dm is considered as in-domain, meaning that the distribution of D' matches that of the downstream test task of interest. The size of Pm, denoted by Nm. is typically small, due to the cost of curation/annotation processes in practice. The train dataset D; consists of out-of-domain samples, possibly noisy, imbalanced, and uncurated, but the size N₂ is usually large. The goal of DPS is to select a (soft weighted) subset of the train set D, with the guidance of the meta set D.,. so that the model trained on the selected train and meta dataset points performs well. Typical baselines include training with the meta-set alone, or union of meta and train sets.
Perhaps one of the most widely adopted DPS techniques is the bi-le optimisation (BLO) formu- lation of the problem 41 58 Letting wi (20) 0) be the weight (or importance) variable associated with the training data point 2f, the main intuition to find the weights w E R.N. such that the model 0 trained with the weighted train data with weights w yields the best performance on the meta set. More formally, this leads to the following BLO problem:
min Σ [l(e (w)) s.t. ('w) arg min Σ w. - I(2;;)). WER
(1)
However, critical drawback is that solving this difficult problem is costly in computation and/or memory, and unrealible due the practical heuristies required. Typical BLO solutions to obtain the
'The code for this paper is available at

---

hypergradient di dw rely on approximate Hessian estimation or reverse mode differentiation with few-step SGD approximation of the inner optimisation. Aside from cost, for practical neural network implementations computed over minibatches, there is no theoretical guarantee for convergence.
2.2 (Our Approach) Bayesian Data Point Selection (BADS)
We view the DPS problem from a completely different per- spective, and tackle it via Bayesian learning. Our model's generative process, that is, the graphical model representation, is depicted in Fig. [ The main neural network model parame- ters set 0 is random variable, which can generate data points in the meta dataset Dm (precisely speaking, the backbone generates the target part of each data point). To make use of the train set Dr in an appropriate way, we constrain & to follow prior distribution governed by the weighted train data with weights w E R. which are also random variables. Before observing the meta set Dm. the weight vector w follows prior distribution p(w) - a specific distributional choice for p(w) will be discussed later. Given w and Dr our backbone has to be compatible with the weight data {ww, x() = This can be interpreted as placing a weighted-datte-driven prior on 0, more specifically,
W
Dr
D m
Figure I: Graphical model for BADS. Shaded nodes, representing curated (Dm) and uncurated (D:) data, are evidence. Unshaded nodes, includ- ing model 0 and instance weights w, are random variables.
N₂ pl ur 210
(Weighted-dates dreighted-data-ddrive prior) p(0|w, D.) x p(0) II
where p(0) is a base prior (e.g., O-centered Gaussian that amounts to weight decay regularisation), and p W., z
can be defined from the loss, e.g., exp(-w;:I(+:) 0)), following the conventional
tricks 36 25
Then given 0, the meta data are generated following the likelihood defined as:
N 10) II exp((1(=2];
(Likelihood)
(3)
The equations 2 and (3) fully constitute the prior and likelihood for our Bayesian model. Our ultimate goal is to describe the distributions of and w after observing all evidences DD and Dm. which boils down to the posterior inference p(0, w/ DD D.n). Formally, we have:
(Posterior)
p(0, w DD, Dm)) x p(w) p(0] ne Dz) p(Dm)00)
The detailed derivations for Eq. (4) can be found in Appendix A However, is widely known that (4) does not admit any closed-form expressions. One main difficulty arises from the intractable
normalizing constant (4)
Stochastic Gradient Langevin Dynamic Sampling, For computationally efficient posterior in- ference, we adopt the stochastic-gres gradient MCMC technique, specifically the Stochastic Gradient Langevin Dynamic (SGLD) sampling (55). Applied to our model, we can obtain samples from the posterior p(0, w (D, Pm) ) by running the Langevin dynamic system (until convergence, i.e., mixing):
10, w] + [0, w| + Vo
9. log wD,,, Dm)) + eVT.. N~,(0,1)
(5)
where 77 is a small (constant) step size. There are two critical benefits: i) Since we differentiate the pog-oortrroo, the difficult normalizing constant in ( will disappear; ii) The update (5) is essentially gradient descent with additive Gaussian noise, leading to a computationally efficient update.
Going one step further, even though the leg-posterior involves the entire train data (and entire meta data), shown in (55) that the muohasstc-gradienn version (SGLD) that replaces the whole batch likelihood with minibatched one, theoretically guarantees that the SGLD update converges to the posterior samples. More specifically, the SGLD update equations (one for 0 and the other for w) can be written follows:
0 - 0 + Vo logp(0) N. (w, 1(2 0)] - Nm EEn.... [[)]:)0]] + co T
"
(6)
2
w + w + v. logp(w) - N₂ BB.B, wm 1(21 ((4:000)) + cw CWV
2
3

---

where B3 and Bm are minibatches from D, and Dm, respectively, and co, cw ~ NO I) are indepen- dent Gaussian samples.
Repeating (6) and (7) for a sufficient amount of iterations (until we reach good mixing) leads us to posterior samples (B, ut). There are several options to take these samples for a final model for test prediction. One option is to collect latest M samples (either consecutive collection or thinning to take every kth samples) from the iterations, and either take the average as posterior means or perform full Bayesian treatment with the collected samples. Alternatively, we can just take the last single iterate (0, w) as a point representative for the posterior distribution. For simplicity, we take the latter approach, which also works well empirically.
2.3 Interpretation and Benefits
Interpretation We discuss several intuitions and implications of our proposed approach (Eq. 6 7 First, looking at the update Eq. (6). our model essentially updates in a way that decreases the loss the combined data of the whole meta data points and the weighted train data points with the current weights. This is a fairly intuitive strategy provided that the weights are properly determined. Then the next question is how the weights are determined. If we inspect the update (Eq. 7 and take the gradient of the expected loss term with respect to w directly, we see that: i) those train data points z[s with smaller losses at current backbone 0 will get higher weights w;s; ii) those train data points z/s with larger losses at current backbone 0 will get lower weights wys. This essentially means that our model performs loss alignment for DPS In the course of training/update, once the backbone 0 enters a good regime in the parameter space such that 0 can assign (valid) low loss values on the in-domain meta data points, then starts putting high weights on those train data points that have low losses under the current backbone. In other words, the model will assign high weights to those train data points that are well-aligned with the meta data points in terms of loss.
Benefits over BLO Our Bayesian approach provides several benefits over BLO: (1) Efficiency. Our SGLD is efficient so does not require computationally demanding Hessian computations like implicit function theorem based methods (cf: (19] or huge memory demand like reverse-mode differentiation methods [41] [9]. (2) Sparsity. Our method straightforwardly achieves sparsity on the w weights allowing efficient sample selection unlike [19]. (3) Reliability, BLO-based methods rely on approximations (truncation, or Hessian approximations) for practical feasibility that make finding optimal solutions unreliable. Our straightforward Bayesian approach has reliable convergence properties thanks being standard application of SGLD.
Convergence of our SGLD algorithm In Appendix C we provide a theorem showing that our SGLD algorithm converges the true posterior. Our analysis is based on (64) where we make some adjustments for our case.
2.4 Implementation Details
Choice of Priors For the base prior p(0), i.e., the prior before being driven by the weighted data, we adopt O-mean Gaussian, which amounts to adding the weight decay regularisation for f. For the weight prior p(w), we have made careful design effort to come up with a viable sparsity inducing prior. Although encouraging sparsity in learned weights is ideal to avoid overfitting, during our initial experiments have found that most of the weights eventually tend vanish to 0, which is not what we actually want. We need to be able to impose both sparsity and a certain level of non-zero weights. To this end, first introduce hyperparameter (e.g., 0.01) for the target sparsity level that want to attain. Roughly saying, among the N₂ training data points, we aim to select [N,-3]. J. For the (soft) weights, we impose EiED, Wi [N. which can be encoded in the prior form as:
p(w) x (E, w. (...S) /202
(8)
where a controls the strength of the regularisation. One technical difficulty in directly plugging S into the weight update (7h is that we have to load the whole {wx JNN in memory for backprop. To avoid this issue, we use the following fact:
Σ IED
wi R3
w; + (N, - -B.1)-
(9)
iEB.

---

where w represent historic running average of the entire weights. We basically build a computation graph only for the first term of batch B, weight sum, and regard the (historic) running average of the entire weights as constant during backprop. After each SGLDD iteration, we update the running weight average with the new updated weights on the recent batch. We use the simple averaging scheme for the running average. To approximate the average weight w precisely, we only conduct the average over the most recent Saeg step.
Introducing impact constants In the SGI.D principle, we have the log-likelihoode terms that are proportional to the sizes of the datasets. In particular, we have N; and Nm. in ( However, this scheme does not properly capture our preference to the in-domain meta data set contrast to the noisy, out-of-domain train data set. To this end, we introduce the impact constants (hyperparameters) in the update equations where we downweigh or upweigh the loss terms of train and meta sets.
Weight Network Instead of directly optimising individual weights w;, we can consider a weight network, w; = w(2 o). a neural network with parameters & that takes the train data point 2 as input and returns its weight w; as output. We can then regard & as random variables and the w update equation can be modified accordingly for e update straightforwardly. This weight network approach can be useful for smoothing/regullrisiin of neural networks. Furthermore, if one needs to supplement the train dataset with extra new train samples after the model training, the learned weight network can be used for assigning weights selecting samples from the new set, without retraining the whole model from the scratch. The posterior distribution similar to Eq. is derived in full detail in Appendix
the output weights thanks to the smooth functional property
3 Experiments: Proof of Concept
In this section, we assess the effectiveness of the proposed BADS method in three critical scenarios where DPS is essential: Data Balancing, Data Denoising, and Efficient Learning. We begin by introducing the baseline systems and then present the experimental results for each of these scenarios.
3.1 Baselines
There are three types of DPS setups with different supervision signals:
Unsupervised DPS selects data without the guidance of a held-out meta set 44 42
Instead, is
guided by human-defined hypotheses, such "'hallenging examples improve model performance". This approach aligns with curriculum learning. We include the online variant of AskLLM [42]. i.e. AskLLM-O, in our baseline comparisons. It selects examples from training set by querying pretrained LaMALLaMA 3B to obtain the sampling score for each training sample
self-uupee DPS selects data with the guidance of held-out meta set. However, the meta set does not share the same data distribution as the targeted test set [13] 4 Typically, the examples in the meta set are selected from the training set based on specific hypotheses. such as "learnable examples enhance model performance". We include two approaches in our baseline comparisons: Contrastive Data Selection (CDS) 49 is tailored for data denoising. The algorithm assigns weights to each data point in DD according to the difference between the denoised and the noisy log probability, predicted using denoised and noisy model trained on a clean dataset and an uncurated dataset, respectively. These weights are then used to sample data points from minibatches in the training of LM [ on a limited portion of D, to calculate learnability scores for the remaining training data points
Similar to CDS, ClassAct utilizes small proxy models trained
Meta-set guided DPS selects data with the guidance of a small meta set that shares the same distribution the test set, aiming to train a model that excels specifically on the target test set. The test set may encompass one or multiple downstream domains or tasks. This DPS is closely related to meta learning, domain adaptation, and transfer learning. Current methods primarily rely on Bilevel Optimization (BLO) for purposes such as data denoising [19 37]. data balancing [41).
Siice 47 showed that unsupervised DPS can amplify class imbalances, and open-source LLMs generally do not accommoaate vision input, compare to M- only in the L.M finc-tuning use case in Scction 4 In our experiments, we use Mixing and Meta_only as the noisy and denoised model.
TTo fairly compare the selection mechanism, replaced their meta using our meta set.
5

---

TesSRCC MNIS
TeaNAGC for CITAR--10
Teat VetriLG
pade
2005
Weighta Patroui Prediced by SADS
Weighta WadNLLG Prodicadd by BLD
Weights Dass Pornts Wer CDS
0.50
40.00 40/2
43/00 <atoe
40000 407122
<0750 -0300
Figure 2: across the three scenarios throughout the training phase, with x and y axis denote the training steps and the evaluation metrics, respectively. The bottom row visualizes the model-predicted weights of data points in each mini-batches in the final 2000 steps in WebNLG training (scenario 3). x and axis show the training steps and average weights, respectively. Data points in blue color are expected to get higher weights compared to their counterparts (in red color).
Proof-of-Concept experiment results. The top row displays the overall test performance
and efficient learning 28 online BLO
Considering both performance and code availability, we use the as our baseline. Our approach, BADS, also falls under this category.
59 29
To ensure a fair comparison, we also introduce several baselines that train the backbone models using different combinations of the meta set and training set: Mixing trains the model using a combination of the train set D, and the meta set Dm- Meta Meta_Only trains the model exclusively on Dmn Random Select uses Dm combined with a randomly selected subset from D.. Duplicate Meta utilize Dr along with multiple copies of the meta set, duplicating Di,. until it matches the size of D,.
Note that, the selection ratio/sparsity level in Ask.I.M--, ClassAct, Random_ Select, BLO, and CDS is the same as in BADS
3.2 Scenario 1: Data Balancing (MNIST)
In this scenario, we assess the model's capability to manage an imbalanced train set, D,, Despite being trained on this imbal- anced dataset, the models are expected to perform effectively on a balanced test set. Following the setup in [1], we use the standard MNIST handwritten digit classification dataset [33] to create class-imbalanced binary classification task. A total of 5,000 images from classes 4 and were selected as the truin set D., with class 9 dominating the training data distribution (4.975 examples) and class having only 25 examples. A balanced meta set Dm is created by selecting another 25 examples from each of these two classes, ensuring no overlap between D; and D,n- The models are tested on the original MNIST test set, including only classes and 9.
NNST ACC with diffrrent number exarrples per class mela
vara
20
Figure 3: The MNIST test accuracy when trained with meta sets in vary- ing sizes (x-aixs).
In this scenario, the loss function is defined using the standard binary cross-entropy loss. Following [1], all classification models are LeNet5 [32] The training is conducted on a single GPU, using
SGD with a fixed learning rate of 1e-3 and a mini-batch size of 100, over a total of 15,000 steps. In
BADS, the weight network implemented as a single-layer Feedforward Neural Network (FNN) with sigmoid activation function. It takes the top-layer image embeddings from LeNet5 input
dwaaieltaaaeeaa https://github. ttnn/danietaan0/// http//eeaannnnngeee tes--to-rrreeeeee reweight- examples... git
We acknowledge existence recent work on DPS with Z or without (59 40] meta meta-data alignment.
However, all these studies use two-stage pipelines, where data points are selected offline and then used in the final model training. Since our approach employs an online selection method, i.c. dynamically selecting data while the model is under training, we have only chosen baselines that follow the same style.

---

and outputs weight w; E [0.1] for each image. The learning rate for the weight network is 1e-3 and the target sparsity level is 0.005. Other hyperparameters, including those in the baselines, are detailed in Table Appendix H
3.2.1 Experiment Results and Ablation Study
The classification accuracy is presented in the top-left plot in Figure 2 Mixing and ClassAct, achieve over 90% accuracy even when trained with a highly imbalanced train set. Meta _anlyyyoyyaaa demonstrates that training with a small amount of balanced data yields significantly better performance compared to training with a larger but imbalanced training set (Mixing). Both BLO and BADS outperform non-DPS baselines in terms of both accuracy and convergence speed, with BADS further outperforming BLO by noticeable margin. CDS underperforms all the non-DPS baselines, which we believe is due to the mini-batch-level discrete data selection. When dealing with an extremely imbalanced train set, it is possible that the minority class might not be present in some mini-batches. Under these circumstances, the model is compelled to learn from the top examples from the majority class in each mini-batch, potentially leading to biased training outcomes. The top row in Figure 13 a3d the left plot in Figure 14 (Appendix F) visualizes the weights assigned to the examples each mini- batch by DPS approaches. All methods, except forClassAct, effectively assigns higher weights to the minority class than to the majority class, thereby directing the classifiers to focus more on the minority class in training.
All approaches, except for
We further evaluate the models' performance using meta sets Dm,, of various sizes, with 5, 10, 25, and 50 examples per class included I examples (5 per class), only BADS and BLO achieve over 90% accuracy on the balanced test set. As the number of available meta data increases, BADS consistently leads in performance. However, when sufficient meta data is provided, the gap between BADS and the other approaches narrows.
As illustrated in Figure 3| with a very limited number of meta
3.3 Scenario 2: Data Denoising (CIFAR)
In this scenario, we evaluate the model's ability to manage a noisy train set Dr Although the models are trained using a dataset with significant noise, they are anticipated to perform well on a clean test set. Our experiment utilizes the standard CIFAR 10-class classification dataset (30 Following the stan- dard CIFAR train/validation set split ratio, we first create a clean and balanced meta set Dm by randomly sampling 1000 examples from each class in the training data. Then, we use the remaining 40,000 examples to create a noisy train set DD by introducing noise based on the symmetric noise injection setup described in [37 S 21. We set the noise ratio to 0.5 and use
Tren0O by 27M0 Net nife-
0.30
Potta
2400
5000
Figure 4: The CIFAR test accuracy when trained with 80% noisy data.
the original CIFAR-10 test set in testing.
We use ResNet32 [23] as the backbone classification model. Training is performed on a single GPU using SGD with a fixed learning rate of le-1 1 and a mini- bini-batch size of 120 over 20,000 steps. The loss function is the standard multi-class cross-entropy loss. For BADS, the weight network structure retains the same as in scenario 1. However, in this scenario, data point weighting takes into account both the image and its associated label. We represent each label using a one-hot hot embedding, which is then concatenated with the top-layer image embeddings from ResNet32 and fed into the weight network. The learning rate for the weight network is set to le-4. Note that in 7 the gradient of 0 become large if N, is big. Therefore, we reduce the learning rate of the backbone classification model to le-4. The target sparsity level 33 is set to 0.8 10
3.3.1 Experiment Results and Ablation Study
The classification accuracy is shown in the top-middle plot in Figure 2 manage to achieve over 60% accuracy, even using a Irain set contaminated by 50%.
All methods, except for the
BLO approact [I]
Sinnee ClassAcr uses a similar approach to CDS but performs worse, we exclude it from this ablation study. Besides the primary results, we also present results for asymmetric noise in the top right plot Figure 11 (see Appendix(F.I). Additionally, we explore performance variations by substituting the weight network with the individual weights strategy outlined in [4]] (refer Appendix
"Thee convergence of BLO is slower than that of other approaches. To further explore this, we conduct an ablation study compare the learning curves for all methods in Appendix F.I
7

---

Notably, CDS, ClassAct and BADS deliver the highest three performances. with BADS surpasses all other methods by a noticeable margin. The bottom row of Figure 13 and the middle plot of Figure 14 (Appendix F) visualizes the weights allocated to the examples in each mini-batch by DPS approaches.
All methods, except for ClassAct, consistently gives lower weights to the noisy examples, guiding the classifiers to dis- regard the noisy examples during training. ClassAct assigned lower weights to the noisy examples during the initial stages of training. However, these weights exceeded those of the clean examples in the later phases of training. Interestingly, this behavior did not significantly impact the model's performance.
TestAAC
If we raise the noise label ratio in the train set to 80% (Figure 4). both BLO and BADS still lead the performance, with BADS exceeds BLO and non-DPS approaches by 15% and 20% in classification accuracy, respectively. CDS does not exceed the performance of non-DPS approaches and starts to overfit the noisy data after 15,000 training steps. When we lower the noise label ratio to 20% (Figure 5), all methods achieve a classification accuracy of around 80%. Although BADS continues outperform both DPS and non-DPS approaches, the lead is narrower
Figure 5: The CIFAR test accuracy when trained with 20% noisy data.
3.4 Scenario 3: Efficient Learning (WebNLG)
In this scenario, we assess the model's ability to adapt to new domains with very few data points. To demonstrate the robustness of BADS across various research topics and backbone models, we focus on the Natural Language Generation (NLG) task in this section. NLG [18] [2] aims to accurately generate textual descriptions from input tuples. An example is shown in Figure Appendix Dp. The English benchmark introduced in WebNLG 2020 [6] includes a training set spanning 16 domains and a test set covering 3 different domains (for details, check Apppendix Dp. We use the original training set (14,239 examples) as our train set D, and create a single clean and balanced meta set Dm by randomly sampling 30 examples from the WebNLG 2020 validation set in each test domain.
All backbone models are the encoder-decoder T5-small (2).. Training is on single GPU using Adam with fixed learning rate of 3e-5 and mini-batch size of 20 over 40,000 steps. The loss function is the standard negative log likelihood loss. In BADS, the weight network structure remains the same as in scenario 1. Its input is the embedding of the input sequence, represented by the contextual embedding of the "[EOS]" token, and the learning rate is set to le-4. The sparsity level 33 is 0.05.
3.4.1 Experiment Result
The BLEU scores are displayed in the top-right plot of Figure 2. BADS leads the performance, achieving 2
Methods
Mem (MB)
Time (s)
Base
9184,4
61.2
BLEU score advantage over the second-best system, Dupli- cate_Meta. and surpassing the remaining systems by more than 5 BLEU scores. The other three DPS approaches do not distinguish themselves from the non-DPS methods.
BLO
22361.3
113.48
CDS
9183.9
62.0
-wegght cale
+700.0
ClassAct
34821.36
269.81
AskLLM-O
32908.89
115.24
In this scenario, we opt for a more controlled examination of data selection effectiveness due to the big amount of domains in the train T3] We select three occupation- centric domains-Arhlere Politician, and Astronaut as our test domains. Additionally, we create meta set Dm by randomly selecting 50 examples in each of these domains from their WebNLG 2020 validation set. We then choose two distinct domains, Artist and City, for training. Artist, as another occupation- centrioco--eeeccc domain, shares a similar schema and vocabulary with the test domains, whereas City does not. Ideally, DPS should prioritize the Artist examples over those from City. The BLEU scores for text descriptions generated by the DPS methods are shown in Figure 12 AAppeddix F). These results reinforce the findings from the original experimental setup, with BADS outperforming both BI.O and CDS by over 10 BLEU scores. The weights given to the
-LLM call
+13932
BADS
14694.58
61.03
Table 1:
The average GPU memory and time usage over 100 steps. "Base" repre- sent all non-DPSs.
""ClasAAer uses a similar approach to CDS and has similar results. We exclude it from this ablation study. 13 Given that the train set spans 16 domains, it is difficult to assess the adequacy of the data selection behavior. We still show the weighting plots for the 16 domain in the bottom of Figure 13 in Appendix F
8

---

examples in each mini- batch are visualized in the bottom row of Figure 2 BADS effectively prioritizes the examples in the Artist domain. Instead, BLO fails to differentiate between the two domains, and CDS incorrectly weights the opposite domain higher. This illustrates that the effectiveness of DPS is linked the overall performance of the models.
3.4.2 Latency
We evaluate the average GPU memory and time consumption using the WebNLG task. Table 1 shows that the training time for BADS and CDS is similar to that of non-DPSs, while BLO and AskT L--O takes nearly twice as long, and ClassAct takes even longer. Note that CDS and AskLLM-O -O requires additional time to weight examples in the train set. Given that the WebNLG 2020 train set consists of 35,426 examples, we execute the weighting with a batch size of 20, CDS takes a total of 700 seconds. For larger-scale tasks, such foundation model fine- tuning (Section A. the weighting process is estimated to consume approximately 22 hours per million training examples. The offline scoring in AskLLM-O O takes around four hours in setup single NVIDIA A40 GPU. In terms of memory usage, CDS aligns with non-DPS approaches, while BADS and BLO require approximately 1.5 and 2.5 times more GPU memory, respectively. ClassAcí and A takes even more GPU memory.
4 Use Case: Large Language Model Instruction Fine-tuning
Instruction Fine-tuning (IFT) for LLMs is a practical application where all three mentioned scenarios are en- countered simultaneously. IFT data can be acquired through prompting LLMs 52 48 56). gathering existing
Methods
MMLU
ARCc
ARCe
HellaSwag
Mixing
25.16
33.79
64.65
51.97
Meta_ Only
25.51
32.08
52.23
52.07
Random Select
25.62
28.92
66.75
51.83
Duplicate Meta
25.30
33.28
65.15
52.76
CDS
25.62
21.16
60.14
51.68
Natural Language Processing (NLP)
ClassAct
24.90
31.91
67.34
52.09
benchmarks [35
54
(50 43
53 38
AsKLLM-O
25.55
35.15
66.88
53.71
56]. or employing human annotation
BADS
26.59
34.39
67.00
52.91
Noise is likely accu- mulate during each of these data col- lection methods. Furthermore, since NLP benchmarks often vary greatly in size, the IFT data typically lacks balance. Additionally, the IFT data does not include data points from the downstream tasks, leading to domain shift in testing.
152
63 48
Table 2:
Test accuracy of LLMs across four popular bench- marks in eval-harness [17. Checkpoint selection is using next token prediction accuracy the selection metric. Mix- ing represents standard IFT.
We use the same IFT data as 51 as our train set D., which is a mix of FLAN V2 B55. COT [4],
DOLLY [TO,, and OPEN ASSISTANT 31. Following (57 5) we focus on four downstream tasks:
MMLU 244,, which consists of multiple-choice questions across 57 sub-tasks, ARC-challenge/-easy
21. and WellaSwag (61] Following [57] 5 examples were selected from each sub-task to create the
meta set Dm for MMLU, totaling 285 examples. Additionally, following [5], for the other tasks, we randomly chose 25 examples from their validation set to create the respective meta sets. To facilitate checkpoint selection, we additionally create validation set of equivalent size to the meta sets.
Due to limited computational resources, we LaAALLaMA 3B as the backbone model. Training uses one A40 GPÚ utilizing Adam with a fixed learning rate of 3e-5 and a mini-batch size of 3. In BADS, while the weight network remains the same described in scenario 3, we modify the input to be the average contextual embedding of all tokens in the sequence. The sparsity level 3 is 0.05. Results are presented in Table 2] We excluded BLO from this experiment due to their prohibitive GPU memory usage. Given that the training data for the Mixing baseline predominantly consists of IFT data points, it is considered individual methods may excel in specific tasks, none of the non-DPS baselines nor the CDS and ClassAct consistently surpass the others across all downstream tasks. However, BADS stands out by consistently outperforming all other basclines across every task, except for AskLLM-O, which, as indicated in Table I demands significantly more computational resources. According to Table (Appendix F) BADS prefers the -oitann--riten--ee from-scratch OPEN ASSISTANT and DOLLY. over the rest two created using existing benchmarks.
standard IFT process Table indicates that while
_b__22

---

To facilitate the use of our proposed datapoint selection method, we provide comprehensive guidelines for hyperparameter tuning in Appendix [E] We also include an in-depth discussion and ablation study on the influence of hyperparameters on the method's effectiveness.
5 Related Work
Recent works on DPS broadly fall into four categories: i) Approaches based on meta learning (or BLO), ii) Gradient- based methods, iii) Methods based on domain adaptation and transfer learning methods, and iv) General sample reweighing strategies.
Meta learning (BLO) approaches. The DPS can be formulated as a meta learning BLO problem where the outer loss is defined with the training data selection variables, and the inner optimisation is to minimize the model's loss with the selected data [9 15 41 45 62]. These methods vary in the choice of their outer optimization variables: either directly using the data point weights A1 45 [9]
or mini-
batch samplers [15]. To solve the BLO, most approaches rely on computing meta-gradients via reverse mode differentiation 45 6 while some works utilised reinforcement learning techniques [15], All these BLO- methods are computationally demanding with large memory footprint, hindering them from being applied to large-scale models/data.
Gradient- methods. The key idea is to measure the importance of training data points based on the alignment scores between the loss gradients on training and meta data points. The rationale behind the gradient alignment can be theoretically underpinned by the BLO perspective. As shown in [41], the one-step inner loss update with zero initial weights in BLO reduces to the gradient alignment (cosine angle) between the train and meta data points [57]. However, the method in (57) requires evaluating and storing gradients of the entire training data points, thus computationally expensive. Furthermore, the final solution may not be optimal since the gradient computation is done with initial network that is warm-up trained with a random subset of the training data. In [16] the weights are optimised by the expected gradient norms during the network training, which serves as a proxy for the importance of data points. In (8 they find the subsets that closely match the gradient of the training and meta sets using an orthogonal matching pursuit algorithm. Under online continual learning setup [2] they formulate sample selection as constraint reduction problem based on the constrained optimization view of continual learning.
Domain adaptation and transfer learning methods. Selecting subset of the train set that are best aligned with the in-domain meta set can be naturally seen as domain adaptation and transfer learning. In the NLP community, (T] finds that the large language models implicitly learn sentence representations that cluster by domains without supervision, while in [20), they show the effectiveness of domain-adaptiv pre- trainiinnng with data augmented using simple data selection strategies. In [39] they proposed domain adaptive transfer learning which computes the importance weights on the target dataset from the ideas in domain adaptation and estimation of the label distribution. In multi-task transfer learning community, some previous works identify the detrimental effects of the gradients from different tasks, and propose strategies to mitigate the conflicting gradients issue [60] 34).
General sample reweighing strategies. Since DPS essentially involves finding the optimal reweighed distribution with the underlying domain itself unchanged, several approaches aim to tackle the problem via importance sampling techniques. In (27) they derive a tractable upper bound to the per-sample gradient norm, and derive estimator of the variance reduction achieved with importance sampling. The curriculum learning [3] is also closely related as one can design an optimal schedule of the successive training distributions. For instance, in 44 they introduce the so-called data parameters, associating samples and classes in the train data with learnable parameters, which governs their importance in the learning process within the curriculum learning framework.
6 Conclusion
We revisited the DPS problem from the perspective of Bayesian learning. By treating the neural network of interest, as well as an auxiliary weighting neural network as random variables and inferring their joint posterior using SGLD, we achieve simple and effective approach to data reweighting that is more reliable and scalable than BLO alternatives. Our framework is straightforwardly applicable to learning with data imbalance, label noise, and automating auxiliary data curation. We demonstrate our framework can apply to automating curation of the wide variety of auxiliary instruction fine-tuning data available for billion-scale language models. Overall this demonstrates a promising new kind of approach to the growing need for data optimization in neural network learning.
10

---

Limitations
Our proposed BADS algorithm has the following limitations, which we plan address our future work.
There are several hyperparameters involved, which need be carefully tuned for best perfor- mance. These include: the sparsity level parameter 8, the relative impact constants p's of the objective terms our SGLD update (detailed in App. E and the standard hyperparameters (e.g., batch size, learning rates, weight decay). Some of these hyperparameters may be optimised via Bayesian model selection in a principled manner. For instance, the sparsity level 8 can be regarded as latent random variable (a part of the model) with proper prior distribution imposed on it, and we can do posterior inference of 3 (or marginalise out) together with w and 0 in our SGLD update equations. We will pursue this in our future study.
2. Although our method computationally far more efficient than BLO and some other DPS approaches, its GPU memory footprint is demanding compared to non-DPS algorithms. This mainly originates from the entire weight variables or the whole weight network parameters baaee//miintiine in the memory for frequent updates. One possible workaround is to load only the weight variables that are assoicated with the current minibatches. We will be investigating this code optimisation further in our ongoing future study.
References
[1] Race Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In Annual Meeting of the Association for Computational Linguistics, 2020.
[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances on Neural Information Processing Systems, 2019.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum lcarning. In International Conference Machine Learning, 2019.
[4] David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. Color-filter: Conditional loss reduction filtering for targeted language model pre-training. arXiv preprint arXiv:2406. 10670, 2024.
[5] Yihan Cao, Yanbin Kang, Chi Wang. and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models, 2024. httttttt::// https:://pperreview
[6] Thiago Castro Ferreira, Claire Gardent, Nikolai llinkhh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). International Workshop on Natural Language Generation from the Semantic Web (WeNNLG+), 2020.
Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv preprint 2305 0024669 2023.
[8] Pengfei Chen, Benben Liao, Guangyong Chen, Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, 2019.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. CoRR, Abs/1803.05557, 2018.
[10] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023. URL /tttps://www.i www.wwddaabriccs.coo wetttabricke.combllog/////
httpbboog/2223/4/1122
[11] Lucio M. Dery. Yann Dauphin. and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral. In International Conference Learning Representations, 2021.
[12] Ondfej Dušek. Jekaterina Novikova, and Verena Rieser. Evaluating the sfate-of of-the--rttttttt of end-to-end -nn natural language generation: The E2E NLG challenge. Comput. Speech Lang., 59(C): jan 2020.
11

---

[13] Talfan Evans, Shreya Pathak. Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, and Olivier J Henaff. Bad students make great teachers: Active learning accelerates large-scale visual understanding. arXiv preprint arXiv:2312.00 12002228.. 2023.
[14] Talfan Evans, Nikhil Parthasarathy, Hamza Merzie, and Olivier J Henaff. Data curation via joint example selection further accelerates multimodal learning. arXiv preprint arXiv:2406.1771 2024.
[15] Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie- Yan Liu. Learning what data to learn. In arXiv, 2017.
[16] Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi, Mohammad Taher Pilchvar, Yadollah Yaghoobzadeh, and Samira Ebrahimi Kahou. Bert data diet: Finding important examples by gradient-based pruning. ArXiv e-prints: arXiv:221 1.05610,, 2022.
[17] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Black, Anthony DiPofi. Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoclkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, pttttpp://zeeodo..oo www.e/annwdo.ooeeererrrodo//e//a 102566836
12 2023. Utt.https::///aatt
[18] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Breez-Beltrachini. The WebNLG challenge: Generating text from RDF data. In International Conference on Natural Language Generation, 2017.
[19] David Grangier. Pierre Ablin, and Awni Hannun. Adaptive training distributions with scalable online bilevel optimization. arXiv preprint arXiv:2311.1197. 2023.
[20] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Annual Meeting of the Association for Computational Linguistics, 2020.
[21] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu. Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances Neural Information Processing Systems, 2018
[22] Haibo He and E.A. Garcia. Learning from imbalanced data. 21(9):1263 -1284, 2009.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL htttt:ttt/:/rretee httpp://aaxvvv.r//bbbbbaa www./boe//////......o...o..eeeee 1112.11122..0 03385
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, In International Conference on Learning Representations, 2021.
[25] A. Immer, M. Bauer, V. Fortuin, G. Ratsch, and M. E. Khan. Scalable marginal likelihood estimation for model selection in deep learning. International Conference on Machine Learning, 2021.
[26] Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. In International Confer- ence on Natural Language Generation, 2020.
[27] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference Machine Learning, 2018
[28] Krishnateja Killamsetty, Durga S, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, Proceedings of Machine Learning Research, 2021.
[29] KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In AAAI, 2021.
[30] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009. URL TR.9-TR.pdf.
sttps://www/ cs. wor.ttrrt.o.... wwwhonttt.uudmm///eeeeee edu/"kriii ptrrz//eeannngg wwwarniin--fttureee
[31] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant conversations democratizing large language model alignment. CoRR, abs/2304.07327. 2023.
12

---

[32] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-22322 1998.
[33] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecum htt.a/aww..cc...oo/eiiiiii. cot//edd///e/sssstt...... 2010.
[34] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conffict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, 2021.
[35] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Lc, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, 2023.
[36] J. Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, 2015.
[37] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovee. Corcsets for robust training of deep neural networks against noisy labels. In NewrIPS, 2020.
[38] Swaroop Mishra, Daniel Khashabi. Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsoureing instructions. In Anmual Meeting the Association for Computational Linguistics, 2022.
[39] Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoe V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. In arXiv preprint, 2018. URL/attps://arx// org/ abs/1811.0700
[40] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Advances on Neural Information Processing Systems, 2021.
[41] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to Reweight Examples for Robust Deep Learning. International Conference on Machine Learning, 2018
[42] Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms. arXiv preprint arXiv:2402. 09668, 2024.
[43] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[44] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste, Data parameters: A new family of parameters for lcarning a differentiable curriculum. In NeurlPS, 2019.
[45] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-nng Learning explicit mapping for sample weighting. In NeurIPS, 2019.
[46] Hwanjun Song, Minseok Kim. Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. TNNLS, 2022.
[47] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openrent
[48] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford alpaca: An instruction-folloiing lama model. httpps://githbb
com/tatsu-lab/staaffeee wtw.aa/ttaardda.aaaaaaaoaa aalppaaaaaddaaaaaaaaaaaaa 2023.
[49] Wei Wang. Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. In Conference on Machine Translation: Research Papers, 2018
13

---

[50] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi. Yeganeh Kordi, Amirreza Mirzaei. Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson. Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mchrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- Naturallnstructions: Generalization via declarative instructions 1600+ NLP tasks. In Conference on Empirical Methods in Natural Language Processing. 2022.
[51] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. In Neural Information Processing Systems Datasets and Benchmarks Truck, 2023.
[52] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- instructions. In Annual Meeting of the Association for Compulational Linguistics, 2023.
[53] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot leamerr. In International Conference on Learning Reprexentations. 2022.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thouu thought prompting elicits reasoning in large language models. Advances on Neural Information Processing Systems, 2022.
[55] Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamies. In International Conference on Machine Learning, 2011.
[56] Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Aji. LaMini-LM: diverse herd of distilled models from large-scale instructions. In Conference of the European Chapter of the Association for Computational Linguistics, 2024.
[57] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. CoRR, abs/2402.04333 2024.
[58] Mengzhou Xia. Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Dangi Chen. LESS: Selecting Influential Data for Targeted Instruction Tuning. arXiv preprint arXiv:2402.0433 2024.
[59] Hu Xu, Saining Xie, Po- Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Cit: Curation in training for effective vision-lange language data. In IEEE International Conference on Computer Vision, 2023.
[60] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman. and Chelsea Finn. Gradient surgery for multi-task lcarning. In Advances in Neural Information Processing Systems, 2020.
[61] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In ACL (1), pages 4791- Association for Computational Linguistics. 2019.
[62] Zizhao Zhang and Tomas Pfister. Learning fast sample re- weighting without reward data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 234,, 2021.
[63] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU. Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Oner Levy. LIMA: Less more for alignment. In Advances on Neural Information Processing Systems, 2023.
[64] Difan Zou, Pan Xu, and Quanquan Gu. Faster convergence of stochastic gradient langevin dynamics for non-log-concave sampling. Uncertainty in Artificial Intelligence (UAI), 2021.
14

---

Appendix
A
Posterior Derivation for Eq. 4
From the graphical model in Fig.
we exploit the conditional independence Dm 1 (w, Dz) 0.
p(0, w D2, Dm) p(0, w, Dm D2)
(10)
p(Dm (t)
(DD -((w) p(0, Dmlue Dz)
(11)
D(Dm|Di) p(w): p( 10, w, D.) p(0/w, D,)
(12)
D.) p(w) pl Dm. 00) p(0) u. D.)
(13)
-p(w) p( (m (0) p(0 D.)
(14)
In 13 we use Dm 1 (uc, Dz) 0, and in 14 p(Pm) is regarded constant for has nothing to do with and w.
B Posterior Derivation for Weight Network Cases
Here we provide posterior derivation for the weight network case w, = w(2i: Φ). First, the weights become deterministic function of D, and ø (here weight network parameters) as shown in Fig. a). But since w is a deterministic function of Φ and Dr. we can simplify by having w absorbed into á while introducing conditional dependence (an arrow) from D, to Φ, as depicted in Fig. b)). Note that D, is always given, we treat ø as a random variable, and wherever w; appears, replace it by w<2;; Φ). More specifically, make the following changes to the equations in the weight net scenario:
Eq.
p(0 D.) x p(0) p(u(2f)))))) 2))
Eq.
p())D() xe (. e -NE)))))
Eq. (with detailed derivations):
p(0, d. Dr)
p(0, D, D...)
(15)
p(Dm DD)
p((0 D.) p(0, D lbb Dz)
(16)
p(Dm D.)
1
p( D.) p(D Φ. D.) p(0lo, D.)
(17)
p(Dm D.)
p(Dm D.) p( D.) 1() p( p D.)
(18)
p.))D.) ((mm)0)) p(00), D.)
(19)
Eq.
5 10, Φ) + [0 + V logr(0, D, Dm,) + e~ N(0,7)
Eq. 61: 0 +- 0 + Vo (ogg(()) - N. [w]
Nm NB..B. [[2]] 01]) + +co/n
Eq.
: Φb f ④ + V. [logr(0)D)) D.) - N.- +){{(=;))) + +CoVT
15

---

Φ
weightmet(z'. 4)
W
Dt
Φ
D m
D
(a)
(b)
Figure 6: Graphical models when the weight network is adopted. (a) One possible representation. (b) More simplified representation by absorbing into & using deterministic weightnet(zf:: Φ). See texts for details.
C Convergence Analysis
Assumption C.1 (Adjusted from Assumption 4.3 in (64))
There exists m >> 0 and b 2 0 such that
0
Vee logp(0, w Do, Dm);
(20)
2
holds for any 0, w.
Assumption C.2 (Adjusted from Assumption 4.4 in 64 Any minibatch gradient of the log-
posterior is Lipschitz continuous. That is, there exists constant L such that for any 2, E D2 and 2j Dm,
Va, logp(w) log Newrl=z;;0) N=)
Vor w (logp(w') logp(0") Nowsl(=f;00) N N= ||
0 w/
-
w
2 (21)
holds for any 0, w. 0', w'.
Theorem C33 (Adjusted from Theorem 4.5 in (64)) Let d dim + N2, B be the batch size, and ginit
p the Cheeger constant. For any e E (0, 1), with the initial iterate satissfyng (l
wiitt
S
p2pdd-1)), R/2 S c/16 R R(cK-1/12)) and step size n = 23
the distribution
PK
SGLD of the K-th iterate in our SGLD iterations Eq. 0
salisfies:
L p(0, w Dr, Dm))lry S >(1 Con) k B- C-n + C21 + e/2 (22)
for some constant 0, C. = Ō(p²) C1 Ö(Rd ). C2 = Ö(dp Here || |lzv stands for the total variation distance, and R is defined as:
625d 4d log( (L/m) + 4b 4d + 8 /ogo 1/2) - 8 1og(1/2)
1/2
R(2) max
m
m
m
(23)
D Details of the Tasks
WebNLG 2020 We use the release release_v3.0_ce version of WebNLG benchmark. Domains training set are: Building, Astronuut, City, University, MeanOffrrussportttinn, SportsTeam, Food, Artist, Company, ComicsCharacter Monument, Airport, Politician, Athlete, Written Work, CelestialBody. Domains in test set are MusicalWork Scientist, Film.
16

---

Input Tuples
Output Text Description
William Anders
Retremant
1963
Wiltamm Anders, who retired Soptember 1st, 1965, was a crew member on Apolo and served under commander Frank Borman. Apollo was operated by IASA with Buzz Aldhe as bachup pllr.
Apollo
commander
Frank Oorman
William Anders
(embbrr of Apollo
Apollo
backup piiot
Apolla operator
NASA
Arguneen
Predieate
Arginnont
Figure 7: An example of Natural Language Generation.
E Details of the Experiments
E.1 Hyperparameters Details
To enhance flexibility in managing the training process, we incorporate pg. Pg and pty into Eq 7 as follows:
0 + 8 ! Vo logp(0) No B..B, [ws (w;;((2(:0)) Pb Nm By-B.. Bm [[]]]: + coV
(24)
- + ! Vi (log((w) Piu Ne- (wr + G 2
(25)
The training hyperparameters for BADS and CDS are shown in the table below. Other hyperparame- ters have been shown in the main paper.
BADS
Pg
Pg
Pm
8
8ang
CDS
ffoor
H
Ir Ir_haly
MNIST
1.0
1.0
1.0
1.0
5e-5 # N,
0.005
10
MNIST
0.005
1000
5000
CIFAR
1.0
0.1
1.0
1.0
5e-5 * Ni
0.8
10
CIFAR
0.8
15000
10000
WebNLG
1.0
1.0
le-5 + 0.05
10
WebNLG
0.05
6700
20000
LLMs
1.0
0.5
1.0
1.0
le-3 N:
0.05
10
LLMs
0.05
4200
15000
Table Hyperparameters for all experiments.
F,2
Guildline for Hyperparameters Tuning
There are two primary sets of hyperparameters in BADS:
hyperparameters for model's parameters update: 17, eg. cwn Pb. P8 and ptu (Eq, 24 25
hyperparameters for the prior distribution of the example weights, which are B, Savg (Eq. 8
In general,
set 17 to I and kept the Gaussian noise small with eq and e equal to le-5;
p'n is set to 1;
we set o3 to 1 and primarily adjust pb:
most cases, P is simply set to I. However, the training set contains wheee the ground truth labels might be incorrect-the the loss from the training examples, particularly in the early stages training, may be unreliable. such cases, we decrease p/;
Sazg should not be too large, it may incorporate outdated weights from earlier training steps.
We set to 10:
we select 3 based on the proportion of training data we consider beneficial for the downstream
tasks. For the LLMs Instruction Fine-tuning experiments, we adopt the same ratio used in the previous studies [52 46).
a controls how tightly the weights should match B. set a based on our confidence in the selection ratio 8: a smaller indicates greater confidence in B.
17

---

data denoising scenario is bit special, Eq 7 shows that high losses from the training batch push the example weights w toward O. With a high noise rate (50% and 80%), the training batch losses remain high throughout the training process. To prevent the weights w from collapsing to 0, we use a high selection ratio 3 and a low a.
E.3
Ablation study: Influence of the impact constants and sparsity level B.
Impact constant o Higher causes the example weights w drift away from B, occasionally collapsing to 0, and may result in incorrect example weights (see Figure B). The right column in Figure TT shows that in all three proof-of-concept scenarios, the models achieve similarly good performance when a is reduced to around 10-5.
Sparsity level / In both WebNLG and MNIST scenarios, high 3 leads to incorrect example weights (see Figure D.. Conversely, due to the reason we mentioned above, in data denoising (CIFAR) scenario, lower 3 leads to incorrect weights. The left column in Figure 11 shows that the backbone models' performance significantly declines in both the WebNLG and MNIST scenarios when the example weights are incorrect. In the CIFAR scenario, the impact is less pronounced.
Duta Dena slng JOHAR (o%)oo
Dara iasinnn with KOl
Data Desalil up JOHN relan)
0310
0274
0.60
0372
0447
0.160
30200 19100 30130 DHOK IRIS T00R0 15.5 1963
14:04 395.0 30700
15900 19030 2000 19753 19:00
39050
18509
19700
PHOM EXIOH 1909 1975
Figure 8: Varying the impact constant σ. The data denoising scenario is conducted on CIFAR- 10 with 50% noisy data, while the sparsity level 3 is maintained at 0.8, consistent with the main experiments. From left to right, a decreases from 0.5 to 5 x 10 When a 0.5, the weighting network assigns higher weights to the noisy examples. At = 0.05, it assigns similar weights to both noisy and clean examples. Only when a sufficiently low (s x 10-9) does the model assign distinctly higher weights to the clean examples.
0.00
LEPERACITY
10250 30100 105M
Figure 9: training with two domains: City and Artists. Since Artists has greater overlap in schema and vocabulary with the downstream domains, the model should assign higher weights to examples from this domain. The impact constant is maintained at 1 x 10-5 consistent with the main experiments As 3 decreases from 0.8 to 0.05, the weighting network behaves as follows: at 3 = 0.8, it assigns higher weights to the City domain; at B 0.2, it assigns similar weights to both domains; and at 0.05, it assigns significantly higher weights to the Artists domain.
Varying the sparsity level 3. This efficient learning scenario is conducted on WebNLG
F Analysis of Main Results
F.1 Learning Curves Study
The experimental results discussed in Section 3.3. demonstrate that BLO converges more slowly compared to other approaches. In this section, we aim to examine the learning curves of DPS approaches, as illustrated in Figure II] The trends for each approach appear consistent across both
18

---

Test/ACC for MNIST (hparam sensitivity. beta)
Test/ACC for MINIST (hparam sensitivity. sigma)
1.0
0.9
sigma,
beta_9.01 9.01
TesAACC
sgma_
sgma 5e-1
bata 0.005
simma 56-2
0.7
beta,
Sigma 56-6
0.6
2000
4000
6000
8000
10000
12000
14000
2000
4000
6000
8000
20000
12000
14000
Stpps
Steps
Test/ACC for CIFAR (hparam sensitivity, beta)
Test/ACC for CIFAR (hparam sensitivity, sigma)
beta_0 0.5
beta_0.5
beta_0.3 0.3
0.70
0.60
TestACC
0.00
0.55
0.55
0.45
sgma_5e>
sgma_
sigma 5m-1
sigma 50.6
0.35
2300
5000
7500
10000
12500
15000
37500
20000
2500
5000
7500
10000
12500
15000
37500
20000
stops
steps
Test/BLEU for WebNLLG 2domain (hparam sensitivity, beta)
WebNLLG2domain Test/BLEU for WebNLG 2domain (hparam sensitivity, sigma)
46
bata 0.2
44-
44
0.8
beta.o.5
bata 0.05
38
sgma_le- sgma, 1e-5
36
sgmma 1e-2
36
sigma sgmm 10-3
5000
10000
15000
20000
25000
30000
35000
40000
25000
steps
Figure 10: Model's performance in the three proof-of-concept scenarios with different 3 and
symmetric and asymmetric noise experiments. The convergence rate of BADS aligns with that of non- DPS approaches, whereas CDS converges faster than BADS, and BLO converges significantly slower. In the asymmetric noise experiments, overfitting is less pronounced compared to the symmetric noise experiments, where after 50,000 training steps, the test accuracy for all approaches decreases. BLO and BADS exhibit notably slower rate of overfitting compared to other methods. Conversely, CDS overfits more quickly than non-DPS approaches, resulting in significantly lower test accuracy even compared to the non-DPS methods.
F.2 Individual Scalar Weights vs. Weight Network
In [1), each training example is associated with a learnable scalar weight. The scalability issues of this method compared to the weight network used BADS are detailed in Section 2.4 this section, we examine the CIFAR-10 denoising task to assess performance differences that result from replacing the BADS weight network with the individual weights strategy described [41. From the top row of Figure 1I it is evident that the performance of BADS with scalar weight (referred as
19

---

Test/ACC for CIFAR-10 (symmetry noise)
Test/ACC for CIFAR-10 (asymmetry naise)
BLO
0.7
0.7
Bayesian
Bayesian_per ppr_ppappleeeeee
random,
TeSUACC
meta_orly
contrastive
0.5
mixing
dup meta
Bayesian, per sample
random select
meta, only
0.3
75000
125000 150000
steps
steps
Train/ACC for CIFAR-10 (symmetry noise)
Train/ACC (asymmetry noise)
1.0
0.9
0.9
0.
0.8
0.7
TrainAAC
0.6
TrainACC
0.7
contrastive
0.6
BLD
0.5
BLO
contrastive
mixing
Bayesian
dup meta
04
dup meta Bayesian
Bayesian_per sample
Bayesian per peppl
random select
03
random, select only
meta, only
0 25000 50000 75000 100000 125000 150000
175000 200000
02
25000 50000 75000 100000 125000 150000 175000 200000
Figure 11: This graph shows the test/train accuracy over 200K training steps. The x-axis denotes the training steps, and the y-axis indicates the aceuracy levels. The top row displays the testing accuracy, while the bottom row shows the training accuracy. In the left column all models are trained using train set contaminated by 50% symmetric noise. While, in the right column, the train set contaminated by 40% asymmetric noise.
Bayesian per per_samplee sample in the legend) slightly surpasses that of the original BADS (labeled Bayesian in the legend).
IFT sets
Avg Example Weights
OPEN ASSISTANT | [3]
0.0063
DOLLY 10
0,0025
Flan-Ve V2 35
0,0023
CoT [54]
0.0005
Table 4: The average scores IFT examples get from BADS.
20

---

Test/BLEU for WebNLGG22dmmaig
44
42-
40
Test/BLEU
38
contrastive
Bayesian
34-
5000 10000
25000
30000
steps
Figure 12: Scenario 3, domain adaptation using WebNI.G benchmark. This plot shows the BLEU scores on WebNL.G benchmark. All DPS models are trained on 2 domains, Artist and City, and tested on other 3 domains - Athlete, Politician, and Astronaut. The x-axis represents the training steps, while the y-axis shows the evaluation metric, BLEU,
Wrights Data Pieets Prodicted by
Weigess
Prodeted SLD
Weyyto
Prodited CDS
34-
0.2
ti bir maa Letal Heburity Clac- 4
44-
meghts esangl Lddeviieetty Cros
coarpipe viderrty Class
33-
03-
Wee Date Phantr CIFA Predeced BADS
Wargtt Data Ponde in CIFAI Preckcted BLD
Weight Cats Poeda CEART Prackded COR
4.70
K770
6:00
6:07-
5.14
-0.0
2009
20500 2059
2000
1125
20500
1075 2040
2000
2016 FIDM
2003 JOME
2150 12000
20300
21500
20003
22000
21003
Weiggre of Poirr
BADS
VWAgTn Daps Peett Pydicee BLO
Wiagra WMANLG Preducted by CDS
03129
0.009
with
eeaigga sod lacal wisi. ouandi dio ulfodhieeees
with
03577
euamge
-320
with
Figure 13: Proof-of-Concept experiment supplementary results. All plots illustrate the average weights of data points within mini-batches during the last 2000 training steps. with the x-axis representing the training steps and the y-axis showing the average weights. Classes depicted in blue expected to receive higher weights compared to those in red. The top row displays the MNIST experiments from scenario 1, the middle row shows the CIFAR experiments from scenario 2, and the bottom row features the WebNLG experiments from scenario 3. The left, middle, and right columns correspond to BADS, BLO, and CDS, respectively.
21

---

Weetthe MNEET Precesed Clanadd
Wegght Paane CEPAIT Predicted Cheeskci
Date Ponde Wetrl
6.00
munapple Mynardy
musepae
800A
8000
0.022
Proof-of-Concept experiment supplementary results. All plots illustrate the average weights of data points within mini-batches during the training of ClassAct method, with the x-axis representing the training steps and the y-axis showing the average weights. Classes depicted in blue are expected to receive higher weights compared those red. The left plot displays the MNIST experiments from scenario I, the middle plot shows the CIFAR experiments from scenario 2, and the right plot features the WebNLG experiments from scenario 3.
Figure 14:
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have included our main claims in the abstract and introduction that accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed several limitations of the current work, together with workarounds and our plans to address them our future work. Please see the Limitations section right before the References.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have described all the information needed to reproduce the main experi- mental results the paper to the extent that it affects the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release code, with sufficient instructions to faithfully reproduce main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The results accompanied by emro bars wherever possible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources needed to reproduce the experi- ments is described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htthhs/ttttts///rrrrr...../////p
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper presents work whose goal is advance the field of Machine Learning. At this point we do not see any negative impacts of the current work on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors cite the original papers that produced the code package dataset under proper licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "d04EbvyDmL", "reviewer": "Reviewer_JeEz", "summary": "This paper proposes a new Bayesian method for Data Point Selection (DPS), called BADS. DPS aims to select training data points that optimize performance on downstream tasks. Instead of relying on bi-level optimization (BLO), BADS frames DPS as a posterior inference problem. The method uses a weight network to learn the importance of each data point and proposes a novel update rule for jointly updating model parameters and data point weights using Stochastic Gradient Langevin Dynamic sampling. Experiments on MNIST, CIFAR, and WebNLG datasets show that BADS outperforms or matches existing DPS methods, including BLO, in terms of accuracy and efficiency.", "questions": "Can the authors provide any theoretical guarantees on the convergence of BADS? What are the conditions under which BADS finds an optimal solution?\n\nWhat is the impact of different hyperparameters, especially the sparsity level parameter and impact constants? How robust is BADS to hyperparameter variability.", "limitations": "The authors have adequately addressed the limitations proposed in their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: This paper introduces a novel perspective on DPS by using a Bayesian framework instead of the BLO approach. This offers a new way to think about data selection and its connection to posterior inference. The paper also leverages Langevin sampling for optimizing the derived posterior inference which adds to the novelty.\n- Clarity: The paper is well-written and easy to understand. The authors clearly explain their method and the motivations behind their design choices. The final derived method is simple and intuitive.", "weaknesses": "- Limited theoretical analysis: The paper focuses on the empirical performance of BADS and lacks a deep theoretical analysis. Providing more theoretical insights on the convergence and optimality properties of BADS would strengthen the paper, especially since the authors claim that their method, unlike BLO, has theoretical convergence guarantees.\n- Hyperparameter sensitivity: The paper mentions that BADS has several hyperparameters that require careful tuning. A more in-depth discussion on the impact of these hyperparameters and potential strategies for automatic tuning would be beneficial.\n- Comparison and claims:  The paper primarily compares BADS with BLO, but there are other methods that deal with selecting data to learn on, such as offline pruning methods [1], and online and offline batch selection [2, 3], among others. In addition to that, some of these methods have already been used for large scale model training, such as [4, 5].\n\nSome important equations and their corresponding assumptions have been omitted. In particular, equation (4) is not derived and on first glance seems to assume independence of $w$ and $\\mathcal{D}$, but it is not clear if this still holds after the introduction of the weight network. \n\n1. https://arxiv.org/abs/2206.14486\n2. https://arxiv.org/abs/2312.05328\n3. https://arxiv.org/abs/2406.10670\n4. https://arxiv.org/abs/2402.09668\n5. https://arxiv.org/abs/2406.17711", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Re: Convergence Analysis]\n\nWe provide a theorem showing that our SGLD algorithm converges to the true posterior to some extent. Our analysis is based on (Zou et al. 2021) where we make some adjustments for our case. \n\n**Assumption 1** (from Assumption 4.3 of (Zou et al. 2021))\nThere exists $m>0$ and $b\\geq 0$ such that\n\n$$\\bigg\\langle\\nabla_{\\theta,w}\\log p(\\theta,w|D_t,D_m),[\\theta, w]\\bigg\\rangle\\geq m ||[\\theta, w]||_2^2-b$$\n\nholds for any $\\theta,w$.\n\n**Assumption 2** (from Assumption 4.4 of (Zou et al. 2021))\nAny minibatch gradient of the log-posterior is Lipschitz continuous. Ie, there exists a constant $L$ such that for any $z_i^t\\in D_t$ and $z_j^m\\in D_m$, \n\n$$\n||\\nabla_{\\theta,w} A(\\theta,w)-\\nabla_{\\theta',w'} A(\\theta',w') ||_2\\leq L ||[\\theta, w]-[\\theta', w']||_2\n$$\nholds for any $\\theta,w,\\theta',w'$, where $A(\\theta,w)=\\log p(w)+\\log p(\\theta)-N_t w_i l(z_i^t;\\theta)-N_m l(z_i^m; \\theta)$.\n\nThen we have the following convergence theorem -- We omit detailed formulas due to space limit, but can be found in (Zou et al. 2021).\n\n**Theorem** (from Theorem 4.5 of (Zou et al. 2021))\nLet $B$ be the batch size and $\\eta$ the step size. Suppose Assumption 1 and 2 hold. For any $\\epsilon\\in(0,1)$, with the initial iterate satisfying a certain ball constraint, the distribution $\\mu_K^{SGLD}$ of the $K$-th iterate in our SGLD iterations Eq.(6-7) satisfies:\n\n$$||\\mu_K^{SGLD}-p(\\theta,w|D_t,D_m)||_{TV} \\leq C(B,\\eta,K,\\epsilon)+\\epsilon/2$$\n\nwhere $C(B,\\eta,K,\\epsilon)$ is constant that can be reduced by adjusting $B$, $\\eta$ and $K$, and $||\\cdot||_{TV}$ stands for the total variation distance.\n\n(Zou et al. 2021) \"Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Sampling\", UAI 2021.\n\n## [Re: Hyperparameter Analysis] \n\nSee [Re-3: Hyperparameter Analysis] in the General Rebuttal\n\n## [Re: Comparison and Claims]\n\nWe thank the reviewer for bringing the recent related work to our attention. We recognize that there are three types of DPS setups based on supervision signals:\n\n* **Unsupervised DPS**: DPS **without** the guidance of a held-out meta set involves models selecting a subset from a training set based on specific hypotheses, such as \"challenging examples improve model performance\". Curriculum learning falls into this category. This method is also widely used in pre-training data point selection (**[1]** and **[4]**).\n\n* **Self-supervised DPS**: DPS **with** a held-out meta set. However, the meta set **does not share the same data distribution** as the targeted test set. Typically, the examples in the meta set are also selected from the training set based on specific hypotheses, such as \"learnable examples enhance model performance\". **[2]**, **[3]** and **[5]** fall into this category.\n\n* **Meta-set guided DPS**: DPS **with** a small meta set that **shares the same distribution** as the test set, in order to train a model that performs well specifically on the target test set. The test set may encompass one or multiple downstream domains or tasks. This DPS is closely related to meta learning, domain adaptation, and transfer learning. Existing approaches are predominantly based on **BLO**. **Our work**, as explained in Section 2.1, focuses on this type.\n\nTo broaden our comparison, we choose studies **[4]** and **[2]** from the first two categories above respectively and evaluate them using our experimental setups. Here are the results: \n\n*  **Compared to Self-supervised DPS**:\n    * To guide DPS, **[2] (ClassAct)** investigates using a reference model trained on a meta set chosen from the training set based on predefined learnability metrics. To fairly compare the selection mechanism, we replaced their meta set using our meta set.\n    * **Figure 4 in the PDF** shows that ClassAct performs worse than BADS with a large margin across all three proof-of-concept scenarios. It completely failed in the Data Balancing and Efficient Learning scenarios.\n    * **Figure 5 in the PDF** shows that the weights predicted by ClassAct for the training examples are unreliable.\n    * **Table 1** below shows that BADS outperform ClassAct on all LLM benchmarks. \n    * **Table 2** below shows that BADS requires significantly less GPU memory and computing time compared to ClassAct.\n\n*  **Compared to unsupervised DPS**:\n    * Paper **[4] (AskLLM)** selects examples from training set by prompting LLMs. We compare to it only in the LLM use case due to two reasons:\n        * Paper **[1]** already showed that unsupervised DPS can amplify class imbalances.\n        * Open-source LLMs typically do not handle vision input.\n    * For fair comparison, we compare BADS with an online variant of [4] (dubbed **AskLLM-O**) where we used the pretrained OpenLLaMA 3B to obtain the sampling score for each training sample.\n    * **Table 1** below shows that BADS outperforms AskLLM-O on 2 out of 4 LLM benchmarks. \n    * **Table 2** below shows that BADS requires significantly less compute and memory compared to AskLLM-O. AskLLM-O also requires an offline scoring of all training samples which took 13932 seconds (3.87 hours) in our setup on a single NVIDIA A40 GPU.  \n\n**[Table 1]** DPS performance in LLM use case\n|Models | MMLU  | ARCc | ARCe | HellaSwag |\n|-----| ------------- | ------------- |  ------------- | ------------- |\n| BADS |  26.59 | 34.39 | 67.00 | 52.91 |\n| ClassAct | 24.90 | 31.91 | 67.34 | 52.09 |\n| AskLLM-O | 25.55 | 35.15 | 66.88 | 53.71 |\n\n**[Table 2]** DPS memory and time usage over 100 steps\n|Models | Avg-Memory (MB) | Time (s) |\n|-----| ------------- | ------------- |\n| BADS | 14694.58 | 61.03 |\n| ClassAct | 34821.36 | 269.81 |\n| AskLLM-O | 32908.89 | 115.24 (+13932) |\n\nWe also note that:\n* An online variant of [3] is similar to the CDS baseline in our paper.\n* [3] and [5] were released after the Neurips deadline\n\n## [Re: Graphical Model and Derivations]\nSee the General Rebuttal:\n* [Re-1: Derivations for Eq 4]\n* [Re-2: Graphical Model and Derivations with Weight NN]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yTbxJpLC1M", "reviewer": "Reviewer_oLJs", "summary": "This work proposes a Bayesian approach to the data point selection task. Specifically, the authors introduce the important weights to each training point and derive the posterior joint probability of the instance-wise weights and network parameters. The parameters and weights are then sampled iteratively based on the stochastic-gradient MCMC technique. Experiments on computer vision and natural language processing have been conducted to show the efficacy of the proposed method.", "questions": "Please see the weaknesses", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work views the data point selection problem from the perspective of Bayesian theory and introduces the posterior distributions of the instance-wise weights and network parameters.\n\n2. Experimental results demonstrate the efficacy of the proposed method.", "weaknesses": "1. I suggest the authors provide a detailed derivative for Eq. 4.\n\n2. Eq. 5 still requires to compute the normalizing constant in Eq. 4, which serves as the bias of the gradient.\n\n3. The authors claim that “Our method straightforwardly achieves sparsity on the w weights allowing efficient sample selection unlike”. I hope the authors provide more analysis and evidence about the sparsity of instance-wise weight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Re: Derivations for Eq 4]\nSee [Re-1: Derivations for Eq 4] in the General Rebuttal\n\n## [Re: Eq5 and Normalizing Constant]\n\nNo. The normalising constant in Eq.(4) is $p(D_m|D_t)$ (please see our derivations, especially, Eq.(R4) in our rebuttal above), and this normalising constant has no dependency on $\\theta$ and $\\lambda$. Thus after taking the gradient with respect to $\\theta$ and $\\lambda$, the normalising constant simply disappears. This technique is well known for Bayesian posterior sampling [Welling & Teh] and also for sampling from energy-based models, eg, T. Du, and I. Mordatch, \"Implicit Generation and Modeling with Energy Based Models,\" NeurIPS 2019.\n\n## [Re: Sparsity]\n\nAlthough we have observed clear distinction in the learned weights between relevant and irrelevant data points (eg. our proof of concept experiments), the weights are not as pronounced as 0/1 like binary selection. We believe that this issue might be diminished by incorporating an additional prior/regularizer that encourages 0/1 weight values. We will investigate it further along this line in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0Y0nDkM1U", "reviewer": "Reviewer_5MKv", "summary": "The paper proposes a method for data point selection (DPS) in the setting where only little data from the target (or meta) distribution is available, together with a lot of data from the training distribution. DPS aims to find the weights of data points in the training distribution such that the model parameters that minimise the loss of the therewith weighted training distribution also minimise the loss on the target distribution. The method proposed for DPS in this paper consists in treating the problem in a Bayesian way and doing posterior inference on both (data) weights and model parameters simultaneously, using stochastic gradient Langevin dynamic sampling (SGLD) to approximate the posterior. The crucial trick is to view the data from the target distribution as generated by the model (parameters), which allows for the inference.\nThey show the effectiveness of their approach on 4 experiments and compare against 2 DPS baselines (bilevel optimisation (BLO) and contrastive data selection (CDS)) as well as a range of naive baselines (mixing available training and target distributions to different extents). \n- The first experiment is on data balancing for MNIST, where the class balance differs significantly between training (imbalanced) and target (balanced) distribution. \n- The second experiment is on data denoising for CIFAR, where the training distribution (noisy) differs from the target distribution (clean) in the degree of noisiness\n- The third experiment is on efficient learning on WebNLG, where there are differences in the topic or domain of the data between training distribution (Artist, City) and target distribution (Athlete, Politician, Astronaut)\n- The fourth experiment is on instruction finetuning (IFT) for LLMs, using an OpenLlama3B model. \nThe proposed method compares favourably in all experiments in the main part of the paper. In the appendix results for the second experiment are additionally shown for longer training times, in which case BLO outperforms the proposed method.", "questions": "- Which part in the prior formulation or in the optimisation ensures that the weights are non-negative? Or are they intentionally not constrained by the method in that sense?\n- How does the causal graph in Figure 1 change if a weight network that is using embeddings of training data as inputs is being used like in the experiments? How does posterior inference change in this case?\n- Why is the weight network being used in the experiments if in the ablation in the appendix, the element wise weights version of the proposed approach is outperforming it?\n- Why are the inputs to the weight network different for the MNIST experiment (image embeddings) and the CIFAR experiment (image embedding + one-hot encoded label)?\n- When comparing to BLO in terms of compute efficiency, have the authors taken into account proposals of making the Hessian vector product calculations more efficient for BLO (see https://iclr-blogposts.github.io/2024/blog/bench-hvp/ for example)?", "limitations": "- Limitations are discussed, but after the page limit", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The idea of framing and solving DPS in a Bayesian way seems novel and innovative\n- The paper is well written in general and is easy to read and follow\n- The proposed method is getting good empirical results as displayed in the paper, in particular also on the relevant use case of IFT for LLMs\n- Compared to BLO, the proposed method seems to have lower memory and compute requirements", "weaknesses": "- For the most part, the method proposed is explained well and in detail. However, when it comes to a crucial aspect in the implementation  that is subsequently used in all experiments (the weight network), details are brushed under the carpet as a 'straightforwardly modification'. However, for me the causal graph in Figure 1 seems to be different if the weights become a function of $z^t$, let alone $\\theta$ if model embeddings $f_{\\theta}(z^t)$ are used instead of raw data $z^t$ like in the experiments. This should be explained in more detail (see also Question about this below). \n- In the main part of the paper, performance of BLO is portrayed as clearly falling behind the proposed method and other methods in terms of accuracy on the CIFAR experiment (Fig 2, Fig 3, line 266). However in Fig 7 in the Appendix, it can be seen that with longer training, BLO is at least on par (symmetry noise) or even slightly better (asymmetry noise) in terms of accuracy. It would be better if this would already be mentioned in the main part of the paper\n- Assuming a non-negativity constraint on the weights and the loss, in Eq (4) and subsequently Eq (7) the gradient wrt $w$ will be negative for the data dependent term and thus each gradient update is only pointing away from $0$ through the gradient of the prior term $\\log p(w)$ (and the noise term at the end). Therefore, preventing collapse to $0$ weights (as described in line 148) seems to crucially depend on a strong enough prior as well as the impact constant and $N_t$. At least for the not so compute intensive experiments on MNIST and CIFAR, it would be good to see an ablation on the effect of these impact constants and the prior governing sparsity parameter $\\beta$. \n- The limitations are only spelled out after the page limit", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Re: Graphical Model and Derivations]\nSee the General Rebuttal:\n* [Re-1: Derivations for Eq 4]\n* [Re-2: Graphical Model and Derivations with Weight NN]\n\n## [Re: Hyperparameter Analysis]\nSee [Re-3: Hyperparameter Analysis] in the General Rebuttal\n\n## [Re: Ensuring the Non-negative Weights]\n\nLines 205-206 of the paper indicate that in BADS, the weight network is implemented as a single-layer Feedforward Neural Network (FNN) with a sigmoid activation function. The sigmoid function ensures that the output weights are valid. We will highlight this in the revised paper.\n\n## [Re: Why Weight NN]\n\nIn the paper, specifically in lines 170-174, we note that this weight network approach is effective for smoothing and regularizing output weights due to the smooth functional properties of neural networks. Importantly, if additional training samples are added after the model has been trained, the learned weight network can be used to assign weights or select samples from the new dataset without needing to retrain the entire model from scratch. \n\nAdditionally, obtaining precise element-wise weights typically requires training the model on the entire training set for multiple epochs, which can be costly, especially for large foundational models fine-tuning. By using the weight network, we do not even need to train the model with one full epoch. After training on a subset of the data for one epoch, the weight network can be employed to assign weights to the remaining examples. We can then select the examples with higher predicted weights from this set and continue training the model on these selected samples. This approach further enhances training efficiency. We plan to conduct related experiments in future work.\n\nWe will explain with more details about this in the revised paper.\n\n## [Re: Different Weight NN for MNIST and CIFAR]\n\nIn the MNIST scenario, we focus on data balancing, where the minority and majority examples can be distinguished solely by the input images. In contrast, the CIFAR scenario targets at data denoising. Here, we generated noisy examples by randomly sampling class labels for the input images (L242-245), making it impossible to determine whether an example has a correct label (clean examples) or an incorrect label (noisy examples) just by examining the input image. As a result, the weight network requires both image and label information for accurate decision.\n\n## [Re: Efficient Hessian Vector Products (HVPs)]\n\nWe use PyTorch's (v2.3) automatic differentiation so there is no explicit Hessian computation. Specifically, we use reverse-over-reverse differentiation and hence is possible that we can get more gains using other methods (e.g. reverse-over-forward) for BLO. Nevertheless, regardless of how HVP is computed in practice, BLO is still more expensive than BADS as BADS doesn't require 2nd-order computation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their thoughtful and detailed feedbacks. 

## [Re-1: Derivations for Eq 4] @Reviewer **JeEz**, **oLJs**, and **5MKv**
From the graphical model in **Fig. 1 in PDF**, $D_m \perp (w, D_t) \ | \  \theta$ is the only conditional independence assumption we make. 
$$
p(\theta,w|D_t,D_m)=\frac{p(\theta,w,D_m|D_t)}{p(D_m|D_t)} 
$$
$$
\ \ =\frac{1}{p(D_m|D_t)}\cdot p(w)\cdot p(\theta,D_m|w,D_t)
$$
$$
\ \ =\frac{1}{p(D_m|D_t)}\cdot p(w)\cdot p(D_m|\theta,w,D_t)\cdot p(\theta|w,D_t)
$$
$$
\ \ = \frac{1}{p(D_m|D_t)}\cdot p(w)\cdot p(D_m|\theta) \cdot p(\theta|w,D_t) \ \ (R1)
$$
$$
\ \ \propto\cdot p(w)\cdot p(D_m|\theta)\cdot p(\theta|w,D_t) \ \ (R2)
$$
where in (R1), we use $D_m \perp (w, D_t) \ | \  \theta$, and in (R2) $\frac{1}{p(D_m|D_t)}$  is regarded as constant for it has nothing to do with $\theta$ and $w$.

## [Re-2: Derivations with Weight NN] @Reviewer **JeEz**, **oLJs**, and **5MKv**
Here are the full details of possible changes when the weight network is adopted. 

First, the weights $w$ become a deterministic function of $D_t$ and $\phi$ (here $\phi=$ weightnet params) as shown in Fig. 1(a) in the attached PDF.  But since $w$ is a deterministic function of $\phi$ and $D_t$, we can simplify it by having $w$ absorbed into $\phi$ while introducing conditional dependence (an arrow) from $D_t$ to $\phi$, as depicted in Fig. 1(b) in the PDF.

Note that $D_t$ is always given, and we treat $\phi$ as random variables, and wherever $w_i$ appears, we replace $w_i$ by $weightnet(z_i^t; \phi)$. More specifically, we make the following changes to the equations in the weight net scenario:

Eq.(2): $p(\theta|\phi,D_t) \propto p(\theta) \cdot \prod_{i=1}^{N_t} p(\textrm{weightnet}(z_i^t; \phi), z^i_t | \theta)$

Eq.(8): $p(\phi|D_t) \propto e^{-(\sum_i weightnet(z_i^t;\phi)-\lfloor N_t\beta\rfloor)^2 / 2\sigma^2}$

Eq.(4) (detailed derivations): 

$$
p(\theta,\phi|D_t,D_m) = \frac{p(\theta,\phi,D_m|D_t)}{p(D_m|D_t)}
$$
$$
\ \ \ \ = \frac{1}{p(D_m|D_t)} \cdot p(\phi|D_t) \cdot p(\theta,D_m|\phi,D_t)
$$
$$
\ \ \ \ = \frac{1}{p(D_m|D_t)} \cdot p(\phi|D_t) \cdot p(D_m|\theta,\phi,D_t) \cdot p(\theta|\phi,D_t)
$$
$$
\ \ \ \ = \frac{1}{p(D_m|D_t)} \cdot p(\phi|D_t) \cdot p(D_m|\theta) \cdot p(\theta|\phi,D_t)
$$
$$
\ \ \ \ \propto \cdot p(\phi|D_t) \cdot p(D_m|\theta) \cdot p(\theta|\phi,D_t)
$$

Eq.(5): $[\theta, \phi] \ \leftarrow \ [\theta, \phi] + \frac{\eta}{2} \nabla_{\theta,\phi} \log p(\theta,\phi|\mathcal{D}_t,\mathcal{D}_m)  + \epsilon \sqrt{\eta}, \ \ \ \ \epsilon\sim\mathcal{N}(0,I)$

Eq.(6): $\theta \ \leftarrow \ \theta + \frac{\eta}{2} \nabla_{\theta} ( \log p(\theta) - N_t \cdot E_{i\sim B_t}[weightnet(z_i^t;\phi) \cdot l(z_i^t;\theta)] - N_m\cdot E_{j\sim B_m}[l(z_j^m;\theta)] ) + \epsilon_\theta \sqrt{\eta}$

Eq.(7): $\phi \ \leftarrow \ \phi + \frac{\eta}{2} \nabla_{\phi} ( \log p(\phi|D_t) - N_t\cdot E_{i\sim B_t}[weightnet(z_i^t;\phi) \cdot l(z_i^t;\theta)] ) + \epsilon_\phi \sqrt{\eta}$


## [Re-3: Hyperparameter Analysis] @Reviewer **JeEz** and **5MKv**

We will incorporate a detailed discussion and ablation study on this topic in the revised paper. Here is a summary: 

**Two main sets of hyperparameters for BADS**:
* For parameters update: $\eta$, $\epsilon_\theta$, $ \epsilon_w$, $\rho_{\theta }^{t}$, $\rho_{\theta }^{m}$, and $\rho_{w}^{t}$ (*Eq 6, 7 and Appendix B*). 
* For the prior distribution of the example weights: $\sigma$, $\beta$, $s_{avg}$ (*Eq 8,9*) . 

**Hparams tuning principles**:
* We set $\eta$ to 1 and kept the Gaussian noise small with $\epsilon_\theta$ and $\epsilon_w$ equal to 1e-5.
* $\rho_{w}^{t}$ is set to 1.
* We set $\rho_{\theta }^{m}$ to 1 and primarily adjust $\rho_{\theta }^{t}$. In most cases, $\rho_{\theta }^{t}$ is simply set to 1. However, if the training set contains noise—where the ground truth labels might be incorrect—the loss from the training examples, particularly in the early stages of training, may be unreliable. In such cases, we decrease $\rho_{\theta }^{t}$.
* $s_{avg}$ should not be too large, as it may incorporate outdated weights from earlier training steps. We set it to 10.
* Generally, we select $\beta$ based on the proportion of training data we consider beneficial for the downstream tasks. For LLMs, we adopt the same ratio used in the previous studies (*paper [52] and [46]*).
* $\sigma$ controls how tightly the weights should match $\beta$. We set $\sigma$ based on our confidence in the selection ratio $\beta$: a smaller $\sigma$ indicates greater confidence in $\beta$. 

Data Denoising scenario is a bit special, Eq 7 shows that high losses from the training batch push the example weights $w$ toward 0. With a high noise rate (50% and 80%), the training batch losses remain high throughout the training process. To prevent the weights $w$ from collapsing to 0, we use a high selection ratio $\beta$ and a low $\sigma$.

**Ablation study conclusions: Influence of the impact constants $\sigma$ and sparsity level $\beta$**:
* Higher $\sigma$ causes the example weights $w$ to drift away from $\beta$, occasionally collapsing to 0, and may result in incorrect example weights (See **Figure 2 in PDF**). In all three proof-of-concept scenarios, the models achieve similarly good performance when $\sigma$ is reduced to around $10^{-5}$.
* In both WebNLG and MNIST scenarios, high $\beta$ leads to incorrect example weights (See **Figure 3 in PDF**). Conversely, due to the reason we mentioned above, in data denoising (CIFAR) scenario, lower $\beta$ leads to incorrect weights.
* In both the WebNLG and MNIST scenarios, the backbone models' performance significantly declines when the example weights are incorrect (See **Figure 1 in PDF**). In the CIFAR scenario, the impact is less pronounced.

Due to space constraints, we present a selective set of plots. The complete results for the three proof-of-concept scenarios will be provided in the revised paper.

Meta-review: All reviewers unanimously voted for acceptance. Weaknesses like missing baselines in the experiments and missing theoretical results, were outbalanced during the rebuttal period. Authors are strongly encouraged to import the additional results into the paper.
Decision: Accept (poster)