Title: A Bayesian Approach to Data Point Selection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian

LLM

Data selection

Data unbalancing

Data denoising

Domain adaptation
Abstract: Data point selection (DPS) is becoming a critical topic in deep learning due to the ease of acquiring uncurated training data compared to the difficulty of obtaining curated or processed data. 
Existing approaches to DPS are predominantly based on a bi-level optimisation (BLO) formulation, which is demanding in terms of memory and computation, and exhibits some theoretical defects regarding minibatches.
Thus, we propose a novel Bayesian approach to DPS. We view the DPS problem as posterior inference in a novel Bayesian model where the posterior distributions of the instance-wise weights and the main neural network parameters are inferred under a reasonable prior and likelihood model.
We employ stochastic gradient Langevin MCMC sampling to learn the main network and instance-wise weights jointly, ensuring convergence even with minibatches. Our update equation is comparable to the widely used SGD and much more efficient than existing BLO-based methods. Through controlled experiments in both the vision and language domains, we present the proof-of-concept. Additionally, we demonstrate that our method scales effectively to large language models and facilitates automated per-task optimization for instruction fine-tuning datasets.

A Bayesian Approach to Data Point Selection

Abstract

Data point selection (DPS) is becoming a critical topic in deep learning due to the ease of acquiring uncurated training data compared to the difficulty of obtaining curated or processed data. Existing approaches to DPS are predominantly based on a bi-level optimisation (BLO) formulation, which is demanding in terms of memory and computation, and exhibits some theoretical defects regarding minibatches. Thus, we propose a novel Bayesian approach to DPS. We view the DPS problem as posterior inference in a novel Bayesian model where the posterior distributions of the instance-wise weights and the main neural network parameters are inferred under a reasonable prior and likelihood model. We employ stochastic gradient Langevin MCMC sampling to learn the main network and instance-wise weights jointly, ensuring convergence even with minibatches. Our update equation is comparable to the widely used SGD and much more efficient than existing BLObased methods. Through controlled experiments in both the vision and language domains, we present the proof-of-concept. Additionally, we demonstrate that our method scales effectively to large language models and facilitates automated per-task optimization for instruction fine-tuning datasets.

1 Introduction

Practical machine learning efficacy is heavily dependent on the choice, quality and quantity of training data, especially so in the case of neural networks that can easily fit every detail of the training set. This leads to challenges from how to learn reliably with imbalanced data [22], noisy data, noisy labels [46], and so on. Similarly there is often a key subset of data, which is most informative for a given learning problem, but buried among a much larger set of less relevant data. If the most salient data could be efficiently identified, learning could potentially be accelerated [15]. All these challenges are only growing in the era of large scale training on web-scraped data, where curation and gold-standard quality control are not feasible.

Data Point Selection (DPS) algorithms aim to address these challenges by filtering or re-weighting the training data to reduce noise, imbalance, irrelevant background data and so on. The most established

family of approaches [19, 15, 41, 45, 62] to DPS falls under the bi-level optimization or meta-learning umbrella, where one wraps the conventional learning problem with an outer loop that optimizes the dataset itself, so as to maximise performance on some validation set. These methods vary in the choice of their outer optimization variable (e.g., data point weights [41, 45, 19] vs mini-batch sampler [15]), the method of computing meta-gradients (e.g., reverse mode differentiation [45, 62] or reinforcement learning [15]), and the customization of their losses and other design parameters for the different scenarios (e.g., label-noise [45, 41], etc). However, all the BLO approaches are quite expensive in computation and/or memory, which limits their applicability to the most salient use case of large models trained on large web data.

In this paper we revisit the DPS problem from the perspective of Bayesian learning. Rather than constructing expensive nested optimization problems whose convergence is hard to analyse, we treat it as a problem of inferring the joint posterior over the main neural network parameters and instance-wise weights induced by a second weight-estimation neural network. This framework has several advantages in terms of being more efficient and scalable than typical BLO competitors and having a clear convergence guarantee. It is also able to address a variety of DPS-related problems from noise and imbalance to data curation - within a single framework.

Our empirical results present proof of concepts for all these capabilities on a variety of learning tasks in vision and language. We also show a use case of automating Large Language Models (LLMs) instruction fine-tuning (IFT) data curation for specific downstream tasks. The available IFT datasets are large, diverse, and of varying quality. This means that a key activity for natural language processing (NLP) researchers and developers is often finding the right composition of IFT data sources to optimize particular use cases. Our framework can automatically resample and curate the wide array of available auxiliary IFT data to optimize performance for each NLP task of interest. To our knowledge, no BLO alternative has been demonstrated on billion-parameter scale LLMs. We name our approach BADS ( Ba yesian D ata Point S election). 3

2 Our Approach

2.1 Problem Setup

For the DPS problem, we assume that we are given two datasets: the train set D t = { z t i } N t i =1 and the meta set D m = { z i m } N m i =1 . Each data point z i can be an input-target pair z i = ( x i , y i ) in the traditional (class-)labeled data scenarios. But in the autoregressive generative model scenarios (e.g., LLM), z i is simply a sequence of tokens in which the inputs and targets are rather implicitly defined (e.g., all the tokens up until current time as input and the next token as target). We will use the notation l ( z i ; θ ) for the loss of the model θ on the data point z i , which must be well-defined in both scenarios.

The meta dataset D m is considered as in-domain , meaning that the distribution of D m matches that of the downstream test task of interest. The size of D m , denoted by N m , is typically small, due to the cost of curation/annotation processes in practice. The train dataset D t consists of out-of-domain samples, possibly noisy, imbalanced, and uncurated, but the size N t is usually large. The goal of DPS is to select a (soft weighted) subset of the train set D t with the guidance of the meta set D m , so that the model trained on the selected train and meta dataset points performs well. Typical baselines include training with the meta-set alone, or union of meta and train sets.

Perhaps one of the most widely adopted DPS techniques is the bi-level optimisation (BLO) formulation of the problem [41, 58]. Letting w i ( ≥ 0 ) be the weight (or importance) variable associated with the training data point z t i , the main intuition is to find the weights w ∈ R N t + such that the model θ trained with the weighted train data with weights w yields the best performance on the meta set. More formally, this leads to the following BLO problem:

However, a critical drawback is that solving this difficult problem is costly in computation and/or memory, and unrealible due the practical heuristics required. Typical BLO solutions to obtain the

hypergradient dl/dω rely on approximate Hessian estimation or reverse mode differentiation with few-step SGD approximation of the inner optimisation. Aside from cost, for practical neural network implementations computed over minibatches , there is no theoretical guarantee for convergence.

2.2 (Our Approach) Bayesian Data Point Selection (BADS)

We view the DPS problem from a completely different perspective, and tackle it via Bayesian learning. Our model's generative process, that is, the graphical model representation, is depicted in Fig. 1. The main neural network model parameters set θ is a random variable, which can generate data points in the meta dataset D m (precisely speaking, the backbone θ generates the target part of each data point). To make use of the train set D t in an appropriate way, we constrain θ to follow a prior distribution governed by the weighted train data with weights w ∈ R N t + which are also random variables. Before observing the meta set D m , the weight vector w follows a

prior distribution p ( w ) - a specific distributional choice for p ( w ) will be discussed later. Given w and D t , our backbone θ has to be compatible with the weight data { ( w i , z t i ) } N t i =1 . This can be interpreted as placing a weighted-data-driven prior on θ , more specifically,

where p ( θ ) is a base prior (e.g., 0-centered Gaussian that amounts to weight decay regularisation), and p ( w i , z t i | θ ) can be defined from the loss, e.g., exp( -w i · l ( z t i ; θ )) , following the conventional tricks [36, 25]. Then given θ , the meta data are generated following the likelihood defined as:

The equations (2) and (3) fully constitute the prior and likelihood for our Bayesian model. Our ultimate goal is to describe the distributions of θ and w after observing all evidences D t and D m , which boils down to the posterior inference p ( θ, w |D t , D m ) . Formally, we have:

The detailed derivations for Eq. (4) can be found in Appendix A. However, it is widely known that (4) does not admit any closed-form expressions. One main difficulty arises from the intractable normalizing constant in (4).

Stochastic Gradient Langevin Dynamic Sampling. For computationally efficient posterior inference, we adopt the stochastic-gradient MCMC technique, specifically the Stochastic Gradient Langevin Dynamic (SGLD) sampling [55]. Applied to our model, we can obtain samples from the posterior p ( θ, w |D t , D m ) by running the Langevin dynamic system (until convergence, i.e., mixing):

where η is a small (constant) step size. There are two critical benefits: i) Since we differentiate the log-posterior, the difficult normalizing constant in (5) will disappear; ii) The update (5) is essentially gradient descent with additive Gaussian noise, leading to a computationally efficient update.

Going one step further, even though the log-posterior involves the entire train data (and entire meta data), it is shown in [55] that the stochastic-gradient version (SGLD) that replaces the whole batch likelihood with a minibatched one, theoretically guarantees that the SGLD update converges to the posterior samples. More specifically, the SGLD update equations (one for θ and the other for w ) can be written as follows:

where B t and B m are minibatches from D t and D m , respectively, and ϵ θ , ϵ w ∼ N (0 , I ) are independent Gaussian samples.

Repeating (6) and (7) for a sufficient amount of iterations (until we reach good mixing) leads us to posterior samples ( θ, w ) . There are several options to take these samples for a final model for test prediction. One option is to collect latest M samples (either consecutive collection or thinning to take every k th samples) from the iterations, and either take the average as posterior means or perform full Bayesian treatment with the collected samples. Alternatively, we can just take the last single iterate ( θ, w ) as a point representative for the posterior distribution. For simplicity, we take the latter approach, which also works well empirically.

2.3 Interpretation and Benefits

Interpretation Wediscuss several intuitions and implications of our proposed approach (Eq. 6-7). First, looking at the θ update Eq. (6), our model essentially updates θ in a way that it decreases the loss on the combined data of the whole meta data points and the weighted train data points with the current weights. This is a fairly intuitive strategy provided that the weights are properly determined. Then the next question is how the weights are determined. If we inspect the w update (Eq. 7), and take the gradient of the expected loss term with respect to w directly, we see that: i) those train data points z t i s with smaller losses at current backbone θ will get higher weights w i s; ii) those train data points z t i s with larger losses at current backbone θ will get lower weights w i s. This essentially means that our model performs loss alignment for DPS - In the course of training/update, once the backbone θ enters a good regime in the parameter space such that θ can assign (valid) low loss values on the in-domain meta data points, then it starts putting high weights on those train data points that have low losses under the current backbone. In other words, the model will assign high weights to those train data points that are well-aligned with the meta data points in terms of loss.

Benefits over BLO Our Bayesian approach provides several benefits over BLO: (1) Efficiency. Our SGLD is efficient so does not require computationally demanding Hessian computations like implicit function theorem based methods (cf: [19]) or huge memory demand like reverse-mode differentiation methods [41, 19]. (2) Sparsity. Our method straightforwardly achieves sparsity on the w weights allowing efficient sample selection unlike [19]. (3) Reliability. BLO-based methods rely on approximations (truncation, or Hessian approximations) for practical feasibility that make finding optimal solutions unreliable. Our straightforward Bayesian approach has reliable convergence properties thanks to being a standard application of SGLD.

Convergence of our SGLD algorithm In Appendix C we provide a theorem showing that our SGLD algorithm converges to the true posterior. Our analysis is based on [64] where we make some adjustments for our case.

2.4 Implementation Details

Choice of Priors For the base prior p ( θ ) , i.e., the prior before being driven by the weighted data, we adopt 0-mean Gaussian, which amounts to adding the weight decay regularisation for θ . For the weight prior p ( w ) , we have made a careful design effort to come up with a viable sparsity inducing prior. Although encouraging sparsity in learned weights is ideal to avoid overfitting, during our initial experiments we have found that most of the weights eventually tend to vanish to 0, which is not what we actually want. We need to be able to impose both sparsity and a certain level of non-zero weights. To this end, we first introduce a hyperparameter β (e.g., 0.01) for the target sparsity level that we want to attain. Roughly saying, among the N t training data points, we aim to select ⌊ N t · β ⌋ . For the (soft) weights, we impose ∑ i ∈D t w i ≈ ⌊ N t · β ⌋ , which can be encoded in the prior form as:

where σ controls the strength of the regularisation. One technical difficulty in directly plugging (8) into the weight update (7) is that we have to load the whole { w i } N t i =1 in memory for backprop. To avoid this issue, we use the following fact:

where ¯ w represent historic running average of the entire weights. We basically build a computation graph only for the first term of batch B t weight sum, and regard the (historic) running average of the entire weights as constant during backprop. After each SGLD iteration, we update the running weight average with the new updated weights on the recent batch. We use the simple averaging scheme for the running average. To approximate the average weight ¯ w precisely, we only conduct the average over the most recent s avg step.

Introducing impact constants In the SGLD principle, we have the log-likelihood terms that are proportional to the sizes of the datasets. In particular, we have N t and N m in (6). However, this scheme does not properly capture our preference to the in-domain meta data set in contrast to the noisy, out-of-domain train data set. To this end, we introduce the impact constants (hyperparameters) in the update equations where we downweigh or upweigh the loss terms of train and meta sets.

Weight Network Instead of directly optimising individual weights w i , we can consider a weight network, w i = w ( z t i ; ϕ ) , a neural network with parameters ϕ that takes the train data point z t i as input and returns its weight w i as output. We can then regard ϕ as random variables and the w update equation can be modified accordingly for ϕ update straightforwardly. This weight network approach can be useful for smoothing/regularising the output weights thanks to the smooth functional property of neural networks. Furthermore, if one needs to supplement the train dataset with extra new train samples after the model training, the learned weight network can be used for assigning weights or selecting samples from the new set, without retraining the whole model from the scratch. The posterior distribution similar to Eq. (4) is derived in full detail in Appendix B.

3 Experiments: Proof of Concept

In this section, we assess the effectiveness of the proposed BADS method in three critical scenarios where DPS is essential: Data Balancing, Data Denoising, and Efficient Learning. We begin by introducing the baseline systems and then present the experimental results for each of these scenarios.

3.1 Baselines

There are three types of DPS setups with different supervision signals:

Unsupervised DPS selects data without the guidance of a held-out meta set [47, 42]. Instead, it is guided by human-defined hypotheses, such as 'challenging examples improve model performance'. This approach aligns with curriculum learning. We include the online variant of AskLLM [42], i.e. AskLLM-O , in our baseline comparisons. It selects examples from training set by querying a pretrained OpenLLaMA 3B to obtain the sampling score for each training sample. 4

Self-supervised DPS selects data with the guidance of a held-out meta set. However, the meta set does not share the same data distribution as the targeted test set [13, 4, 14, 49]. Typically, the examples in the meta set are selected from the training set based on specific hypotheses, such as 'learnable examples enhance model performance'. We include two approaches in our baseline comparisons: Contrastive Data Selection (CDS) [49] is tailored for data denoising. The algorithm assigns weights to each data point in D t according to the difference between the denoised and the noisy log probability, predicted using a denoised and a noisy model trained on a clean dataset and an uncurated dataset, respectively. These weights are then used to sample data points from minibatches in the training of LM. 5 Similar to CDS, ClassAct utilizes small proxy models trained on a limited portion of D t to calculate learnability scores for the remaining training data points. 6

Meta-set guided DPS selects data with the guidance of a small meta set that shares the same distribution as the test set, aiming to train a model that excels specifically on the target test set. The test set may encompass one or multiple downstream domains or tasks. This DPS is closely related to meta learning, domain adaptation, and transfer learning. Current methods primarily rely on Bilevel Optimization (BLO) for purposes such as data denoising [19, 37], data balancing [41],

Test/ACCforMNIST

Test/ACC for CIFAR-10

Test/BLEU for WebNLG

38

BLO

contrastive

0.9

random_select

Test/ACC

0.8

Bayesian

contrastive

meta_only

BLO

Test/ACC

0.6

Test/BLEU

36

34

mixing

Bayesian

ClassAct

meta_only

0.7

mixing

ClassAct

random_select

0.5

BLO

mixing

contrastive

32

0.6

0.4

ClassAct

Bayesian

30

random_select

0.5

meta_only

0

2000

4000

6000

8000

10000

1200014000

2500

5000

7500

10000125001500017500

5000

10000

15000

20000

25000

30000

35000

40000

Weights of DataPoints in WebNLG Predicted by BADS

Weights of Data Points in WebNLG Predicted by BLO

Weights of Data Points in WebNLG Predicted by CDS

0.0495

00500

= weights for example with Label=City Domain

weights for example with Label=Artist Domain

0.055

weights for example with Label=City Domain

weights for example with Label=Artist Domain

90.0

Wawiamtinmmy

Weights

0.0485

0.0490

0.050

90.5

0.0480

0.045

91.0

weights for example with Label=Artist Domain

- weights for example with Label=City Domain

0.0475

0.0470

0.040

91.5

0.0465

92.0 -

0.035

0.0460

40250

40500

40750

4100041250

41500

41750

42000

40250

40500

40750

41000412504150041750

42000

4025040500

40750

41000

41250

41500

41750

42000

Steps

and efficient learning [28, 59, 29]. Considering both performance and code availability, we use the online BLO 7 [41, 19] as our baseline. Our approach, BADS , also falls under this category.

To ensure a fair comparison, we also introduce several baselines that train the backbone models using different combinations of the meta set and training set: Mixing trains the model using a combination of the train set D t and the meta set D m . Meta_Only trains the model exclusively on D m . Random_Select uses D m combined with a randomly selected subset from D t . Duplicate_Meta utilize D t along with multiple copies of the meta set, duplicating D m until it matches the size of D t .

Note that, the selection ratio/sparsity level in AskLLM-O , ClassAct , Random_Select , BLO , and CDS is the same as in BADS . 8

3.2 Scenario 1: Data Balancing (MNIST)

In this scenario, we assess the model's capability to manage an imbalanced train set, D t . Despite being trained on this imbalanced dataset, the models are expected to perform effectively on a balanced test set. Following the setup in [41], we use the standard MNIST handwritten digit classification dataset [33] to create a class-imbalanced binary classification task. A total of 5,000 images from classes 4 and 9 were selected as the train set D t , with class 9 dominating the training data distribution (4,975 examples) and class 4 having only 25 examples. A balanced meta set D m is created by selecting another 25 examples from each of these two classes, ensuring no overlap between D t and D m . The models are tested on the original MNIST test set, including only classes 4 and 9.

MNIST ACC with different number of examples per class in meta set

0.9

mixing

0.8

random_select

meta_only

dup_meta

0.7

contrastive

BLO

Bayesian

0.6

0.5

10

20

30

40

50

number of examples per class in meta set

In this scenario, the loss function is defined using the standard binary cross-entropy loss. Following [41], all classification models are LeNet5 [32]. The training is conducted on a single GPU, using SGD with a fixed learning rate of 1e-3 and a mini-batch size of 100, over a total of 15,000 steps. In BADS , the weight network is implemented as a single-layer Feedforward Neural Network (FNN) with a sigmoid activation function. It takes the top-layer image embeddings from LeNet5 as input

and outputs a weight w i ∈ [0 , 1] for each image. The learning rate for the weight network is 1e-3 and the target sparsity level β is 0.005. Other hyperparameters, including those in the baselines, are detailed in Table 3 (Appendix E).

3.2.1 Experiment Results and Ablation Study

The classification accuracy is presented in the top-left plot in Figure 2. All approaches, except for Mixing and ClassAct , achieve over 90% accuracy even when trained with a highly imbalanced train set. Meta_only demonstrates that training with a small amount of balanced data yields significantly better performance compared to training with a larger but imbalanced training set ( Mixing ). Both BLO and BADS outperform non-DPS baselines in terms of both accuracy and convergence speed, with BADS further outperforming BLO by a noticeable margin. CDS underperforms all the non-DPS baselines, which we believe is due to the mini-batch-level discrete data selection. When dealing with an extremely imbalanced train set, it is possible that the minority class might not be present in some mini-batches. Under these circumstances, the model is compelled to learn from the top examples from the majority class in each mini-batch, potentially leading to biased training outcomes. The top row in Figure 13 and the left plot in Figure 14 (Appendix F) visualizes the weights assigned to the examples in each mini-batch by DPS approaches. All methods, except for ClassAct , effectively assigns higher weights to the minority class than to the majority class, thereby directing the classifiers to focus more on the minority class in training.

We further evaluate the models' performance using meta sets D m of various sizes, with 5, 10, 25, and 50 examples per class included. 9 As illustrated in Figure 3, with a very limited number of meta examples (5 per class), only BADS and BLO achieve over 90% accuracy on the balanced test set. As the number of available meta data increases, BADS consistently leads in performance. However, when sufficient meta data is provided, the gap between BADS and the other approaches narrows.

3.3 Scenario 2: Data Denoising (CIFAR)

In this scenario, we evaluate the model's ability to manage a noisy train set D t . Although the models are trained using a dataset with significant noise, they are anticipated to perform well on a clean test set. Our experiment utilizes the standard CIFAR 10-class classification dataset [30]. Following the standard CIFAR train/validation set split ratio, we first create a clean and balanced meta set D m by randomly sampling 1000 examples from each class in the training data. Then, we use the remaining 40,000 examples to create a noisy train set D t by introducing noise based on the symmetric noise injection setup described in [37, 8, 21]. We set the noise ratio to 0.5 and use the original CIFAR-10 test set in testing.

We use ResNet32 [23] as the backbone classification model. Training is performed on a single GPU using SGD with a fixed learning rate of 1e-1 and a mini-batch size of 120 over 20,000 steps. The loss function is the standard multi-class cross-entropy loss. For BADS, the weight network structure retains the same as in scenario 1. However, in this scenario, data point weighting takes into account both the image and its associated label. We represent each label using a one-hot embedding, which is then concatenated with the top-layer image embeddings from ResNet32 and fed into the weight network. The learning rate for the weight network is set to 1e-4. Note that in Eq 7, the gradient of θ become large if N t is big. Therefore, we reduce the learning rate of the backbone classification model to 1e-4. The target sparsity level β is set to 0.8. 10 .

3.3.1 Experiment Results and Ablation Study

The classification accuracy is shown in the top-middle plot in Figure 2. All methods, except for the BLO approach 11 , manage to achieve over 60% accuracy, even using a train set contaminated by 50%.

Notably, CDS , ClassAct and BADS deliver the highest three performances, with BADS surpasses all other methods by a noticeable margin. The bottom row of Figure 13 and the middle plot of Figure 14 (Appendix F) visualizes the weights allocated to the examples in each mini-batch by DPS approaches.

All methods, except for ClassAct , consistently gives lower weights to the noisy examples, guiding the classifiers to disregard the noisy examples during training. ClassAct assigned lower weights to the noisy examples during the initial stages of training. However, these weights exceeded those of the clean examples in the later phases of training. Interestingly, this behavior did not significantly impact the model's performance.

If we raise the noise label ratio in the train set to 80% (Figure 4), both BLO and BADS still lead the performance, with BADS exceeds BLO and non-DPS approaches by 15% and 20% in classification accuracy, respectively. CDS does not exceed the

performance of non-DPS approaches and starts to overfit on the noisy data after 15,000 training steps. When we lower the noise label ratio to 20% (Figure 5), all methods achieve a classification accuracy of around 80%. Although BADS continues to outperform both DPS and non-DPS approaches, the lead is narrower. 12

3.4 Scenario 3: Efficient Learning (WebNLG)

In this scenario, we assess the model's ability to adapt to new domains with very few data points. To demonstrate the robustness of BADS across various research topics and backbone models, we focus on the Natural Language Generation (NLG) task in this section. NLG [18, 12] aims to accurately generate textual descriptions from input tuples. An example is shown in Figure 7 (Appendix D). The English benchmark introduced in WebNLG 2020 [6] includes a training set spanning 16 domains and a test set covering 3 different domains (for details, check Appendix D). We use the original training set (14,239 examples) as our train set D t , and create a single clean and balanced meta set D m by randomly sampling 30 examples from the WebNLG 2020 validation set in each test domain.

All backbone models are the encoder-decoder T5-small [26]. Training is on a single GPU using Adam with a fixed learning rate of 3e-5 and a mini-batch size of 20 over 40,000 steps. The loss function is the standard negative log likelihood loss. In BADS, the weight network structure remains the same as in scenario 1. Its input is the embedding of the input sequence, represented by the contextual embedding of the '[EOS]' token, and the learning rate is set to 1e-4. The sparsity level β is 0.05.

3.4.1 Experiment Result

The BLEU scores are displayed in the top-right plot of Figure 2. BADS leads the performance, achieving a 2 BLEUscore advantage over the second-best system, Dupli-cate_Meta , and surpassing the remaining systems by more than 5 BLEU scores. The other three DPS approaches do not distinguish themselves from the non-DPS methods.

In this scenario, we opt for a more controlled examination of data selection effectiveness due to the big amount of domains in the train set 13 . We select three occupationcentric domainsAthlete , Politician , and Astronaut -as our test domains. Additionally, we create a meta set D m by randomly selecting 50 examples in each of these domains

from their WebNLG 2020 validation set. We then choose two distinct domains, Artist and City , for training. Artist , as another occupation-centric domain, shares a similar schema and vocabulary with the test domains, whereas City does not. Ideally, DPS should prioritize the Artist examples over those from City . The BLEU scores for text descriptions generated by the DPS methods are shown in Figure 12 (Appendix F). These results reinforce the findings from the original experimental setup, with BADS outperforming both BLO and CDS by over 10 BLEU scores. The weights given to the

examples in each mini-batch are visualized in the bottom row of Figure 2. BADS effectively prioritizes the examples in the Artist domain. Instead, BLO fails to differentiate between the two domains, and CDS incorrectly weights the opposite domain higher. This illustrates that the effectiveness of DPS is linked to the overall performance of the models.

3.4.2 Latency

We evaluate the average GPU memory and time consumption using the WebNLG task. Table 1 shows that the training time for BADS and CDS is similar to that of non-DPSs, while BLO and AskLLM-O takes nearly twice as long, and ClassAct takes even longer. Note that CDS and AskLLM-O requires additional time to weight examples in the train set. Given that the WebNLG 2020 train set consists of 35,426 examples, we execute the weighting with a batch size of 20, CDS takes a total of 700 seconds. For larger-scale tasks, such as foundation model fine-tuning (Section 4), the weighting process is estimated to consume approximately 22 hours per 1 million training examples. The offline scoring in AskLLM-O takes around four hours in our setup on a single NVIDIA A40 GPU. In terms of memory usage, CDS aligns with non-DPS approaches, while BADS and BLO require approximately 1.5 and 2.5 times more GPU memory, respectively. ClassAct and AskLLM-O takes even more GPU memory.

4 Use Case: Large Language Model Instruction Fine-tuning

Instruction Fine-tuning (IFT) for LLMs is a practical application where all three mentioned scenarios are encountered simultaneously. IFT data can be acquired through prompting LLMs [52, 48, 56], gathering existing Natural Language Processing (NLP) benchmarks [35, 54, 50, 43, 53, 38, 56], or employing human annotation [52, 63, 48]. Noise is likely to accumulate during each of these data collection methods. Furthermore, since NLP benchmarks often vary greatly in size, the IFT data typically lacks

balance. Additionally, the IFT data does not include data points from the downstream tasks, leading to domain shift in testing.

We use the same IFT data as [57, 51] as our train set D t , which is a mix of FLAN V2 [35], COT [54], DOLLY [10], and OPEN ASSISTANT 1 [31]. Following [57, 5], we focus on four downstream tasks: MMLU[24], which consists of multiple-choice questions across 57 sub-tasks, ARC-challenge/-easy [9], and HellaSwag [61]. Following [57], 5 examples were selected from each sub-task to create the meta set D m for MMLU, totaling 285 examples. Additionally, following [5], for the other tasks, we randomly chose 25 examples from their validation set to create the respective meta sets. To facilitate checkpoint selection, we additionally create a validation set of equivalent size to the meta sets.

Due to limited computational resources, we use OpenLLaMA 3B 14 as the backbone model. Training uses one A40 GPU utilizing Adam with a fixed learning rate of 3e-5 and a mini-batch size of 3. In BADS, while the weight network remains the same as described in scenario 3, we modify the input to be the average contextual embedding of all tokens in the sequence. The sparsity level β is 0.05. Results are presented in Table 2. We excluded BLO from this experiment due to their prohibitive GPU memory usage. Given that the training data for the Mixing baseline predominantly consists of IFT data points, it is considered as a standard IFT process. Table 2 indicates that while individual methods may excel in specific tasks, none of the non-DPS baselines nor the CDS and ClassAct consistently surpass the others across all downstream tasks. However, BADS stands out by consistently outperforming all other baselines across every task, except for AskLLM-O, which, as indicated in Table 1, demands significantly more computational resources. According to Table 4 (Appendix F) BADS prefers the human-written-from-scratch, OPEN ASSISTANT 1 and DOLLY, over the rest two created using existing benchmarks.

To facilitate the use of our proposed datapoint selection method, we provide comprehensive guidelines for hyperparameter tuning in Appendix E. We also include an in-depth discussion and ablation study on the influence of hyperparameters on the method's effectiveness.

5 Related Work

Recent works on DPS broadly fall into four categories: i) Approaches based on meta learning (or BLO), ii) Gradient-based methods, iii) Methods based on domain adaptation and transfer learning methods, and iv) General sample reweighing strategies.

· Meta learning (BLO) approaches. The DPS can be formulated as a meta learning BLO problem where the outer loss is defined with the training data selection variables, and the inner optimisation is to minimize the model's loss with the selected data [19, 15, 41, 45, 62]. These methods vary in the choice of their outer optimization variables: either directly using the data point weights [41, 45, 19] or minibatch samplers [15]. To solve the BLO, most approaches rely on computing meta-gradients via reverse mode differentiation [45, 62] while some works utilised reinforcement learning techniques [15]. All these BLO-based methods are computationally demanding with large memory footprint, hindering them from being applied to large-scale models/data.

· Gradient-based methods. The key idea is to measure the importance of training data points based on the alignment scores between the loss gradients on training and meta data points. The rationale behind the gradient alignment can be theoretically underpinned by the BLO perspective. As shown in [41], the one-step inner loss update with zero initial weights in BLO reduces to the gradient alignment (cosine angle) between the train and meta data points [57]. However, the method in [57] requires evaluating and storing gradients of the entire training data points, thus computationally expensive. Furthermore, the final solution may not be optimal since the gradient computation is done with an initial network that is warm-up trained with a random subset of the training data. In [16] the weights are optimised by the expected gradient norms during the network training, which serves as a proxy for the importance of data points. In [28] they find the subsets that closely match the gradient of the training and meta sets using an orthogonal matching pursuit algorithm. Under online continual learning setup [2], they formulate sample selection as a constraint reduction problem based on the constrained optimization view of continual learning.

· Domain adaptation and transfer learning methods. Selecting a subset of the train set that are best aligned with the in-domain meta set can be naturally seen as domain adaptation and transfer learning . In the NLP community, [1] finds that the large language models implicitly learn sentence representations that cluster by domains without supervision, while in [20], they show the effectiveness of domain-adaptive pre-training with data augmented using simple data selection strategies. In [39] they proposed domain adaptive transfer learning which computes the importance weights on the target dataset from the ideas in domain adaptation and estimation of the label distribution. In multi-task transfer learning community, some previous works identify the detrimental effects of the gradients from different tasks, and propose strategies to mitigate the conflicting gradients issue [60, 11, 34].

· General sample reweighing strategies. Since DPS essentially involves finding the optimal reweighed distribution with the underlying domain itself unchanged, several approaches aim to tackle the problem via importance sampling techniques. In [27] they derive a tractable upper bound to the per-sample gradient norm, and derive an estimator of the variance reduction achieved with importance sampling. The curriculum learning [3] is also closely related as one can design an optimal schedule of the successive training distributions. For instance, in [44] they introduce the so-called data parameters, associating samples and classes in the train data with learnable parameters, which governs their importance in the learning process within the curriculum learning framework.

6 Conclusion

We revisited the DPS problem from the perspective of Bayesian learning. By treating the neural network of interest, as well as an auxiliary weighting neural network as random variables and inferring their joint posterior using SGLD, we achieve a simple and effective approach to data reweighting that is more reliable and scalable than BLO alternatives. Our framework is straightforwardly applicable to learning with data imbalance, label noise, and automating auxiliary data curation. We demonstrate our framework can apply to automating curation of the wide variety of auxiliary instruction fine-tuning data available for billion-scale language models. Overall this demonstrates a promising new kind of approach to the growing need for data optimization in neural network learning.

Limitations

Our proposed BADS algorithm has the following limitations, which we plan to address as our future work.

There are several hyperparameters involved, which need to be carefully tuned for best performance. These include: the sparsity level parameter β , the relative impact constants ρ 's of the objective terms in our SGLD update (detailed in App. E), and the standard hyperparameters (e.g., batch size, learning rates, weight decay). Some of these hyperparameters may be optimised via Bayesian model selection in a principled manner. For instance, the sparsity level β can be regarded as a latent random variable (a part of the model) with a proper prior distribution imposed on it, and we can do posterior inference of β (or marginalise it out) together with w and θ in our SGLD update equations. We will pursue this in our future study.

Although our method is computationally far more efficient than BLO and some other DPS approaches, its GPU memory footprint is demanding compared to non-DPS algorithms. This mainly originates from the entire weight variables or the whole weight network parameters loaded/maintained in the memory for frequent updates. One possible workaround is to load only the weight variables that are assoicated with the current minibatches. We will be investigating this code optimisation further in our ongoing future study.

References

Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. In Annual Meeting of the Association for Computational Linguistics , 2020.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances on Neural Information Processing Systems , 2019.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning , 2019.

David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. Color-filter: Conditional loss reduction filtering for targeted language model pre-training. arXiv preprint arXiv:2406.10670 , 2024.

Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models, 2024. URL https://openreview.net/forum?id=kce6LTZ5vY .

Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , 2020.

Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv preprint arXiv:2305.09246 , 2023.

Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning , 2019.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. CoRR , abs/1803.05457, 2018.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023. URL https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm .

Lucio M. Dery, Yann Dauphin, and David Grangier. Auxiliary task update decomposition: The good, the bad and the neutral. In International Conference on Learning Representations , 2021.

Ondˇ rej Dušek, Jekaterina Novikova, and Verena Rieser. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. Comput. Speech Lang. , 59(C):123-156, jan 2020.

Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, and Olivier J Henaff. Bad students make great teachers: Active learning accelerates large-scale visual understanding. arXiv preprint arXiv:2312.05328 , 2023.

Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier J Henaff. Data curation via joint example selection further accelerates multimodal learning. arXiv preprint arXiv:2406.17711 , 2024.

Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie-Yan Liu. Learning what data to learn. In arXiv , 2017.

Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi, Mohammad Taher Pilehvar, Yadollah Yaghoobzadeh, and Samira Ebrahimi Kahou. Bert on a data diet: Finding important examples by gradient-based pruning. ArXiv e-prints: arXiv:2211.05610, 2022.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836 .

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. In International Conference on Natural Language Generation , 2017.

David Grangier, Pierre Ablin, and Awni Hannun. Adaptive training distributions with scalable online bilevel optimization. arXiv preprint arXiv:2311.11973 , 2023.

Suchin Gururangan, Ana Marasovi´ c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Annual Meeting of the Association for Computational Linguistics , 2020.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances on Neural Information Processing Systems , 2018.

Haibo He and E.A. Garcia. Learning from imbalanced data. 21(9):1263 -1284, 2009.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL http://arxiv.org/abs/1512.03385 .

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021.

A. Immer, M. Bauer, V. Fortuin, G. Ratsch, and M. E. Khan. Scalable marginal likelihood estimation for model selection in deep learning. In International Conference on Machine Learning , 2021.

Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. In International Conference on Natural Language Generation , 2020.

Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning , 2018.

Krishnateja Killamsetty, Durga S, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning , Proceedings of Machine Learning Research, 2021.

KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh K. Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In AAAI , 2021.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant conversations - democratizing large language model alignment. CoRR , abs/2304.07327, 2023.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems , 2021.

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , 2023.

J. Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning , 2015.

Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. In NeurIPS , 2020.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Annual Meeting of the Association for Computational Linguistics , 2022.

Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. In arXiv preprint , 2018. URL https://arxiv.org/ abs/1811.07056 .

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Advances on Neural Information Processing Systems , 2021.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to Reweight Examples for Robust Deep Learning. International Conference on Machine Learning , 2018.

Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668 , 2024.

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022.

Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In NeurIPS , 2019.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. In NeurIPS , 2019.

Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. TNNLS , 2022.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=UmvSlP-PyV .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github. com/tatsu-lab/stanford_alpaca , 2023.

Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. In Conference on Machine Translation: Research Papers , 2018.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. SuperNaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Conference on Empirical Methods in Natural Language Processing , 2022.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. In Neural Information Processing Systems - Datasets and Benchmarks Track , 2023.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics , 2023.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances on Neural Information Processing Systems , 2022.

Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning , 2011.

Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Aji. LaMini-LM: A diverse herd of distilled models from large-scale instructions. In Conference of the European Chapter of the Association for Computational Linguistics , 2024.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. CoRR , abs/2402.04333, 2024.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting Influential Data for Targeted Instruction Tuning. arXiv preprint arXiv:2402.04333 , 2024.

Hu Xu, Saining Xie, Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Cit: Curation in training for effective vision-language data. In IEEE International Conference on Computer Vision , 2023.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems , 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In ACL (1) , pages 4791-4800. Association for Computational Linguistics, 2019.

Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 725-734, 2021.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Advances on Neural Information Processing Systems , 2023.

Difan Zou, Pan Xu, and Quanquan Gu. Faster convergence of stochastic gradient langevin dynamics for non-log-concave sampling. Uncertainty in Artificial Intelligence (UAI) , 2021.

Appendix

A Posterior Derivation for Eq. 4

From the graphical model in Fig. 1, we exploit the conditional independence D m ⊥ ( w,D t ) | θ .

In (13) we use D m ⊥ ( w,D t ) | θ , and in (14) 1 p ( D m | D t ) is regarded as constant for it has nothing to do with θ and w .

B Posterior Derivation for Weight Network Cases

Here we provide posterior derivation for the weight network case w i = w ( z t i ; ϕ ) . First, the weights become a deterministic function of D t and ϕ (here ϕ = weight network parameters) as shown in Fig. 6(a). But since w is a deterministic function of ϕ and D t , we can simplify it by having w absorbed into ϕ while introducing conditional dependence (an arrow) from D t to ϕ , as depicted in Fig. 6(b). Note that D t is always given, we treat ϕ as a random variable, and wherever w i appears, we replace it by w ( z t i ; ϕ ) . More specifically, we make the following changes to the equations in the weight net scenario:

Eq. 4 (with detailed derivations):

(a)

(b)

C Convergence Analysis

Assumption C.1 (Adjusted from Assumption 4.3 in [64]) . There exists m> 0 and b ≥ 0 such that

holds for any θ, w .

Assumption C.2 (Adjusted from Assumption 4.4 in [64]) . Any minibatch gradient of the logposterior is Lipschitz continuous. That is, there exists a constant L such that for any z t i ∈ D t and z j m ∈ D m ,

holds for any θ, w, θ ′ , w ′ .

Theorem C.3 (Adjusted from Theorem 4.5 in [64]) . Let d = dim( θ ) + N t , B be the batch size, and ρ be the Cheeger constant. For any ϵ ∈ (0 , 1) , with the initial iterate satisfying p (∥ ∥ ∥ ∥ [ θ init w init ]∥ ∥ ∥ ∥ ≤

R/ 2 ) ≤ ϵ/ 16 for R = R ( ϵK -1 / 12) , and step size η = ˜ O (min { ρ 2 d -2 , B 2 ρ 2 d -4 } ) , the distribution µ SGLD K of the K -th iterate in our SGLD iterations Eq. (6-7) satisfies:

for some constant λ > 0 , C 0 = ˜ O ( ρ 2 ) , C 1 = ˜ O ( Rdρ -1 ) , C 2 = ˜ O ( dρ -1 ) . Here ∥ · ∥ TV stands for the total variation distance, and R is defined as:

D Details of the Tasks

WebNLG 2020 We use the release_v3.0_en version of WebNLG benchmark. Domains in training set are: Building, Astronaut, City, University, MeanOfTransportation, SportsTeam, Food, Artist, Company, ComicsCharacter, Monument, Airport, Politician, Athlete, WrittenWork, CelestialBody . Domains in test set are MusicalWork, Scientist, Film .

E Details of the Experiments

E.1 Hyperparameters Details

To enhance flexibility in managing the training process, we incorporate ρ t θ , ρ m θ , and ρ t w into Eq 7 as follows:

The training hyperparameters for BADS and CDS are shown in the table below. Other hyperparameters have been shown in the main paper.

E.2 Guildline for Hyperparameters Tuning

There are two primary sets of hyperparameters in BADS :

hyperparameters for model's parameters update: η , ϵ θ , ϵ w , ρ t θ , ρ m θ , and ρ t w (Eq. 24-25),

hyperparameters for the prior distribution of the example weights, which are σ , β , s avg (Eq. 8-9).

In general,

we set η to 1 and kept the Gaussian noise small with ϵ θ and ϵ w equal to 1e-5;

ρ t w is set to 1;

we set ρ m θ to 1 and primarily adjust ρ t θ ;

in most cases, ρ t θ is simply set to 1. However, if the training set contains noise-where the ground truth labels might be incorrect-the loss from the training examples, particularly in the early stages of training, may be unreliable. In such cases, we decrease ρ t θ ;

s avg should not be too large, as it may incorporate outdated weights from earlier training steps. We set it to 10;

we select β based on the proportion of training data we consider beneficial for the downstream tasks. For the LLMs Instruction Fine-tuning experiments, we adopt the same ratio used in the previous studies [52, 46].

σ controls how tightly the weights should match β . We set σ based on our confidence in the selection ratio β : a smaller σ indicates greater confidence in β .

data denoising scenario is a bit special, Eq 7 shows that high losses from the training batch push the example weights w toward 0. With a high noise rate (50% and 80%), the training batch losses remain high throughout the training process. To prevent the weights w from collapsing to 0, we use a high selection ratio β and a low σ .

E.3 Ablation study: Influence of the impact constants σ and sparsity level β .

Impact constant σ Higher σ causes the example weights w to drift away from β , occasionally collapsing to 0, and may result in incorrect example weights (see Figure 8). The right column in Figure 11 shows that in all three proof-of-concept scenarios, the models achieve similarly good performance when σ is reduced to around 10 -5 .

Sparsity level β In both WebNLG and MNIST scenarios, high β leads to incorrect example weights (see Figure 9). Conversely, due to the reason we mentioned above, in data denoising (CIFAR) scenario, lower β leads to incorrect weights. The left column in Figure 11 shows that the backbone models' performance significantly declines in both the WebNLG and MNIST scenarios when the example weights are incorrect. In the CIFAR scenario, the impact is less pronounced.

Data Denoising (CIFAR with 50% noise)β= 0.8 = 0.5

Data Denoising (CIFAR with 50% noise)β= 0.8  = 0.005

Data Denoising (CIFAR with 50% noise)β= 0.8 = 5 *10-s

0.382

-= weights for example with Label=Noisy Annotation

0.453

weights for example with Label=Noisy Annotation

0.778

weights for example with Label=Noisy Annotation

0.380

weights for example with Label=Clean Annotation

0.452

weights for example with Label=Clean Annotation

0.776

weights for example with Label=Clean Annotation

0.451

0.774

0.378

0.450

0.772

0.376

0.449

0.770

0.374

0.448

0.768

0.766

0.372

0.447

0.764

0.370

0.446

0.762

18250

18500

18750

19000

19250

19500

19750

20000

18250

18500

18750

19000

19250

19500

19750

20000

18250

18500

18750

19000

19250

19500

19750

20000

Steps

Efficient Learning (WebNLG 2 training domains)β = 0.8  = 1 * 10-s

Efficient Learning (WebNLG 2 training domains)β= 0.2 = 1 *10-5

0.804

0.2000

weights for example with Label=Artist

weights for example with Label=City

0.0495

0.802

0.1995

0.0490

0.800

0.1990

0.0485

0.1985

weights for example with Label=Artist

0.0480

weights for example with Label=Artist

8640

weights for example with Label=City

0.1980

weights for example with Label=City

0.0475

0.1975

964°0

0.0470

0.794

0.1970

0.1965

0.0465

0.0460

38500

0006E

00S6E

39750

40000

38250

38500

38750

39000

Steps

39250

0056E

39750

40000

38500

0006E

39250

39500

39750

40000

F Analysis of Main Results

F.1 Learning Curves Study

The experimental results discussed in Section 3.3.1 demonstrate that BLO converges more slowly compared to other approaches. In this section, we aim to examine the learning curves of DPS approaches, as illustrated in Figure 11. The trends for each approach appear consistent across both

Test/ACCforMNIST(hparamsensitivity,beta)

Test/ACCforMNIST(hparamsensitivity,sigma)

1.0

0.9

60

Test/ACC

0.8

beta_0.001

beta_0.1

Test/ACC

0.8

sigma_5e-3

beta_0.01

sigma_5e-1

0.7

beta_0.005

beta_0.5

0.7

sigma_5e-6

sigma_5e-2

sigma_5e-4

0.6

0.6

0.5

0.5

0

2000

4000

6000

8000

10000

12000

14000

2000

4000

6000

8000

10000

12000

14000

steps

steps

Test/ACCfor CIFAR(hparam sensitivity,beta)

Test/ACC for CIFAR (hparam sensitivity, sigma)

beta_0.2

0.75

0.75

beta_0.8

0.70

.....

beta_0.9

beta_0.5

0.70

0.65

0.65

Test/ACC

0.60

Test/ACC

0.60

0.55

0.55

0.50

0.50

0.45

sigma_5e-3

0.45

0.40

sigma_5e-1

0.40

sigma_5e-6

0.35

2500

5000

7500

100001250015000 17500 20000

2500

5000

7500

10000

12500

15000

17500 20000

steps

steps

Test/BLEU forWebNLG_2domain(hparamsensitivity,beta)

Test/BLEU for WebNLG_2domain(hparam sensitivity,sigma)

46

beta_0.2

beta_0.005

44

44

beta_0.8

beta_0.5

42

42

Test/BLEU

Test/BLEU

40

40

38

38

sigma_1e-4

sigma_1e-5

36

sigma_1e-2

sigma_1e-6

34

36

sigma_1e-1

sigma_1e-3

5000

10000

15000

2000025000

30000

35000

40000

5000

10000

15000

20000

25000

30000

35000

40000

steps

steps

symmetric and asymmetric noise experiments. The convergence rate of BADS aligns with that of nonDPS approaches, whereas CDS converges faster than BADS , and BLO converges significantly slower. In the asymmetric noise experiments, overfitting is less pronounced compared to the symmetric noise experiments, where after 50,000 training steps, the test accuracy for all approaches decreases. BLO and BADS exhibit a notably slower rate of overfitting compared to other methods. Conversely, CDS overfits more quickly than non-DPS approaches, resulting in significantly lower test accuracy even compared to the non-DPS methods.

F.2 Individual Scalar Weights vs . Weight Network

In [41], each training example is associated with a learnable scalar weight. The scalability issues of this method compared to the weight network used in BADS are detailed in Section 2.4. In this section, we examine the CIFAR-10 denoising task to assess performance differences that result from replacing the BADS weight network with the individual weights strategy described in [41]. From the top row of Figure 11, it is evident that the performance of BADS with scalar weight (referred to as

Test/ACCforCIFAR-10(symmetrynoise)

Test/ACCforCIFAR-10(asymmetrynoise)

contrastive

BLO

0.8

mixing

0.7

dnp

0.7

Bayesian

Bayesian_per_sample

0.6

random_select

Test/ACC

Test/ACC

0.6

meta_only

0.5

contrastive

BLO

0.5

mixing

0.4

Bayesian

0.4

Bayesian_per_sample

random_select

0.3

meta_only

0.3

0

2500050000 75000 100000 125000 150000 175000 200000

0

2500050000 75000 100000 125000 150000 175000 200000

steps

steps

Train/ACCforCIFAR-10(symmetrynoise)

Train/ACCforCIFAR-10(asymmetrynoise)

1.0

0.9

0.9

80

0.8

0.7

Train/ACC

0.7

0.6

Train/ACC

0.5

contrastive

0.6

contrastive

BLO

BLO

0.4

dup_meta

mixing

0.5

mixing

dup_meta

Bayesian

0.4

Bayesian

Bayesian_per_sample

Bayesian_per_sample

0.2

random_select

meta_only

random_select

meta_only

0.2

0

2500050000 75000 100000 125000 150000 175000 200000

0

2500050000 75000 100000 125000 150000 175000 200000

steps

steps

Bayesian_per_sample in the legend) slightly surpasses that of the original BADS (labeled as Bayesian in the legend).

Test/BLEUforWebNLG.2domains

44

42

40

Test/BLEU

38

contrastive

BLO

Bayesian

36

34

32

5000

10000

15000

20000

25000

30000

steps

0.8

Weights of DataPoints in MNiST Predicted byBADS

Weights of Data Points in MNISTPredicted by BLO

Weights of DataPoints in MNIST Predicted by CDS

.7

3.5

0.8 -

3.0

0.6

2.5

S°0

0.6

0.4

weights for example with Label=Majority Class -9

weights for example with Label=Majority Class -9

2.0

weights for example with Label=Majority Class -9

0.3

weights for example with Label=Minority Class -4

0.4

weights for example with Label=Minority Class -4

1.5

weights for example with Label=Minority Class -4

1.0

0.2

0.2

0.1

0.0

0°0

00

15250

15500

15750 16000 16250 16500

1675017000

15250 15500

15750 16000 16250 16500 16750 17000

15250

15500

15750 16000 16250 16500 16750 17000

Weights of DataPoints in CIFARPredicted by BADS

Weights of Data Points in CIFAR Predicted by BLO

Weights of DataPoints in CIFARPredicted by CDS

weights for example with Label=Noisy Annotation

0.011

0.06

0.784

weights for example with Label=Clean Annotation

0.010

0.04

0.782

Weights

0.780

0.009

0.02

weights for example with Label=Noisy Annotation

0.778

weights for example with Label=Clean Annotation]

00°0

0.008

0.776

0.02

0.007

0.774

0.04

weights for example with Label=Noisy Annotation

0.772 -

0.006

weights for example with Label=Clean Annotation

20250 20500 20750 21000 21250 21500 21750 22000

20250 20500 20750 21000 21250 21500 21750 22000

20500

21000

21500

22000

22500

23000

Weightsof DataPointsinWebNLGPredicted by BADS

Weights of DataPoints inWebNLGPredicted by BLO

Weights of Data Points in WebNLG Predicted by CDS

weights for example with Label=University

weights for example with Label=MeanOfTransportation

0.0325

weights for example with Label=Company

weights for example with Label=Food

weightsforexample withLabel=Politician

0.0010

weights for example with Label=Airport/

weights for example with Label=writtenwork

00E00

weightsforexample withLabel=Artist

weights forexample with Label=City

60

weights for example with Label=Food

weights for example with Label=Airport

weights for example with Label=City

weights for example with Label=CelestialBody

0.0275

weights forexample with Label=CelestialBody

weightsforexample with Label=Politician

70

weights for example with Label=SportsTeam

weightsforexample withLabel=ComicsCharacter

6000°0

weightsforexamplewithLabel=Company

weights forexample withLabel=SportsTeam

05200

weights forexample withLabel=Airport

weightsforexample withLabel=University

80

weightsforexample with Label=City

weightsforexample with Label=Astronaut

weights for example with Label=Building

weightsforexample with Label=Food

0.0225

weights forexample withLabel=WrittenWork

weightsfor example with Label=Meanoftransportatr

06-

weights forexample with Label=WrittenWork

weights forexample with Label=MeanofTransportation

80000

L0000

weights forexamplewithLabelMonument

weights for example with Label=Athlete

weights for example with Label=ComicsCharacter

weights forexample with Label=Astronaut

weights forexample with Label=Artist

weights for example with Label=Politician

OST00

00200

0.0175

weights for example with Label=SportsTeam

weights for example with Label=Astronaut

weights for example with Label=ComicsCharacter

weights forexample with Label=Athlete

weights for example with Label=Monument

weightsfor example with Label=Building

110

100

weights for example with Label=Artist

weights forexample withLabel=Monument

weights for example with Label=Building

weights for example with Label=Athlete

weights forexample with Label=Company

weights for example with Label=University

weights for example with Label=CelestialBody

120

43000

43500

44000

44500

45000

43000

43500

44000

44500

45000

Steps

WeightsofDataPointsinMNiSTPredictedbyClassAct

WeightsofDataPointsinCIFARPredictedbyClassAct

WeightsofDataPointsinWebNLGPredictedbyClassAct

weights for example with Label=Majority Class -9

weights for example with Label=Minority Class -4

0.0070

weights for example with Label=Clean Annotation

weights for example with Label=Noisy Annotation

0.018

0.006

0.0065

0.005

0.0060

0.016

0.0055

0.014

Weights

0.004

0.0050

0.012

weights for example with Label=Artist Domain

weights for example with Label=City Domain

000

0.0045

0.010

0.002

0.0040

0.008

0.0035

0.001

0.0030

0.006

0

2000

4000

6000

8000

100001200014000

000000000000009000000000000000000000000

10000

20000

30000

40000

Steps

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have included our main claims in the abstract and introduction that accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed several limitations of the current work, together with workarounds and our plans to address them in our future work. Please see the Limitations section right before the References.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have described all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to release code, with sufficient instructions to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results are accompanied by error bars wherever possible.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on the computer resources needed to reproduce the experiments is described in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper presents work whose goal is to advance the field of Machine Learning. At this point we do not see any negative impacts of the current work on society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors cite the original papers that produced the code package or dataset under proper licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for data point selection (DPS) in the setting where only little data from the target (or meta) distribution is available, together with a lot of data from the training distribution. DPS aims to find the weights of data points in the training distribution such that the model parameters that minimise the loss of the therewith weighted training distribution also minimise the loss on the target distribution. The method proposed for DPS in this paper consists in treating the problem in a Bayesian way and doing posterior inference on both (data) weights and model parameters simultaneously, using stochastic gradient Langevin dynamic sampling (SGLD) to approximate the posterior. The crucial trick is to view the data from the target distribution as generated by the model (parameters), which allows for the inference.\nThey show the effectiveness of their approach on 4 experiments and compare against 2 DPS baselines (bilevel optimisation (BLO) and contrastive data selection (CDS)) as well as a range of naive baselines (mixing available training and target distributions to different extents). \n- The first experiment is on data balancing for MNIST, where the class balance differs significantly between training (imbalanced) and target (balanced) distribution. \n- The second experiment is on data denoising for CIFAR, where the training distribution (noisy) differs from the target distribution (clean) in the degree of noisiness\n- The third experiment is on efficient learning on WebNLG, where there are differences in the topic or domain of the data between training distribution (Artist, City) and target distribution (Athlete, Politician, Astronaut)\n- The fourth experiment is on instruction finetuning (IFT) for LLMs, using an OpenLlama3B model. \nThe proposed method compares favourably in all experiments in the main part of the paper. In the appendix results for the second experiment are additionally shown for longer training times, in which case BLO outperforms the proposed method.\n\nSTRENGTHS:\n- The idea of framing and solving DPS in a Bayesian way seems novel and innovative\n- The paper is well written in general and is easy to read and follow\n- The proposed method is getting good empirical results as displayed in the paper, in particular also on the relevant use case of IFT for LLMs\n- Compared to BLO, the proposed method seems to have lower memory and compute requirements\n\nWEAKNESSES:\n- For the most part, the method proposed is explained well and in detail. However, when it comes to a crucial aspect in the implementation  that is subsequently used in all experiments (the weight network), details are brushed under the carpet as a 'straightforwardly modification'. However, for me the causal graph in Figure 1 seems to be different if the weights become a function of $z^t$, let alone $\\theta$ if model embeddings $f_{\\theta}(z^t)$ are used instead of raw data $z^t$ like in the experiments. This should be explained in more detail (see also Question about this below). \n- In the main part of the paper, performance of BLO is portrayed as clearly falling behind the proposed method and other methods in terms of accuracy on the CIFAR experiment (Fig 2, Fig 3, line 266). However in Fig 7 in the Appendix, it can be seen that with longer training, BLO is at least on par (symmetry noise) or even slightly better (asymmetry noise) in terms of accuracy. It would be better if this would already be mentioned in the main part of the paper\n- Assuming a non-negativity constraint on the weights and the loss, in Eq (4) and subsequently Eq (7) the gradient wrt $w$ will be negative for the data dependent term and thus each gradient update is only pointing away from $0$ through the gradient of the prior term $\\log p(w)$ (and the noise term at the end). Therefore, preventing collapse to $0$ weights (as described in line 148) seems to crucially depend on a strong enough prior as well as the impact constant and $N_t$. At least for the not so compute intensive experiments on MNIST and CIFAR, it would be good to see an ablation on the effect of these impact constants and the prior governing sparsity parameter $\\beta$. \n- The limitations are only spelled out after the page limit\n\nQUESTIONS:\n- Which part in the prior formulation or in the optimisation ensures that the weights are non-negative? Or are they intentionally not constrained by the method in that sense?\n- How does the causal graph in Figure 1 change if a weight network that is using embeddings of training data as inputs is being used like in the experiments? How does posterior inference change in this case?\n- Why is the weight network being used in the experiments if in the ablation in the appendix, the element wise weights version of the proposed approach is outperforming it?\n- Why are the inputs to the weight network different for the MNIST experiment (image embeddings) and the CIFAR experiment (image embedding + one-hot encoded label)?\n- When comparing to BLO in terms of compute efficiency, have the authors taken into account proposals of making the Hessian vector product calculations more efficient for BLO (see https://iclr-blogposts.github.io/2024/blog/bench-hvp/ for example)?\n\nLIMITATIONS:\n- Limitations are discussed, but after the page limit", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

comments: SUMMARY:
This work proposes a Bayesian approach to the data point selection task. Specifically, the authors introduce the important weights to each training point and derive the posterior joint probability of the instance-wise weights and network parameters. The parameters and weights are then sampled iteratively based on the stochastic-gradient MCMC technique. Experiments on computer vision and natural language processing have been conducted to show the efficacy of the proposed method.

STRENGTHS:
1. This work views the data point selection problem from the perspective of Bayesian theory and introduces the posterior distributions of the instance-wise weights and network parameters.

2. Experimental results demonstrate the efficacy of the proposed method.

WEAKNESSES:
1. I suggest the authors provide a detailed derivative for Eq. 4.

2. Eq. 5 still requires to compute the normalizing constant in Eq. 4, which serves as the bias of the gradient.

3. The authors claim that “Our method straightforwardly achieves sparsity on the w weights allowing efficient sample selection unlike”. I hope the authors provide more analysis and evidence about the sparsity of instance-wise weight.

QUESTIONS:
Please see the weaknesses

LIMITATIONS:
No potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new Bayesian method for Data Point Selection (DPS), called BADS. DPS aims to select training data points that optimize performance on downstream tasks. Instead of relying on bi-level optimization (BLO), BADS frames DPS as a posterior inference problem. The method uses a weight network to learn the importance of each data point and proposes a novel update rule for jointly updating model parameters and data point weights using Stochastic Gradient Langevin Dynamic sampling. Experiments on MNIST, CIFAR, and WebNLG datasets show that BADS outperforms or matches existing DPS methods, including BLO, in terms of accuracy and efficiency.\n\nSTRENGTHS:\n- Originality: This paper introduces a novel perspective on DPS by using a Bayesian framework instead of the BLO approach. This offers a new way to think about data selection and its connection to posterior inference. The paper also leverages Langevin sampling for optimizing the derived posterior inference which adds to the novelty.\n- Clarity: The paper is well-written and easy to understand. The authors clearly explain their method and the motivations behind their design choices. The final derived method is simple and intuitive.\n\nWEAKNESSES:\n- Limited theoretical analysis: The paper focuses on the empirical performance of BADS and lacks a deep theoretical analysis. Providing more theoretical insights on the convergence and optimality properties of BADS would strengthen the paper, especially since the authors claim that their method, unlike BLO, has theoretical convergence guarantees.\n- Hyperparameter sensitivity: The paper mentions that BADS has several hyperparameters that require careful tuning. A more in-depth discussion on the impact of these hyperparameters and potential strategies for automatic tuning would be beneficial.\n- Comparison and claims:  The paper primarily compares BADS with BLO, but there are other methods that deal with selecting data to learn on, such as offline pruning methods [1], and online and offline batch selection [2, 3], among others. In addition to that, some of these methods have already been used for large scale model training, such as [4, 5].\n\nSome important equations and their corresponding assumptions have been omitted. In particular, equation (4) is not derived and on first glance seems to assume independence of $w$ and $\\mathcal{D}$, but it is not clear if this still holds after the introduction of the weight network. \n\n1. https://arxiv.org/abs/2206.14486\n2. https://arxiv.org/abs/2312.05328\n3. https://arxiv.org/abs/2406.10670\n4. https://arxiv.org/abs/2402.09668\n5. https://arxiv.org/abs/2406.17711\n\nQUESTIONS:\nCan the authors provide any theoretical guarantees on the convergence of BADS? What are the conditions under which BADS finds an optimal solution?\n\nWhat is the impact of different hyperparameters, especially the sparsity level parameter and impact constants? How robust is BADS to hyperparameter variability.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations proposed in their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}