LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Authors: Yang Yu
Kai Han
Hang Zhou
Yehui Tang
Kaiqi Huang
Yunhe Wang
Dacheng Tao
Venue: ICML 2025 poster (2025)
Keywords: LLM
Data Selection
Primary area: deep_learning->large_language_models
TL;DR: A novel bi-level optimization framework with a data weighting model is proposed to improve data utilization during the training of LLMs.

Abstract: While large-scale training data is fundamental for developing capable large language models (LLMs), strategically selecting high-quality data has emerged as a critical approach to enhance training efficiency and reduce computational costs. Current data selection methodologies predominantly rely on static, training-agnostic criteria, failing to account for the dynamic model training and data interactions. In this paper, we propose a new Data Weighting Model (DWM) to adjust the weight of selected data within each batch to achieve a dynamic data utilization during LLM training. Specially, to better capture the dynamic data preference of the trained model, a bi-level optimization framework is implemented to update the weighting model. Our experiments demonstrate that DWM enhances the performance of models trained with randomly-selected data, and the learned weighting model can be transferred to enhance other data selection methods and models of different sizes. Moreover, we further analyze how a model’s data preferences evolve throughout training, providing new insights into the data preference of the model during training.

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Yang Yu 123
Kai Han
Hang Zhou 34
Yehui Tang?
Kaiqi Huang Yunhe Wang Dacheng Tao
Abstract
are essential for enabling the model generalize across various tasks and domains (Bai et al., 2024: Xia et al., 2024; Zhou et al., 2024). As the scale of the pre-training data increases, language models exhibit remarkable capacity to perform downstream tasks with minimal task-specific tuning, showcasing the power of data-driven approaches in natural language processing (Kaplan et al., 2020).
While large-scale training data is fundamen- tal for developing capable large language mod- els (LLMs), strategically selecting high-quality data has emerged as critical approach en- hance training efficiency and reduce computa- tional costs. Current data selection methodologies predominantly rely on static, training-un -agnostic cri- teria, failing to account for the dynamic model training and data interactions. In this paper, we propose a new Data Weighting Model (DWM) to adjust the weight of selected data within each batch to achieve dynamic data utilization dur- ing LLM training. Specially, to better capture the dynamic data preference of the trained model, a bi-level optimization framework is implemented to update the weighting model. Our experiments demonstrate that DWM enhances the performance of models trained with randomly-selecter data, and the learned weighting model can be trans- ferred to enhance other data selection methods and models of different sizes. Moreover, we further analyze how a model's data preferences evolve throughout training, providing new in- sights into the data preference of the model during training.
Though the power of large language models rises with the use of enormous and ever-growing datasets for pre-training, naively training a model on all available data may not be op- timal, as the quality of available data varies, and the training increases the carbon footprint and financial costs. Recently, there is increasing evidence that choosing the right training data is more essential for producing state-of-the-ie large language models (Albalak al., 2024; Zhao al., 2023), and researchers have focused on studying data selection, the mechanism to determine which candidate data to include in the training, to improve the training efficiency of model pre-training. Specially, these data selection methods usually utilize referring data or referring models for an effective selection process. For example, in DSIR (Xie et al., 2023), Wikipedia and books are used as the high-quality data to help data classification. In DsDM (Engstrom et al., 2024) and MATES (Yu et al., 2024), LAMBADA dataset (Paperno et al., 2016) is used as the target dataset to help evaluate the influence of the candidate data, i.e., the decrease of the loss in the target dataset when the model is trained with and without the candidate data. Besides, referring models are also utilized to help evaluate the quality of the data, The relationship between the perplexity of reference models (e.g., Llama) applied to candidate datasets and data quality is examined, with perplexity proposed as a potential met- ric for assessing data quality (Ankner et al., 2024). And QuRating (Wettig et al., 2024) utilizes the data preference of GPT-3.5-turbo to train the data rating model.
1. Introduction
The success of modern language models has demonstrated the critical role that large-scale pre-traine mre-training data plays in shaping their performance (Brown et al., 2020; Touvron et al., 2023a). The diversity and scale of the training data
'School of Artificial Intelligence, University of Chinese Academy of Sciences "The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute Au- tomation, Chinese Academy of Sciences 'Huawei Noah's Ark Lab *College of Intelligence and Computing, Tianjin Uni- versity "Nayyang Technological University. Correspondence to: Kaiqi Huang <kqhuang@nnpp @nipriaa.c.cn.. Yunhe Wang <yunhe. grunhe.wang&ee @ huawei.com>.
Though these methods filter out data and decrease the train- ing cost, they merely focus the data selection isolatedly without considering the dynamic training process of LLMs. On the one hand, most of the previous methods selected data before model training, ignoring the dynamic data pref- erence of the model during training. On the other hand, the data samples in existing methods are selected separately and utilized within batch indiscriminately, without considering the joint effects between different samples. In fact, the data
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
samples interact with each other, and the combination of them determines the model update direction together. Hence, the existing methods which select data separately and ignore the data utilization will limit the potential performance of the trained LLMs with selected data.
model- methods. Our Method also belongs to sample- level data selection.
Sample-Level Data Selection Sample-level data selec- tion methods can be classified into two sub-types: reference and reference model-based approaches. Refer- ring data- bast- methods mainly select data samples with the help of other data. For example, in DSIR (Xie et al., 2023), Wikipedia and books are utilized as high-quality reference data facilitate data classification. In DsDM (Engstrom et al., 2024) and MATES (Yu et al., 2024), the LAMBADA dataset (Paperno et al., 2016) serves as the target dataset to assess the impact of candidate data data-specifically, by mea- suring the reduction in loss on the target dataset when the model is trained with versus without the candidate data, Additionally, a line of research investigates leveraging ref- erence models guide data selection, i.e., referring model- based methods. For instance, the relationship between the perplexity of reference models (e.g.. LLaMA) on candidate datasets and data quality has been analyzed, with perplexity proposed as an effective metric (Ankner et al., 2024). An- other approach, QuRating (Wettig et al., 2024). leverages the implicit data preferences of the referring model GPT- 3.5-turbo to train a model for data rating. However, most data selection methods remain isolated and fail to consider the dynamic training process of the model. While MATE partially accounts for the multi-stage training process of the model, it still overlooks the joint effects of selected data, as the chosen data is used indiscriminately during training. In contrast, our method focuses more on data utilization and proposed DWM to capture the dynamic data preference of the model, and the joint effects of different data samples during model training.
In this paper, to improve the data utilization for large lan- guage models training with selected data, we propose a bi-level optimization framework to capture the dynamic data preference of the model, and the joint effects of differ- ent data samples. In the framework, a plug-and-play Data Weighting Model (DWM) is introduced to weigh the data samples within each batch during model training, and there- fore focuses on the joint effects of selected data. Specifically, to guarantee the weighting model knows the data preference of the trained model, we introduce a bi-level optimization to help learn the weighting model. The lower level will first optimized the trained model with data weighted by the weighting model, and the upper level will optimized the trained model updated by the lower-lever optimization, where the weighting model can be optimized with the help of the chain rule. Furthermore, to better capture the dy- namic data preference of the trained model, we learn the DWM via the above bi-level optimization at different stages during model training, and hence learn the data preference dynamically and adaptively.
We conduct extensive experiments to validate the effective- ness of DWM. First, using randomly-selected data from SlimPajama, we pre-train 370M model from scratch. The model trained with DWM and randomly selected data out- performs both models trained with randomly-selected data and those with carefully selected data. We then transfer DWM to larger LLM (i.e., 1.3B) and other data selection methods, which also achieve a consistent performance im- provement. Finally, we further analyze how the weighting model preferences evolve during training to provide more insights about the model data preference.
3. Approach
3.1. Motivation
The performance of large language models is strongly in- fluenced by the scale and diversity of pre-training data, and data selection mitigates the inefficiency of model pre- training caused by the variations of data quality and data redundancy, Despite these advances, existing methods typ- ically focus on pre- pre-training data selection without consid- ering the nature of model training. Most approaches select data in isolation before training and use data indiscrimi- nately during training, overlooking the interactions between data points and the shifty data preference of the trained model.
2. Related Work
We review existing data selection methods for large lan- guage models and analyze their relevance to our proposed approach.
Categories of Data Selection Methods As discussed in RegMix (Liu et al., 2024), existing data selection methods for LLM pre-training can be categorized into three cate- gories: (I) Token-level selection deals with the filtering of tokens, like Rho- 1(Lin et al., 2024); (2) Group-level selec- tion groups data into pools and focuses on the pool-mixing, like RegMix (Liu et al., 2024): (3) Sample-level selection is about choosing individual training examples. The category of sample-level selection methods can be further divided into two types: referring data-based methods and referring
In this paper, to better utilize selected data for model pre- training, and capture the nature of model training, we pro- pose a novel plug-and-play Data Weighting Model (DWM) to assist model training. DWM will be used to weigh the data in a batch, and hence adapt to the joint interaction
2

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Weighring Model
Data
LLM Model
Train Loss
LLM model training
L.M model training
LLM model training
LLM model training
S
S
S
weighting model
weighting model updiale
weighting model update
update
Weighting Model
Back Propagation
Data
Train Loss
LLM
Validation Loss
Eq. 1
Figure 1. The framework of the proposed bi-level optimization process with DWM, where the LLM model and the weighting model are trained alternated. During model training, the weighting model frozen and the pre-training loss is the weighted of the loss of each data sample in one batch. Besides, to capture the data preference of the trained model, the weighting model is updated to minimize the validation loss of the model trained with the weighting model with the chain rule.
between data during training. And in order to capture the dynamic data preference of the trained model, bi-level optimization framework introduced to help update the weighting model by stages. The framework of DWM is shown Figure 1, where the model training and the weight- ing model update alternated by stages. In the following, will present the bi-level optimization process and the details of the update of the trained LLM model and the weighting model.
does not contribute to the performance improvement of the trained model.
To achieve a better optimization the weighting model and help the training of the trained model, in this work we propose novel weighting model evaluation metric called weight influence, which is defined as:
The performance of the trained LLM model in the validation dataset when optimized with the weighting model.
3.2. Dynamic Bi-level Optimization
To jointly optimize both the LLM and the importance of the training data (i.e., DWM), while accounting for the dynamic nature of L.M training, we propose a dynamic bi-level optimization framework for iterative improvement. This framework involves updating both the trained model and the weighting model capture the evolving data prefer- ences. The weighting model learns the data preferences of the current LLM based on the interactions within single batch and provides better update directions by assigning different weights to the data samples. The challenge lies in effectively optimizing this weighting model. A straightfor- ward approach is to evaluate the weighting model using the standard auto-regressive loss the training data. However, since the pre- pre-training loss is also dependent the output of the weighting model, updating the weighting model with this loss may lead to sub-optimal colutions-suce as set- ting the weight of all data or easily predicted data points to zero in an attempt to minimize training loss. This, however,
Then we can optimize the weighting model by maximizing its weight influence. Concretely, the process of jointly con- sidering the trained model 0 and the weighting model O., for updating can be viewed bi-level optimization problem:
max Hh
Rvai (0w:)
(1)
s.t. 0 ( arg min Lusin (8, Ow),
where the first term is to optimize 6n by maximising the reward performance Real (e.g., accuracy) on validation set, while the second term is the LLM training based on training data re-weighted by the data weight model. And therefore. when optimizing the upper level object, i.e., the performance of the trained L.M model in the validation set, the weighting model can be optimized with the help of the chain rule. Via this bi-level optimization, the weighting model learns how to weigh sampled data to improve the generalization of the trained model, which is of benefit to the model pre-training.
3

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
In the following, we'll describe the details of the bi-level optimization of each part.
the updated trained model will be used to calculate the validation performance:
M
3.3. LLM Training
Reot( Σ
Ri
In DWM, model pre-training is similar to the normal pre- training process except for the utilization of the weighting model, where data in one batch will be weighted instead of used indiscriminately. The contribution weight of each sample X; to the training of LLM can be obtained by the introduced DWM:
(5)
M R
bs
Σ
Σ w/ VO,).
where M is the number of data samples in the validation set.
Wi = Ox,, (na)is
(2)
Consequently, the weighting model can be updated by opti- mizing the validation performance of the trained model in chained:
where i 2. bs is the sample index and bs is the total number of data samples in one batch. Here DWM will consider the data interaction in one batch to weigh the contribution of each sample.
ORval
Be (0*)
00
00
00
26..
M 0R iRwat,
bs Owj V02)
Here we use the output of the weighting model 0₂2 to weigh the training loss of each data sample, and therefore, the loss during pre-training stage is:
(-a
00
80w
(6)
M
bs Owi v0
bs
a Σ
Lerain (0, Ow) Σ wiLtroin,
00
D0w
Hence the weighting model can be optimized to improve the performance of the trained model. The LLM is kept fixed while updating DWM to prevent any potential leakage of validation data.
The training loss Lerain, i is the normal nuto-regressive loss for LLM training. It is important to note that DWM remains fixed during LLM training for improving data utilization and ensuring stable learning dynamics, allowing the model to fo- cus on optimizing the LLM parameters without interference from fluctuating data weights.
3.5. Multi-stage Alternative Iteration
As discussed above, we update the trained model or the weighting model and keep the other fixed to ensure the stability of training or avoid the knowledge leakage. In order to capture the dynamic data preference of the trained model, here we employ multi-stage alternative iteration process to jointly optimize the trained model and the weighting model. In this iteration process, the trained model parameters @ and weighting model parameters O.r are updated in a stage-wise manner. Concretely, starting from parameters gi-1 and 0t-1 inherited from stage 1, the iteration proceeds at stage t as follows:
3.4. Data Weighting Model Training
As discussed above, the weighting model is optimized to maximize the trained model performance in the validation dataset. Similar the motivation behind meta-learning, the goal of learning the weighting model is to capture more effective data weighting mechanism, thereby improving the generalization of the trained model in later stages of training.
Referring to Eq. 1, it is worth noting that the trained model updated with the output of the weighting model, and the updated model is evaluated in the validation set. There- fore we could optimize the weighting model directly by the trained model performance in the validation dataset with the chain rule. Concretely, we will first record the updated trained model parameters explicitly, which is:
1. Weighting Model Update. Fixing the trained model, we first update 0.p referring to Eq. 6:
0 = 0/- 1) Reat (0--1,- (0')):
(7)
bs 8" = 0 a
2. Trained Model Update. With the updated weighting model 6 we then optimize the trained model by mini- mizing the training loss referring to Eq. 3 in this stage:
W V0,
(4)
bs
aLerain. (0)
= 0
Of = arg min Lerain (0, o..).
(8)
00
where a is the learning rate of the model and V8, is the gradient by training the model with data sample X,. Then
Each stage E {1,2, T} strictly enforces an alternating update order to resolve the interdependence between 0 and
4

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Ow This iterative process ensures that both models mutu- ally enhance each other's performance through progressive refinement.
and is orthogonal to existing data selection methods. In this paper, verify the effect of DWM, first compare DWM using random-selected data to naive random selection, as well as state-of-the-arr data selection baselines including DSIR (Xie et al., 2023) and QuRating (Wettig et al., 2024). Here DSIR uses Wikipedia and books as target data dis- tribution and selects data classified in the target distribu- tion. QuRating first uses the model GPT-3.5- (OpenAI, 2023) as the referring model to judge the data samples in dif- ferent dimensions, then uses the model's preference data to train data quality rating models to select data. Moreaver, we also provide results of DWM transferred to the data selected by DSIR or QuRating to justify its scalability.
In summary, the proposed method effectively optimizes data utilization during training by dynamically adjusting data weights through DWM. By capturing and leveraging the model's evolving data preferences, DWM improves both the training effectiveness and generalization of the model. This approach allows for better utilization of selected data, leading to enhanced performance even with randomly se- lected data. Additionally, the trained weighting model can be transferred to larger models or applied to other data se- lection strategies, making a robust solution for optimizing L.M training across various scenarios.
Evaluation Benchmarks To evaluate the effectiveness of DWM, we compared the performance of the pre-trained models trained with DWM and different data selection baselines on the widely-used Im-evaluation -haitattioonhaeessss harness frame- work Following Wettig et al. (2024) and Xie et al. (2023), we perform a holistic evaluation of pre-trained mod- els across 9 downstream tasks (ARC-easy/challlnge (Clark et al., 2018), SciQA (Auer et al., 2023), LogiQA (Liu et al., 2020), BoolQ (Clark et al., 2019), OBQA (Mihaylov et al., 2018), HellaSwag (Zellers al., 2019), PIQA (Bisk et al., 2020). and WinoGrande (Sakaguchi et al., 2021)), compris- ing reading comprehension tasks, commonsense reasoning and knowledge question answering. To assess both inherent knowledge and in-context learning capabilities, we evaluate the models under zero-shot and two-shot settings referring to MATES (Yu et al., 2024). For each evaluation task, nor- malized accuracy is reported when available; otherwise, standard accuracy is adopted as the evaluation metric.
4. Experiment Setup
Implementation Details To evaluate different data selec- tion methods, we utilize the training data selected from the popular dataset SlimPajama (Soboleva et al., 2023), which is the largest multi-corpora, open-source dataset for train- ing large language models. SlimPajama is a cleaned and deduplicated version of the RedPajama dataset (Weber et al., 2024), includes 627 billion tokens and provides the meta- data about the domain information for each sample. To verify the performance of different data selection methods for model pre-training, we adopt the model architecture from Llama-2 (Touvron et al., 2023b) at the scale of 370 million and 1.3 billion parameters separately to do the model pre-training. Following the principles of the scaling law and the QuRating (Wettig et al., 2024) framework, total of 30 billion tokens are selected by different methods to train the model. As for the weighting model, we adopt the architec- ture from Llama-2 with 370 million parameters to learn the sample embedding, with additionally one attention block and two linear layers to generate the data weights consid- ering the combination of the data in batch. We set the sequence length to 1024 and the batch size to 4096, with micro batch size (bs) as 8 to balance the effectiveness of the weighting model and the GPU memory constraints. Follow- ing DSDM and MATES, we adopt LAMBADA (Paperno et al., 2016) as the validation set, which is a widely-used language modeling task and often serves as a validation task for language model pre-training, and split the training process into 5 stages learn the dynamic data preference. At the first stage, the trained model learns from scratch and weighs sampled data uniformly, and in the second stage, the weighting model will be partially initialized from the previous trained model and be continued trained in the later stage. More details can be found in Appendix A.
100004
140
Siget
Eayo
Staget
20ge
Dayet
Day
kxpo2
190070
Raps
(a) Zero-shot
(b) Two-shot
Figure 2. Multi-stage performance of the 370M model using randomly-selectee selected data with and without DWM
5. Evaluation Results
In this section, we want to evaluate the effect of the proposed DWM. We conduct experiments to answer the following questions: 1) Is it necessary to consider the data utilization in model pre-training with data selection? 2) What is the transferring performance of the DWM to larger models or other selected datasets? 3) How does the data preference change of the trained model during training? 4) What is
Baselines It is worth noting that our method DWM aims to improve the training data utilization of model pre-training,
httppa:/ / /tt:/gitttt /ithbb. com/ /ln/Elewe EethhrrA mm eva 1 pat on oo-ooaneee harness
5

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Table 1. Zero-shot performance of 370M pre-trained models using random-selected data with and without DWM
STAGES
ARC-C
ARC-E
BooLQ
H.S.
LOQIQA
OBQA
PIQA
ScIQ
W.G.
AVERAGE
w/o
22.0
39.3
53.2
33.2
25.2
28.8
63.8
65.2
51.1
42.4
STAGE 2
W/
23.2
40.4
55.6
33.2
26.1
28.2
62.5
63.1
52.4
42.7
W/O
23.8
41.0
58.1
35.0
26.4
27.2
64.4
64.7
51.5
43.6
STAGE 3
W/
22.5
41.6
50.3
34.5
26.3
30.2
64.4
66.9
52.3
43.2
W/O
24.0
40.7
52.9
36.1
25.8
27.6
64.6
69.7
49.3
43.4
STAGE 4
W/
22.8
41.9
58.4
35.8
25.4
30.0
65.9
66.5
52.6
44.4
W/O
24.1
41.2
52.7
36.8
26.6
28.0
65.2
70.9
50.8
44.0
STAGE 5
W 24.3
42.5
59.9
36.4
26.4
29.8
65.3
68.1
52.7
45.0
Table 2.
Two-shot performance of 370M pre-trained models using random-selected data with and without DWM
STAGES
ARC-C
ARC-E
BOOlQ
H.S.
LoQIQA
OBQA
PIQA
ScIQ
W.G.
AVERAGE
W/o
22.9
41.5
48.3
33.0
26.6
27.2
63
75.9
50.9
43.3
STAGE 2
W/
22.9
41.9
55.0
32.9
25.2
25.4
63.4
73.1
51.8
43.5
W/o
24.8
44.0
41.8
34.9
25.8
28.2
64.3
76.2
51.7
43.5
STAGE 3
W/
23.8
44.4
49.3
34.9
24.7
28.4
63.8
78.3
52.2
44.4
W/O
24.1
45.3
53.7
35.9
22.3
28.2
64.6
76.4
50.8
44.6
STAGE 4
W/
23.3
45.4
53.9
35.9
24.4
28.0
64.3
80.6
51.8
45.3
W/O
25.5
46.6
51.6
36.6
22.9
28.4
65.0
78.9
50.8
45.1
STAGE 5
24.7
46.8
56.6
36.5
25.8
28.2
65.0
80.5
53.4
46.4
affecting the effect of DWM? In the following, we will
which mainly helps improve the model's 's text understanding and prediction ability and is thus more suitable for such in-context learning settings.
answer the questions above one by one.
5.1. Effectiveness of DWM
5.2. Transferability of DWM
To evaluate the impact of data utilization in addition to data selection, we compare the performance of the 370M model with and without the weighting model, using randomly se- lected data from the SlimPajama dataset. Since we split the whole training process into 5 stages, here we provide the performance of these approaches in the end of each stage.
Note that in Sec. 5.1, we evaluate the effect of DWM in 370M model and random-selected data. However, we want to emphasize that DWM is agnostic to the model size or data selection methods, operating in an orthogonal manner with respect these factors. Hence, in the following, to justify the generalization ability of the weighting model, and we transfer the weighting model trained before based on the randomly-selected data and 370M models to other data selection methods or larger models. Specially, here we apply DWM to the data selected by DSIR and QuRating, and transfer DWM to a larger Llama2-style model with 1.3B parameters. Note that the model with 1.3B parameters is commonly studied to verify the effect of different data selection methods (Wettig et al., 2024; Yu et al., 2024; Xie et al., 2023).
aim
Results are shown in Table 1, 2 and Figure 2, where the zero- shot and two-shot performance in the downstream tasks mentioned above are provided. Results demonstrate that the model trained with DWM (RANDOM+DWM) is better than that trained without weighting model but only randomly- selected data (RANDOM), illustrating the importance of considering data utilization. Specially, in Figure we found that the weighting model generally has a more pronounced effect on model performance in the later stages of training. In contrast, during the early stages training, the difference performance between using and not using the weighting model relatively minor. Additionally, the DWM not only enhances the model's performance on reading comprehen- sion tasks, such as SciQA and BoolQ, but also improves its performance on commonsense reasoning tasks, such as the WinoGrande. Moreover, we found that compared to the model's direct generalization capability (zero-shot per- formance), DWM primarily enhances the model's few-shot ability. Its performance under the few-shot testing setting consistently outperforms models trained without the weight- ing model. We speculate that this may be because the vali- dation set used to train the weighting model is LAMBADA,
Firstly, we provide the zero-shot and two-shot performance of DWM when transferred to different data-selection meth- ods, and results are shown in Table 8 and Table Results demonstrate that the DWM achieves consistent performance improvements, whether applied to randomly selected data or carefully curated data like DSIR.
Moreover, to investigate the effectiveness of DWM in larger models, we transfer the weighting model trained in the 370M model larger model with 1.3B parameters di- rectly. We provide the comparison Table and where the zero-shot and two-shot ability of the model trained with DWM and different data-sclection methods provided. To
6

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Table 3. Two-Shot performance of 370M pre-trained models using different selected data with and without DWM
METHOD
ARC-C
ARC-E
BoolQ
H.S.
LOQIQA
OBQA
PIQA
ScIQ
W.G.
AVG
RANDOM
25.5
46.6
51.6
36.6
22.9
28.4
65.0
78.9
50.8
45.1
RANDOM+DWM
24.7
46.8
56.6
36.5
25.8
28.2
65.0
80.5
53.4
46.4
DSIR
23.6
45.7
58.6
35.9
24.9
26.4
65.2
74.9
52.3
45.3
DSIR+DWM
24.9
46.3
60.0
36.0
25.8
29.2
65.3
78.4
51.5
46.4
QURATING
27.9
56.6
58.6
38.1
25.0
32.0
63.6
82.3
52.5
48.5
QURATING+DWM 28.1
55.6
59.7
37.7
24.1
31.2
63.3
84.6
53.1
48.6
Table 4. Two-Shot performance of 1.3B pre-trained models using different selected data with and without DWM. Unless otherwise specified, the data size is 30B tokens.
METHOD
ARC-C
ARC-E
BoOl
H.S.
LoQIQA
OBQA
PIQA
SciQ
W.G.
AVG
RANDOM660
28.7
55.9
58.9
48.7
23.7
30.8
70.8
89.9
54.9
51.4
RANDOM
25.1
48.9
56.0
40.7
26.6
28.0
67.3
81.4
54.2
47.6
RANDOM+DWM
25.1
53.3
51.1
44.8
25.7
30.8
68.7
85.7
53.0
48.7
DSIR
27.7
53.6
49.7
44.1
24.6
31.4
68.8
85.5
52.8
48.7
DSIR+DWM
28.2
54.3
51.0
43.3
26.7
30.6
67.4
82.9
54.1
48.7
QURATING 33.3
60.8
61.7
39.3
25.4
32.6
61.9
86.9
50.7
50.3
QURATING-DWM
32.0
62.2
54.5
43.5
27.7
32.4
65.9
88.0
53.0
51.0
verify the method's improvement in model pre-training effi- ciency, we also provide the performance of models trained on the dataset that is twice the size (60B tokens), i.e., RAN- DOM_60B. Firstly, results justify that DWM trained in small models can be transferred the larger model directly, which helps improve the pre-training efficiency effectively. More- over, we also present results for the 1.3B model trained on data selected by SOTA baselines, such as DSIR and QuRat- ing, both with and without the DWM module. Results illustrate that the improvement DWM when applied to different kinds of selected data is consistent across different model scales.
reasoning examples. As a result, its direct application to larger models provides limited performance improvements compared to randomly selected data, leaving less room for DWM to further optimize the data selected by DSIR in the context of larger models.
It is worth noting that in DWM, bi-level optimization strategy is employed on the 370M model to separately train the weighting model and the language model. Once the training completed, the learned weighting model can be directly transferred to larger models without additional training. Besides, using trained data weighting model for model training does introduce additional computational overhead. Referring to (Hoffmann et al., 2022), the training cost in FLOPs can be approximated as:
Specially, we found that, in 370M models shown in Tale 3, the improvement of DWM when applied to QuRating is minor, while applied to DSIR is more significant. However, in the 1.3B model shown in Table 4, the opposite trend is observed. And the marginal average improvements these transfer settings result in mixed task-wise outcomes. We sus- pect this is due to the incompatibility between the model and the high-quality reasoning data during training. Here, we use data selected by QuRating along the educational dimen- sion, which includes a significant amount of high-quality step-by-step reasoning data. For smaller models, such high- quality data can quickly lead to training saturation, leaving limited room for further performance gains through DWM. In contrast, larger models have greater capacity absorb high-quality data, enabling DWM to further exploit the po- tential advantages of such data for model training when applied on QuRating's high-quality selections. Similarly, since DSIR selects data based on its similarity to Wikipedia and books, the chosen data may contain fewer high-quality
Training FL.OPs R C' x Model Parameters x Token Count,
(9)
where the constant C' depends on whether back propagation
is performed. In our case, since the weighting model only performs forward inference when assisting the training of larger models, C can be approximated 2 (compared to 6 for full back propagation). Therefore, when transferring the 370M weighting model to the 1.3B model, the additional training overhead is roughly 9%, and this overhead contin- ues to decrease as the size of the target model increases.
5.3. Analysis of Model Dynamic Data Preference
In normal large language models training, the training data are often pro-collected and treated equally important, with little attention given to selectively emphasizing or de- emphasizing specific data based on value for improving
7

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Table 5. Comparison of using different validation tasks in DWM. DWLL means using the held-out validation dataset.
SBTTING
METHON
READING
REASONING
QA
AVG
RANDOM
43.1
50,9
28.0
44.0
ZERO-SHOT
R+DWM. VAL
43.7
51.0
29.5
44,6
Stage1 1
Stage2
Stage
Stage4
Stage
R+DWM
44.2
51.5
29.8
45.0
RANDOM
45.1
50.8
28.4
45.1
TWO-SHOT
R+DWM.VAL
46.8
50.8
28.2
46.1
R+DWM 46.9
51.6
28.2
46.4
indeed closely tied to the model performance. And in this work, we primarily aim to highlight the importance of lever- aging data during model training, as well as the effectiveness of transferring the trained weighting model. How to better obtain weighting model that consistently improves perfor- mance throughout the entire training stages remains topic for future research. The analysis of the preferred data of the weighting model across different data domains is provided in Figure 5 in Appendix B.2.
Figure 3. Preferred (red) and unpreferred (blue) data of the weight- ing model in different training stages, considering properties of writing, expertise, facts and educational values.
generalization, reducing bias, or enhancing specific capabil- ities. On the contrary, DWM aims to capture the dynamic data preference during model training, which focuses on the interaction of data as well as the shifty data preference of model during training. Therefore, in this portion, we want to analyze the change of the model's data preference, and provide more insights in the role of different kinds of data during model training.
Here we analyze the preferred and unpreferred data of DWM during different stages of training, and provide the results in Figure 3 and Figure 5. Specially, to investigate the proper- ties of data preferred by the weighting model, we utilize the open-source scoring model from QuRating to analyze the scores of preferred (and non-preferred) data across different dimensions. Concretely, we choose a batch size of 4096 data samples, and use weighting model obtained by DWM in different training stages to weigh these data, and split the data into two sections, preferred data and unpreferred data according to their weights. Then we evaluate the pros- perity of these two kinds of data, including their writing style, required expertise, facts and trivia included, as well as educational values. Results in Figure 3 demonstrate that after initial training stages like stage 1, the weighting model prefers favor data that performs well across all these four dimensions (preferred data consistently scores positively in these dimensions), and data requiring expertise or possess- ing educational value is not prioritized as highly. However, counter-intuiee connter-ntuutivelly during the later stages of training, data with a better writing style is less preferred and data required expertise are more preferred, which provides new insights for data selection strategies.
7501 Srace
(a) Zero-shot
(b) Two-shot
Figure 4. Comparison of static and dynamic weighting model. DWM_wI means only using the weighting model of the first stage. DWM_w4 means using the weighting model in the last stage.
5.4. Ablation Study
Note that DWM emphasizes to learn the data preference dynamically, and uses a bi-level optimization framework to help the weighting model training. In this portion, we will evaluate the importance of these two factors. Firstly, since DWM Icarns data preference by multi-stage alter- native iterations, we compare DWM with models trained with fixed weighting model. Here we present the result of models trained with only the weighting model after the first stage (RANDOM+DWM.w)) or in the last stage (RANDOM+DWM w4). Results are shown in Figure 4. Referring to Figure 3, in the early training stages (RAN- DOM+DWM I)), the weighting model distributes its atten- tion more evenly across different kinds of data, which limits the model's potential in later stages. In contrast, the weight- ing model in the last stages (RANDOM+DWM. 44) focuses more high-quality reasoning or expertise data. While this approach significantly boosts performance in the early stages of training, its bias toward high-quality data reduces the exploration of diverse data, ultimately leading to a per- formance gap compared to methods that dynamically learn data preferences throughout training (R (RANDOM+DWM),
Specially, we also note that the weighting model after the stage 2 is inconsistent with those of the weighting model in subsequent stages, which also leads to a performance drop compared to uniformly utilizing randomly-selected data (as shown in the left corner of the Figure 3). This demonstrates that data preferences during training are
8

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Table 6. Two-Shot performance of 370M pre-trained models with different numbers of training stages.
METHOD
ARC-C
ARC-E
BoolQ
H.S.
LOQIQA
OBQA
PIQA
SCIQ
W.G.
AVG
RANDOM
25.5
46.6
51.6
36.6
22.9
28.4
65.0
78.9
50.8
45.1
DWM.2_STACE 2_STAGES
24.5
43.9
57.5
35.1
25.0
27.6
64.7
76.5
52.9
45.3
DWM
24.7
46.8
56.6
36.5
25.8
28.2
65.0
80.5
53.4
46.4
DWM 8 STAGES
25.5
46.3
60.1
36.4
25.2
28.8
64.9
77.2
53.7
46.5
More details are shown in Table 10 and 11 in the Appendix.
Acknowledgements
Moreover, considering that DWM uses the bi-level opti- mization framework and relates the weighting model's opti- mization to the performance in the validation set, here we justify the rationality of this objective. Here we replace the validation task LAMBADA used in DWM with the valida- tion dataset held-out in SlimPajama (DWM VAL) to justify the importance of the choice of the validation task. The performance of the ablation is shown in Table 5. Results demonstrate that using a validation set independent and identically distributed (i.i.d.) with the training data is less effective than using LAMBADA, task designed to encour- age text understanding and prediction. This highlights the importance of validation task selection.
This work is supported in part by the National Science and Technology Major Project (Grant No.2022ZD01 16403), and the Key Research and Development Program of China (Grant No.2023YFED (E003F60001600)
Impact Statement
This paper presents research aimed at advancing the devel- opment of large language models. The proposed method is designed to facilitate data selection for large language model pre-training, thereby enhancing training efficiency and reducing computational costs. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
In DWM, the training process is divided into five stages. To validate this design choice, we conduct ablation studies on the number of stages. Table 14 and Table 6 report the zero- shot and two-shot performance under different stage settings. Increasing the number of stages helps DWM better capture the model's dynamic data preferences, but also introduces additional training overhead for the weighting model. The results suggest that using five stages strikes a favorable balance between performance and efficiency, implying that the model's preferences remain relatively stable within each stage.
References
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L.,, Jeong, H., et al. A survey on data selection for language models. arXiv preprint arXiv:2402.1 16827, 2024.
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-baset pruning with small reference models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
6. Conclusion
In this paper, we proposed a novel bi-level optimization framework with a data weighting model, designed to im- prove data utilization during the training of LLMs. By dynamically adjusting the weight of selected data within each batch, DWM enables more effective data usage and enhances model performance. Our experimental results demonstrate that DWM not only improves the performance of models trained with carefully selected data but also en- ables models trained with randomly selected data to achieve competitive results. Additionally, we show that transferring DWM to larger models yields consistent performance im- provements, and provide insights into how model's data preferences evolve throughout training. This work opens new avenues for optimizing data selection and utilization in LLM training, providing a promising direction for more efficient and cost-effective model training.
Auer, S.,, Barone, D. A., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., et al. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, 2023.
Bai, T., Yang, L., Wong, Z. H., Peng. J., Zhuang, X., Zhang, C., Wu, L.., Qiu, J., Zhang, W., Yuan, B., et al. Multi-agent collaborative data selection for efficient I1m pretraining. arXiv preprini
2024.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432- -439,749,,9 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.
9

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Advances in neural information processing systems, 33: 1877-1901, 2020.
Fernández, R.
The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers), pp. 1525- 1534, 2016.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1eXiv:100..1000 2019.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Clark, P., Cowhey, I., Etzioni, O., Khot, T.. Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.v:8800.05555 05045,, 2018.
Soboleva, D., Al-Khateeb, F,, Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama, 2023. URL jama- - datasetss/eeeerrsss//
Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model- aware dataset selection with datamodels, 2024. URL hutps://axxiv. urg/abs/2401.1299 2024.
.c/
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E.,, Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:200.11555, 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.0928: 2023b.
Langley, P. Crafting papers on machine learning. Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (000L pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Weber, M., Fu, D. Y., Anthony, Q. G., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, open dataset for training large
V., et al. Redpajama: language models. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Bench- marks Track, 2024.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., Chen, W., al. Not all tokens are what you need for pretraining. Advances in Neural Information Processing Systems, 37:29029-29066 2024.
Wettig, A., Gupta, A., Malik, S., and Chen, D. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739 2024.
Liu, J., Cui, I.,, Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124 2020.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Confer- ence Machine Learning, pp. 54104-54132, 2024.
Liu, Q., Zheng, X.. Muennighoff, N., Zeng, G., Dou. L., Pang, T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training. arXiv preprint -xXx:24477004999 2024.
Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 17X1v:1809.02789, 2018.
Yu, Z., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence mod- els. arXiv preprint arXiv:2406.066 06046, 2024.
OpenAl. Chatgpt, 2023. chatgpt.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv 1905.0783, 2019.
Paperno, D.,. Kruszewski, G., Lazaridou, A., Pham, N.-Q.,,
Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and
10

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Zhao, W. X., Zhou, K., J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong. Z., al. survey of large language models. arXiv preprint arXiv: 18223, 2023.
Zhou, H., Tang, Y., Qin, H., Yang. Y., Jin, R., Xiong, D.,
Han, K., and Wang. Y. Star-agents: Automatic data optimization with Ilm agents for instruction tuning. Ad- vances in Neural Information Processing Systems, 37: 4575-4597, 2024.

---

LLM Data Selection and Utilization via Dynamic Bi-level Optimization
A. Training Details
The architecture details of the pre-training models with 370M and 1.3B parameters are presented in Table 7.
Table Architecture pre-training models with 370M 1.3B parameters.
Hypeeraarameter
1.3B Model Value
370M Model Value
Vocabulary Size
32,000
32,000
Hidden Size
2048
1024
FFN Hidden Size
5504
2812
Number of Layers
24
24
Number Attention Heads
Number Attention Heads
Maximum Context Window Length
1024
1024
Number of Parameters
1,345,423,360 (1.3B)
373,867,520 (370M)
In the training process, global batch size of 4 million tokens was utilized. The training was completed in approximately 7500 steps. The learning rate was set at 10 10-4. The Adam optimizer was used, with the hyperparameters configured as 31 = 0.9. B2 = 0.95, 6 10-8
Table Zero-Shot performance of 370M pre-trained models using selected data with and without DWM
METHOD
ARC-C
ARC-E
BooLQ
H.S.
LOQIQA
OBQA
PIQA
SCIQ
W.G.
AVERAGE
RANDOM
24.1
41.2
52.7
36.8
26.6
28.0
65.2
70.9
50.8
44.0
RANDOM+DWM
24.3
42.5
59.9
36.4
26.4
29.8
65.3
68.1
52.7
45.0
DSIR
24.2
40.4
60.2
35.6
26.7
29.0
65.9
51.5
DSTR+DWM
23.2
41.2
61.9
35.8
26.6
29.0
67.5
52.0
44.7
QURATING
26.0
49.3
61.4
38.3
26.6
32.0
63.6
70.5
51.9
46.6
QURATING+DWM
26.2
50.8
57.7
37.5
27.3
33.6
63.7
72.7
52.2
46.9
B. A Additional Results
B.I. Transferability of DWM
We provide the zero-shot performance of the trained model with different kinds of selected data and different scales of parameters. Results are shown in Table and Table 9, where DWM demonstrates consistent performance improvements across different model sizes and various data selection methods.
B.2. Analysis the Domain of Preferred Data
We provide the domain of the preferred data of the weighting model in different training stages in Figure 5, where our code data, such as
weighting model also exhibits stronger focus on expertise data, such as arXiv, and reasoning-intense StackExchange, during the later stages of training. Due the significant variance in data quality within domains and the relatively weak correlation between domain and data quality, more detailed domain analysis will be left for future research.
B.3. A blation Study
Detailed experimental results of different ablations are provided below. Table 10 and 11 show the comparison of static and dynamic weighting model learning. Table 12 and 13 show the detailed performance of the weighting model with different validation tasks. Table 14 shows the zero-shot performance of 370M model trained with different numbers of training stages.
12

---

LLM Data Selection and Utilization via Dynamic Bi-Ievel Optimization
Table 9. Zero-Shot performance of 1.3B pre-trained models using random- mendom-selectdd data with and without DWM
METHOD
ARC-C
ARC-E
BooLQ
H.S.
LOQIQA
OBQA
PIQA
SciQ
W.G.
AVERAGE
RANDOM
25.3
44.8
59.5
40.6
27.0
30.2
66.9
69.3
52.5
46.2
RANDOM+DWM
25.1
46.6
59.9
44.8
26.9
31.6
68.3
73.0
53.4
47.7
DSIR
25.9
47.0
56.3
44.2
25.8
30.4
68.4
70.9
52.5
46.8
DWI
25.6
45.5
59.1
43.4
26.7
31.0
68.6
68.7
52.9
46.8
QURATING
30.0
55.6
61.1
39.4
25.8
33.8
62.0
76.3
50.9
48.3
QURATING+DWM
30.1
55.9
59.2
43.6
27.3
34.2
66.1
77.1
53.4
49.7
Preffredd Lire
Stage
Stage 2
Stage 3
Stage 4
Stage 5
arntia Dera
Data
42
MACONICOW WCM
Stape3
Stape2
Thining
Stuge3
Staget
Stage5
CorwiCae Otiva Sandbe
AWh
80
Creper
Figure Preferred (red) and unpreferred (blue) weighting model different training stages, considering properties of different data domains.
Table 10. Detailed zero-shot performance of trained models using static and dynamic weighting model
STAGE
METHOD
ARC-C
ARC-E
BooLQ
H.S..
LOQIQA
OBQA
PIQA
ScIQ
W.G.
AVG
RANDOM
22.0
39.3
53.2
33.2
25.2
28.8
63.8
65.2
51.1
42.4
R+DWM.W
23.2
40.4
55.6
33.2
26.1
28.2
62.5
63.1
52.4
42.7
STAGE 2
R+DWM w4
23.0
40.2
33.1
26.9
27.8
64.0
63.3
51.7
43.5
R+DWM
23.2
40.4
55.6
33.2
26.1
28.2
62.5
63.1
52.4
42.7
RANDOM
23.8
41.0
58.1
35.0
27.2
64.4
64.7
51.5
43.6
R+DWM w1
23.0
41.2
57.2
35.3
24.6
30.4
65.0
66.6
53.5
44.1
3
R+DWM W4
23.1
40.2
58.7
34.8
27.3
29.2
64.1
64.4
52.0
43.8
R+DWM
22.5
41.6
50.3
34.5
26.3
30.2
64.4
66.9
52.3
43.2
RANDOM
24.0
40.7
52.9
36.1
25.8
27.6
64.6
69.7
49.3
43.4
R+DWM w1
22.1
41.5
55.7
36.1
23.7
29.8
65.2
66.3
52.4
43.6
STAGE 4
R+DWM.W4
23.1
40.6
35.8
25.5
29.0
64.6
64.0
53.1
44.1
R+DWM
22.8
41.9
58.4
35.8
25.4
30.0
65.9
66.5
52.6
44.4
RANDOM
24.1
41.2
52.7
36.8
28.0
65.2
70.9
50.8
44.0
R+DWM
22.7
41.5
56.0
36.8
25.0
30.6
65.0
67.7
52.8
44.2
5
R+DWM.W4
23.5
41.3
61.0
36.5
25.5
29.6
65.0
66.8
51.3
44.5
R+DWM
24.3
42.5
59.9
36.4
26.4
29.8
65.3
68.1
52.7
45.0
13

---

LLM Data Selection and Utilization via Dynamic Bi-Ievel Optimization
Table 11. Detailed two-shol performance trained models using static and dynamic weighting model
STAGE
METHOD
ARC-C
ARC-E
BooLQ
H.S.
LOQIQA
OBQA
PIQA
SCIQ
W.G.
AVG
RANDOM
22.9
41.5
33.0
27.2
63.0
75.9
50.9
43.3
R+DWM w1
22.9
41.9
55.0
32.9
25.2
25.4
63.4
73.1
51.8
43.5
2TAGE
R+DWM w4
23.3
42.6
56.9
32.9
26.4
26.0
63.8
73.4
51.3
44.1
R+DWM
22.9
41.9
55.0
32.9
25.2
25.4
63.4
73.1
51.8
43.5
RANDOM
24.8
44.0
41.8
34.9
25.8
28.2
64.3
76.2
51.7
43.5
R+DWM_W
23.2
43.4
52.7
34.9
24.7
26.8
64.5
75.0
53.6
44.3
3
R+DWM_WA
23.8
44.0
49.3
35.2
27.5
29.4
64.6
76.9
51.8
44.7
R+DWM
23.8
44.4
49.3
34.9
24.7
28.4
63.8
78.3
52.2
44.4
RANDOM
24.1
45.3
53.7
35.9
22.3
28.2
64.6
76.4
50.8
44.6
R+DWM_W]
24.2
44.6
52.4
36.0
24.1
28.8
64.9
76.5
52.1
44.8
STAGE
R+DWM_W4
23.1
45.1
55.3
36.2
24.0
29.2
64.9
77.2
53.3
45.4
R+DWM
23.3
45.4
53.9
35.9
24.4
28.0
64.3
80.6
51.8
45.3
RANDOM
25.5
46.6
51.6
36.6
22.9
28.4
65.0
78.9
50.8
45.1
R+DWM_W
24.1
48.7
50.6
36.4
24.7
29.2
65.3
77.4
52.7
45.5
STAGE
R+DWM_W4
24.1
46.3
56.5
36.7
23.4
29.0
65.6
80.1
52.6
46.0
R+DWM
24.7
46.8
56.6
36.5
25.8
28.2
65.0
80.5
53.4
46.4
Table 12. Detailed zero-shot performance trained models with DWM using different validation tasks
METHOD
ARC-C
ARC-E
BOoLQ
H.S.
LOQIQA
OBQA
PIQA
SCIQ
W.G.
AVERAGE
RANDOM
24.1
41.2
52.7
36.8
26.6
28.0
65.2
70.9
50.8
44.0
R+DWM VAL
23.6
42.4
59.9
36.4
25.0
29.5
65.6
67.8
51.1
44.6
RANDOM+DWM
24.3
42.5
59.9
36.4
26.4
29.8
65.3
68.1
52.7
45.0
Table 13. Detailed two-shot performance of trained models with DWM using different validation tasks
METHOD
ARC-C
ARC-E
BOOLQ
H.S.
LOQIQA
OBQA
PIQA
ScIQ
W.G.
AVERAGE
RANDOM
25,5
46.6
51.6
36.6
22.9
28.4
65.0
78.9
50.8
45.1
R+DWM VAL
24.8
46.5
57.0
36.5
25.3
28.2
65.1
80.6
50.8
46.
RANDOM-DWM
24.7
46.8
56.6
36.5
25.8
28.2
65.0
80.5
53.4
46.4
Table 14. Zero-Shot performance of 370M pre-trained models with different numbers of training stages.
METHOD
ARC-C
ARC-E
BoolQ
H.S.
LoqIQA
OBQA
PIQA
ScrQ
W.G.
AVG
RANDOM
24.1
41.2
52.7
36.8
26.6
28.0
65.2
70.9
50.8
44.0
DWM.ESSTAGES
23.2
39.9
59.7
35.4
27.1
29.9
65.8
64.6
53.4
44.3
DWM
24.3
42.5
59.9
36.4
26.4
29.8
65.3
68.1
52.7
45.0
DWM.&.STAGE
25.2
41.7
60.6
36.0
27.3
30.6
65.7
67.3
52.6
45.2
14


Reviews:
[{"review_id": "fJi2yLRGrW", "reviewer": "Reviewer_ECYs", "summary": "This work proposes DWM, to address the limitations of existing data selection methods that ignore dynamic model training dynamics during LLM pre-training. Based on a bi-level optimization, DWM adaptively sets the weights of each data sample in a batch. In experiments, as a plug-and-play module, DWM improves the performance in downstream tasks across various settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and valuable review. Below, we address your questions and concerns in detail. \n\n---\n\n**Q1. Training Cost of DWM**\n\n**A1.** DWM does introduce additional computational overhead. We provide an analysis of this issue in our response to **R3Q1** and will include the discussion in revision.\n\n---\n\n**Q2. The Potential Limitation of DWM**\n\n**A2.** As a plug-and-play module, DWM captures the model’s dynamic data preferences during training and can be applied to either randomly sampled or curated data. However, its performance is ultimately bounded by the quality of the applied data. Notably, the SOTA (QURATING) leverages data selected from 260B tokens using knowledge of GPT-3.5-turbo, offering significantly higher quality than the 30B randomly sampled tokens. As a result, DWM on random data cannot surpass QURATING, but when applied to QURATING data, it further enhances performance. We will include this discussion in the revision.\n\n**Q3. The Performance Decrease of DWM in Partial Downstream Tasks**\n\n**A3.**  First, we clarify that DWM is trained on the 370M model using randomly selected data, and it consistently improves performance across nearly all downstream tasks under both zero-shot and two-shot settings. At most, a slight performance drop occurs on a single task, a phenomenon also reported in prior work [1][2], suggesting possible optimization conflicts among downstream tasks.\n\nSecond, performance degradation on certain tasks mainly arises when transferring the DWM (trained on 370M with random data) to the 370M model with QURATING data or the 1.3B model with DSIR data. This results from incompatibility between the model and the data during training.\n\nAs discussed in the paper (line 354) and in reference[3], models of different scales vary in their capacity to absorb high-quality reasoning data. Furthermore, as shown in Sec. 5.3 (line 401) and Fig. 3, DWM encourages data diversity in early stages and gradually shifts toward expert and instructional data. For the 370M model with QURATING data, high-quality reasoning data quickly saturates learning, leaving little room for further improvement through DWM. Although larger models have greater capacity, DSIR data (knowledge-centric text) similarly limits DWM's optimization effect on the larger 1.3B model. Consequently, DWM yields marginal average improvements in these transfer settings, resulting in mixed task-wise outcomes. Notably, DWM performs well on the 370M model with DSIR data and the 1.3B model with QURATING data, which supports our hypothesis.\n\nThese observations indicate that DWM’s effectiveness remains constrained by the applied data, highlighting the need for more robust weighting strategies adaptable to diverse data types.\n\nThanks. We will add this discussion in revision.\n\n---\n\n**Q4. The Two-Shot Setting**\n\n**A4.** We focus on the two-shot setting referring to exsiting data selection methods[4] to analyze the model's capacity for reasoning and generalization. In general, an appropriate number of examples can help the model better understand the task. Moreover, for models with limited capacity, an excessive number of examples may lead to overfitting on the samples and a decline in generalization ability. \n\n---\n**Q5. The Ablation of the Number of Stages**\n\n**A5.** Thanks. We provide the ablation studies of this number below, where increasing the number of stages facilitates DWM in better capturing the model's dynamic data preferences, but it also introduces additional training overhead for the weighting model. The results show that setting the number of stages to 5 achieves a good balance between performance and efficiency, indicating that the model preferences may not dramatically change within a single stage. We will add this portion in revision. \n\n| stages            | arc-c | arc-e | boolq | hellaswag | logiqa | obqa | piqa | sciq | winogrande | avg  |\n| :--------------- | :---: | :---: | :---: | :-------: | :----: | :--: | :--: | :--: | :--------: | :--: |\n| 2-stages          | 24.5  | 43.9  | 57.5  |   35.1    |  25.0  | 27.6 | 64.7 | 76.5 |    52.9    | 45.3 |\n| 8-stages          | 25.5  | 46.3  | 60.1  |   36.4    |  25.2  | 28.8 | 64.9 | 77.2 |    53.7    | 46.5 |\n\n---\n**Q6. Why Report the Stage-Wise Performance of 370M Model**\n\n**A6.** In our paper, we perform bi-level optimization on a 370M model, where the language model and the weighting model are trained separately. The trained weighting model is then directly transferred to a larger 1.3B model. Therefore, we report the stage-wise performance of the 370M model in Table 1 and Table 2 to analyze the effect of this bi-level optimization on model training.\n\n[1] Data Selection for Language Models  via Importance Resampling, 2023.\n[2] QuRating: Selecting High-Quality Data for Training Language Models, 2024.\n[3] Small Models Struggle to Learn from Strong Reasoners, 2025.\n[4] MATES : Model-Aware Data Selection for Efficient Pretraining with Data Influence Models, 2024.", "claims_and_evidence": "1. In introduction, the authors claim the training on all available data may not be optimal and training increase the financial costs, but the authors do not conduct efficiency test on the proposed methods. On the contrary, it seems that the additionally component will even introduce much more training time, which should be properly discussed in experiments. \n2. Moreover, the authors discuss the existing methods for data selection, such as selection before training and without considering the data samples in a batch indiscriminately. However, experiments show that, random sampling with DWM cannot outperform the SOTA, so I think the authors should also discuss the potential limitation of DWM.", "methods_and_evaluation_criteria": "The method design and evaluation criteria are generally convincing.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. While in average, we observe the performance improvement, the performance of lots of tasks also decreases with DWM. Could the authors also provide some explanations which may potentially instrumental to the improvement of DWM?\n\n2. In few-shot learning, why do authors only focus on 2 shots? What is the effect of the number of samples?\n\n3. Moreover, the number of stages are not properly ablated.\n\n4. Why do the authors report Table 1 and Table 2 with the 370M model? Since the authors have trained 1.3B model, reporting the performance of 1.3B model in different stages is much more convincing given that the paper focuses on LLM.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nThe proposed method is sound and the research topic is interesting and important.\n\nWeaknesses:\n\nSee claim and experiments setting.", "other_comments_or_suggestions": "Line 261, 'dose' -> 'does'", "questions_for_authors": "If the authors are able to solve the problems in experiments and claims, I would possibly adjust my rate.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "4YjcWG4yjF", "reviewer": "Reviewer_PMBR", "summary": "This paper introduces a novel Data Weighting Model (DWM) to enhance data utilization during large language model (LLM) pre-training. DWM provides a dynamic data selection method by dynamically adjusting the weights of data samples within each training batch using a bi-level optimization framework. This framework trains a weighting model to learn the data preferences of the LLM as it evolves, allowing for more effective data utilization and improved model performance. The authors demonstrate that DWM can improves the performance of LLMs trained with randomly selected data and can also enhance existing data selection methods like DSIR and QuRating. DWM presents a promising approach for optimizing data selection and utilization.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Review 2 Rebuttal\n---\n\nThank you for your thoughtful and detailed review. Below, we address your questions and concerns in detail. \n\n---\n\n**Q1. The Missing Literature**\n\n**A1.** Thanks for your suggestions, and we have added the discussion of these related work below as well as in revision. \n\nExisting data selection methods fall into three categories: (1) **Token-level**, which filters individual tokens (e.g., Rho-1); (2) **Group-level**, which mixes data pools (e.g., Regmix); and (3) **Sample-level**, which selects individual examples. Sample-level methods include heuristic or learning-based approaches (e.g., TAPT, S2L, DSIR, QURATING) and theoretically grounded coreset-based methods (e.g., IG, CREST).\n\nOur method also belongs to the sample-level category but differs by modeling the model’s dynamic data preferences and capturing joint data effects during training. Through data weighting, DWM improves data utilization and can be easily transferred across models or combined with other selection strategies.\n\n---\n\n**Q2. The Missing Baseline**\n\n**A2.** Thanks for your suggestion and we provided the dicussion and comparion of Data Shapley.\n\nUnlike DATA Shapley, our method (1) generalizes via end-to-end learning without per-dataset recomputation, and (2) optimizes the model directly rather than estimating fair contributions. For comparison, we trained a Shapley-based weighting model on the same data as DWM. DWM outperforms it, highlighting the benefit of learning data utility directly from the model.\n\n| method           | arc-c | arc-e | boolq | hellaswag | logiqa | obqa | piqa | sciq | winogrande | avg  |\n| :------------- | :---: | :---: | :---: | :-------: | :----: | :--: | :--: | :--: | :--------: | :--: |\n| random-shapley | 24.3  | 44.5  | 54.2  |   36.3    |  24.6  | 28.8 | 64.2 | 77.7 |    52.2    | 45.2 |\n| random-DWM     | 24.7  | 46.8  | 56.6  |   36.5    |  25.8  | 28.2 | 65.0 | 80.5 |    53.4    | 46.4 |\n\n---\n\n**Q3. The Reliance on the LAMBADA**\n\n**A3.**  We would like to emphasize that DWM is not heavily dependent on the validation set.\n\nDWM uses LAMBADA as validation due to its common use in language model pretraining [1–2]. Other reasoning datasets, such as HellaSwag (Training set), also serve well as shown below, where DWM trained with HellaSwag validation maintains strong performance on the 370M model.\n\n| method                   | arc-c | arc-e | boolq | hellaswag | logiqa | obqa | piqa | sciq | winogrande | avg  |\n| :--------------------- | :---: | :---: | :---: | :-------: | :----: | :--: | :--: | :--: | :--------: | :--: |\n| hellaswag-training set | 24.7  | 46.8  | 56.9  |   36.5    |  26.3  | 28.2 | 64.7 | 80.9 |    51.5    | 46.3 |\n| lambada (DWM)          | 24.7  | 46.8  | 56.6  |   36.5    |  25.8  | 28.2 | 65.0 | 80.5 |    53.4    | 46.4 |\n\n---\n\n**Q4. The Experimental Results**\n\n**A4.** Here we address the concerns related to the experimental results，which are obtained using the same model architecture or benchmark as in existing methods [2,3].\n\n**1. The Performance Improvement.** We would like to clarify that the performance gains of the DWM algorithm on average can match or even surpass the results reported in reference [1,3,4], demonstrating the effectiveness of our method. \n\n**2. The Performance Decrease of DWM in Partial Downstream Tasks**  We would like to emphasize that DWM is trained on the 370M model, and it improves the performance of the 370M model on nearly all downstream tasks. The performance drops on partial downstream tasks primarily occur when transferring the trained DWM model to specific model–data combinations, which is mainly caused by incompatibility between the training model and the training data. More detailed explaination can be found in our reply to **R1 Q3**.\n\n---\n\n**Q5. Training Cost DWM**\n\n**A5.** DWM does introduce additional computational overhead. We provide an analysis of this issue in our response to **R3Q1** and will include the discussion in revision.\n\nIn addition,  We show that DWM trained on 30B tokens matches the performance of a 370M model trained on 48B random tokens, yielding a 1.6× gain in training efficiency.\n\n| stages      | arc-c | arc-e | boolq | hellaswag | logiqa | obqa | piqa | sciq | winogrande | avg  |\n| :---------- | :---: | :---: | :---: | :-------: | :----: | :--: | :--: | :--: | :--------: | :--: |\n| random-48B  | 24.9  | 46.9  | 58.4  |   38.1    |  26.4  | 28.8 | 65.2 | 78.6 |    51.1    | 46.5 |\n| DWM         | 24.7  | 46.8  | 56.6  |   36.5    |  25.8  | 28.2 | 65.0 | 80.5 |    53.4    | 46.4 |\n\n[1] DsDm: Model-Aware Dataset Selection with Datamodels, 2024.\n[2] MATES : Model-Aware Data Selection for Efficient Pretraining with Data Influence Models, 2024.\n[3] QuRating: Selecting High-Quality Data for Training Language Models, 2024.\n[4] Data Selection for Language Models  via Importance Resampling, 2023.", "claims_and_evidence": "Yes.\n\nThe experiments do support the inclusion of the method to improve performance.", "methods_and_evaluation_criteria": "Yes.\n\nThe datasets used for evaluation and training are reasonable. The models are reasonable however small or outdated.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "It seems that with addition of DWM, the performance on some tasks improve but on others decrease significantly (Table 4). It is not clear how to understand the benefit of having DWM and disadvantage.\n\nThe reliance on the LAMBADA as the sole validation set seems limiting.", "relation_to_broader_scientific_literature": "The contribution of data selection is important in general and provides another method for data selection.", "essential_references_not_discussed": "Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. \nDon’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020\n\nLin, Zhenghao, et al. \"Rho-1: Not all tokens are what you need.\" arXiv preprint arXiv:2404.07965 (2024).\n\nLiu, Qian, et al. \"Regmix: Data mixture as regression for language model pre-training.\" arXiv preprint arXiv:2407.01492 (2024).\n\nYang, Yu, et al. \"Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models.\" Advances in Neural Information Processing Systems 37 (2024): 83465-83496.\n\nMirzasoleiman, Baharan, Jeff Bilmes, and Jure Leskovec. \"Coresets for data-efficient training of machine learning models.\" International Conference on Machine Learning. PMLR, 2020.\n\nYang, Yu, Hao Kang, and Baharan Mirzasoleiman. \"Towards sustainable learning: Coresets for data-efficient deep learning.\" International Conference on Machine Learning. PMLR, 2023.", "other_strengths_and_weaknesses": "Strengths:\n\n- The generalizability of the method to be adapted to other selection methods is beneficial.\n- The method shows some improvements in performance when adapted.\n\nWeaknesses:\n\n- The method requires an additional model for weighting requiring additional computation.\n- Some missing literature as well as baseline methods to study (provided in the list of missing references).\n- The missing baseline is missing (provided in missing literature, e.g., Data Shapley).\n- The method does not provide much improvement in performance. \n- Newer benchmarks could be used for evaluation of the model.", "other_comments_or_suggestions": "Figures can be more readable (Figure 2).\n\nTime and memory complexity could be provided in the paper.\n\nAdding more qualitative understanding (through examples) of the model's dynamic changes could be helpful to better understand the weights.", "questions_for_authors": "It would be interesting how much additional random tokens are required to match the performance with DWM with less tokens.\nWould computational time be saved?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6xtfcFD69x", "reviewer": "Reviewer_6wW8", "summary": "The paper introduces a Data Weighting Model (DWM) that dynamically adjusts data weights during LLM training using a bi-level optimization framework. DWM captures evolving data preferences by iteratively updating a weighting model based on validation performance. Experiments on 370M and 1.3B models demonstrate improved performance over static data selection methods, transferability across models and datasets, and insights into evolving data preferences.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and encouraging review. Below, we address your questions and concerns in detail.\n\n---\n\n**Q1. The Training Cost of the Bi-Level Optimization** \n\n**A1.**  We would like to clarify that in DWM, we employ a bi-level optimization strategy on the 370M model to separately train the weighting model and the language model. Once training is completed, the learned weighting model can be directly transferred to larger models without additional training. \n\nBesides, using a trained data weighting model does introduce some computational overhead. Referring to [1], the training cost in FLOPs can be approximated as:  \n\n**Training FLOPs ≈ C × Model Parameters × Token Count**,  \n\nwhere the constant **C** depends on whether backpropagation is performed. In our case, since the weighting model only performs forward inference when assisting the training of larger models, **C** can be approximated as 2 (compared to 8 for full backpropagation). Therefore, when transferring the 370M weighting model to the 1.3B model, the additional training overhead is roughly 9%, and this overhead decreases as the size of the target model increases.\n\nThanks, and we will add this discussion in revision.\n\n[1] Training Compute-Optimal Large Language Models, 2022.", "claims_and_evidence": "Overall, both theoretical perspectives and solid empirical findings support to their claims.", "methods_and_evaluation_criteria": "The methods in this paper are composed of a bi-level optimization framework, dynamic data weighting, and multi-stage alternative iteration. The bi-level optimization framework jointly optimizes the LLM and the weighting model, with the weighting model updated to maximize the trained model's performance on a validation set. Dynamic data weighting is implemented by DWM, which assigns weights to data samples within each batch, considering their interactions and the model's current preferences. In the training process, the multi-stage alternative iteration alternates between updating the weighting model and the LLM parameters in stages to capture dynamic data preferences. As for the evaluation criteria, the model performance is evaluated on nine downstream tasks under zero-shot and two-shot settings, with normalized accuracy as the metric. These criteria align with standard evaluation practices in the field of data selection methods.", "theoretical_claims": "The bi-level optimization approach effectively captures the dynamic data preferences of the model, improving data utilization and generalization. the weighting model learns to assign higher weights to data samples that are more beneficial for the model's performance, adapting as the model evolves during training.", "experimental_designs_or_analyses": "The paper presents a comprehensive experimental design. In terms of data selection, it utilizes randomly selected data and compares results with state-of-the-art data selection methods like DSIR and QuRating. For the model architecture, Llama-2 models with 370M and 1.3B parameters are employed, which are trained on 30B tokens selected from the SlimPajama dataset. The training setup uses LAMBADA as the validation set, splits training into five stages, and adopts a micro-batch size of 8. Moreover, an ablation study is conducted to compare models trained with static (fixed after early or late stages) versus dynamic weighting models, effectively highlighting the significance of continuous adaptation.", "relation_to_broader_scientific_literature": "The authors contextualize their work well within the literature on data selection, such as Qurating, DSIR.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: \n1.\tThe authors propose a novel bi-level optimization framework that effectively tackles the limitations of static data selection by explicitly modeling dynamic interactions and preferences, which is a significant advancement in the field. \n2.\tThe paper demonstrates the transferability of DWM by achieving consistent improvements across larger models (1.3B) and a variety of data selection methods such as DSIR and QuRating, highlighting its formidable transferability and generalization ability.  \n3.\tThe authors undertake comprehensive evaluation across 9 benchmarks under zero-/few-shot settings and supplements it with ablation studies that strongly validate the design choices, ensuring the robustness and reliability of the proposed approach.  \n4.\tThis paper offers insightful analysis on how data preferences evolve during training, such as the shift from prioritizing expertise to writing quality in later stages, which adds a deeper understanding to the training dynamics. \nWeaknesses:  \n1.\tThe bi-level optimization framework, while innovative, likely increases training costs. However, these additional costs are not quantified in the paper. In my opinion, the additional costs should be analyzed.", "other_comments_or_suggestions": "The costs of bi-level optimization framework should be analyzed.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCZnQH36O2", "reviewer": "Reviewer_QCUf", "summary": "The paper proposes a dynamic bi-level optimization framework to improve data selection and utilization during LLM training. The bi-level optimization includes updating model parameters using data weighted by a weighting model, and optimizing the weighting model based on the model’s updated performance. Experiments show the proposed method enhances training efficiency and model performance, and can transfer across models and data selection methods.\n\n## update after rebuttal\n\nI have no more concerns, and recommend to accept this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and encouraging review. Below, we address your questions and concerns in detail.\n\n---\n\n**Q1. The Computing Cost of the Bi-Level Optimization** \n\n**A1.** DWM does introduce additional computational overhead. We provide an analysis of this issue in our response to **R3Q1** and will include the discussion in revision.\n\n---\n\n**Q2. The discussion of the Contributions of Dynamic Weighting and Bi-Level Optimization** \n\n**A2.** Sorry for this unclear expression. We compare the contributions of dynamic weighting and bi-level optimization in Sec.5.4 and Fig. 4 in the paper. We present the result of RANDOM, as well as RANDOM_DWM_W1 and RANDOM_DWM_W4, which apply the weighting models from the first and final stages, respectively, throughout the whole training stages. Note that the weighting model used in RANDOM_DWM_W1 or RANDOM_DWM_W4 is trained by the bi-level optimization. We also include our DWM method that dynamically learns the weighting model during training. \n\nAs shown in Fig. 4, although using a single-stage weighting model obtained through bi-level optimization (RANDOM_DWM_W1 or RANDOM_DWM_W4) improves model performance, dynamically learning the weighting model (DWM) allows for adaptation to the model's evolving data preferences across training stages, resulting in more robust performance gains.\n\nThank you for your suggestion, and we will revise the writing of the paper and explicitly highlight the contributions of these two components.", "claims_and_evidence": "The paper claims the method improves model performance and training efficiency via dynamic data weighting. The evidence is shown in Table 1~4: the proposed DWM improve the accuracy across different model sizes and base data selection methods. Figure 3 shows DWM shifts weights from high-perplexity to diversity-focused data as training progresses.", "methods_and_evaluation_criteria": "Yes, the paper compares the proposed method with a series of sota compression methods. The evaluation makes sense and reasonable.", "theoretical_claims": "No theoretical proof in the paper.", "experimental_designs_or_analyses": "The experiments are well organized, evaluating DWM on the diverse benchmarks, comparing against static data selection baselines (e.g, random, DSIR, MATES). The method is also transferred to larger models and other selection methods to validate generalizability. The ablation studies isolate the impact of bi-level optimization and dynamic weighting are well designed.", "relation_to_broader_scientific_literature": "The work builds on static data selection methods and optimization frameworks like meta-learning, addressing their limitation of ignoring dynamic model preferences during LLM training. Deeper comparisons with adaptive sampling strategies like active learning and gradient-agnostic dynamic weighting approaches could further clarify its positioning.", "essential_references_not_discussed": "No Essential References Not Discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n-\tThe proposed method addresses the limitation of static data selection by dynamically adjusting data weights based on the model’s preferences.\n\n-\tThe bi-level optimization framework can jointly consider effects of data samples and model update, improving data utilization effectiveness.\n\n-\tExtensive experiments demonstrates its applicability to LLM training and compatibility with other selection methods, enhancing its practical utility.\n\nWeaknesses:\n\n-\tBi-level optimization may introduce significant computational costs, compared to the baseline static selection methods like DSIR. Please clarify this issue.\n\n-\tThe paper does not clearly show the contributions of dynamic weighting and bi-level optimization respectively, making it unclear which aspect drives the improvement.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a Data Weighting Model (DWM) that dynamically adjusts data weights during LLM training using a bi-level optimization framework. DWM captures evolving data preferences by iteratively updating a weighting model based on validation performance. Experiments on 370M and 1.3B models demonstrate improved performance over static data selection methods, transferability across models and datasets, and insights into evolving data preferences.

This paper has many strengths. For example, the authors propose a novel bi-level optimization framework that effectively tackles the limitations of static data selection by explicitly modeling dynamic interactions and preferences, which is a significant advancement in the field. The paper demonstrates the transferability of DWM by achieving consistent improvements across larger models (1.3B) and a variety of data selection methods such as DSIR and QuRating, highlighting its formidable transferability and generalization ability. The authors undertake comprehensive evaluation across 9 benchmarks under zero-/few-shot settings and supplements it with ablation studies that strongly validate the design choices, ensuring the robustness and reliability of the proposed approach. This paper offers insightful analysis on how data preferences evolve during training, such as the shift from prioritizing expertise to writing quality in later stages, which adds a deeper understanding to the training dynamics.

While the reviewers had some concerns about experiments and validation set, please include the additional discussion in the next version.
Decision: Accept (poster)