Self-Influence Guided Data Reweighting for Language Model Pre-training
Authors: Megh Thakkar
Tolga Bolukbasi
Sriram Ganapathy
Shikhar Vashishth
Sarath Chandar
Partha Talukdar
Venue: EMNLP 2023 Main (2023)
Keywords: pre-training
multilingual pretraining
self-influence
language models
data reweighting
TL;DR: A dynamic and online adaptation of self-influence computation for reweighting data for pre-training

Abstract: Language Models (LMs) pre-trained with selfsupervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are treated with equal importance during LM pre-training. However, due to varying levels of relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored in the context of task-specific supervised learning and LM fine-tuning, model-driven reweighting for pretraining data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores as an indicator of sample importance and pre-training. PRESENCE promotes novelty and stability for model pre-training. Through extensive analysis spanning multiple model sizes, datasets, and tasks, we present PRESENCE as an important first step in the research direction of sample reweighting for pre-training language models.

Self-Influence Guided Data Reweighting for Language Model Pre-training
Megh Thakkar
Tolga Bolukbasi?
Sriram Ganapathy
Shikhar Vashishth
Sarath Chandar
Partha Talukdar
Mila Quebec AI Institute
Google Deepmind Google Research India
Indian Institute of Science *Polytechnique Montréal
Canada CIFAR AI Chair
{megh. thakkar, faiikarrsarrr sarath. chandar Jemila yemila.quebec {srigana, shikharv, tolgab, par partha) Jegoole com
Abstract
Prior works for both model-based sample se- lection (Swayamdipta al., 2020) and reweight- ing (Mindermann et al., 2022) use supervised learning setup. They often rely on curating special validation sets (Jain and Shenoy, 2022), proxy mod- els (Pleiss et al., 2020; Mindermann et al., 2022), or utilizing loss and prediction uncertainty signals based on ground- trrut labels (Kawaguchi and Lu, 2020; Coleman al., 2020). Adaptation of these methods pre-training is often non-trivial. Per- formance of a pre-trained model on downstream tasks cannot be predicted by its pre-training valida- tion performance. Moreover, offline filtering using proxy models is quite expensive for the massive scale of pre-training data (Liu et al., 2022).
Language Models (LMs) pre-trained with self- supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus treated with equal impor- tance during LM pre- training.. However. due varying levels relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored the context of task-specific supervised learning and LM lin-ttning, model-driven reweighting for pre- training data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores in- dicator of sample importance and pre-training. PRESENCE promotes novelty and stability for model pre-training. Through extensive analysis spanning multiple model sizes, datasets, and tasks, we present PRESENCE as an important first step in the research direction of sample reweighting for pre-training language models.
In this paper, we attempt to develop an effective data reweighting framework for language model pre-training. We use self-infuunnc (SI), the degree to which a given training sample affects model training and its own prediction, as an indicator of sample importance for pre-training, SI scores have been previously shown to effective in identify- ing noisy and outlier samples (Yeh et al., 2018), but these evaluations have been limited to super- vised settings. We first verify the ability of SI scores to predict sample quality of pre-training data, such noisy text and domain mismatched samples. We then probe their effectiveness for pre-training dataset selection by using them to filter out noisy samples in the pre-training data derived from the web.
1 Introduction
Language models (LM), typically pre-trained on large volumes of unlabeled text data, have become ubiquitous model choices for various challenging downstream tasks (Lewkowyez al., 2022; Driess al., 2023). The fundamental direction pursued for improving language model pre-training involves increasing the amount of training data or scaling model size (Scao ct al., 2022). The training data is generally assembled from scraping the web and filtered using manually crafted heuristics that often require domain expertise (Xue et al., 2021). A key similarity in these prior works is the uniform treat- ment of the data samples available in the assembled corpora, without any regard for the data quality. Work done while at Google Rescarch India
Based on our analysis which shows that self- influence scores can be used as an indicator of sample importance, we propose PRESENCE: Pre- training data Re Re-weighting with Self-influence. PRESENCE is online and adaptive data weighting method that uses self-influence scores to weigh samples a training batch. We note that during pre-training, the training loss decreases exponentially in the initial steps, with a minimal de- crease in loss values in the subsequent stages (Yang

---

al., 2021). Furthermore, well-trained models can identify noisy samples better when used cal- culate SI scores, as compared to models in very early stages of training (Pruthi et al., 2020). Based on these observations, we formulate two stage reweighting strategy: (i) in the first stage of learn- ing, data samples with higher SI scores are em- phasized more to drive learning using influential samples, while (ii) in the second stage, data sam- ples with higher SI scores are de-emphasized. This limits the impact of noisy and unreliable samples while giving more weight to the higher quality sam- ples. To the best of our knowledge, this is the first work that evaluates the use of influence functions for sample selection and reweighting at the scale of pre-training. Our contributions are as follows:
0.08
Engitsh Hindi
0.33
comain
Physica Chemistry
Bengal
S8 Soore
0.08
0.02
Noley Text
0.01
Figure I: Left Average SI scores across English, Hindi, and Bengali for clean and noisy (jumbled) text; Right Average SI scores for randomly sampled and domain mismatched information in Finglish. We observe high SI scores on average for noisy and domain mismatched text. More discussions in Section 2.1.
We initiate study into data reweighting for pre- training and establish the relationship of self- influence (SI) scores with sample characteristics such as noise and domain mismatched informa- tion in the training data.
order gradient approximation. For model f with parameters @ and loss function I(fo, +). the gradient g(O; ((0-)) for sample is g(fe, 2) v((fe, The TracIn(Je: (fe.n)( :) influence of training sample on test sample z' is given by.
We present sequential data filtering using SI scores as an effective data selection strategy for pre-training, and evaluate the performance of models pre-trained on large-scale filtered datasets. We call this method PRESENCE- Sequential.
TracIn(Jo. x, g(fo, 2) g(fo,==))
(I)
Self lelf-influence score measures the influence sam- ple has on itself. This identical replacing a' with 2 in Equation I, giving TracInSI(fo. -) as,
TracInSI g(fo, g(fo,z)
Building our findings, we propose PRESENCE, model-driven sample reweighting method using self-influence scores that jointly weighs samples and enables learning. PRESENCE promotes nov- city and stability for model pre-training.
2.1 Relationship between Self-Influence and Sample Quality
We investigate the relationship between self- influence (SI) scores and sample quality by probing noisy and domain mismatched samples. We expect these samples to have high self-influence scores as they tend to reduce the loss w.r.t. well-trained model (Yeh et al., 2018). We use a pre-trained mT5-base (Xue et al., 2021) model and calculate self-influence with a span-corruption loss. We ran- domly sample 10, 000 samples for three languages in mC4 (Raffel et al., 2020) and calculate the self- influence scores of clean or original samples and their corresponding noisy samples, i.e., samples with permuted word order. Similarly, we cal- culate average self-influence scores over domain mismatched samples and compare them with aver- age scores over randomly sampled English corpus.
Through extensive experiments and analyses spanning multiple model sizes, datasets, and tasks, we demonstrate that PRESENCE provides consistent gains over tre-tnaining using randomly sampled pre-training corpora or SI score based filtered data. We believe PRESENCE is an impor- tant step in the research direction of data sample weighting for pretraining.
2 Background: TracIn and Self-influence
Though PRESENCE can be used with any influence function, we use TracIn (Pruthi et al., 2020) based self-influence score due to its scalability, generaliz- ability, and effectiveness in identifying outliers.
Self-influence using TracIn: TracIn computes influence i.e., how the loss on the test point changes during the training process whenever the training sample of interest was utilized by a first-
As shown in Figure 1, we observe substantially high average self-influence scores for noisy sam- ples across all languages as well as for domain mismatched text in English. The results indicate

---

Random Sampling
Scoring Model (Fa)
high
score
Final Model
PRESENCE- Sequential
Tracin Scoring
D
low
Web Assembled Corpus (D)
Batch
Batch
Batch Weighting
Weighting
Weighting
Random (Ma)
M,
Misst
Final Model
PRESENCE
initialization
score
Si score
Si score
Batch: Bq
B2
Bm
Direct Weighting (t
Inverse Weighting (t <)
(Somples with high emphasized)
(Samples with SI are de-emphosized)
Figure 2: Overview of PREsENCE-Sequential (top, Section and PRESENCE (bottom, Section 4.1). PRESENCE- Sequential filters out data in sequential manner, first training scoring model and then using it to filter data. PRESENCE is a joint sample reweighting strategy that leverages SI scores for sample weighting within a minibatch.
that SI scores be used to distinguish between correct and noisy text and they also be used to detect data from novel domain.
Dataset XQuAD
Task
# Languages
Metric
Question Answering
10
Fl
Question Answering
TyDi QA
Question Answering
11
Fl
XNLI
Pair
Accuracy
3 Sequuettl: Filtering Pre-training Data using Self- Leff-Infuennc
WikiAnn NER
Structured Prediction
40
Span-F1
Table 1: Datasets, tasks, languages and metrics.
Extending TracIn based Self-influence for Pre- training: As pre-training is computationally ex- pensive, we leverage the layer agnostic nature of TracIn and introduce an optimized layer-wise self-influence calculation. For layers K [ki, kg,...,kk) of model fo. let fo.k denote the parameters of layer k. Self-influence for any layer set K. C K is,
randomly sampled data (SI Scoring Model). To obtain relevant pre-training set D (IDI N). from the larger corpora D' (ID') N²), we use the scoring model F- to calculate the SI scores using Equation 3 for all samples di E D',
TracInSI< D') (Fr D')) (4)
TracInSl<(fo.2)
Σ fo..)))l(fo...:) AER.
We sort TracInSIx: D') in increasing order of SI scores and filter out N' - N samples with the highest score. The remaining N samples comprise the filtered D used for pre-training.
As shown in Section 2.1, there is a relation be- tween SI scores and the sample quality. We lever- age this property to filter large-scale web corpora in an offline manner to create more suitable pre- training data. We present an overview of our offline sequential filtering strategy using self-influence scores, called PRESENCE-Sequenttal in Figure 2. Assuming that a model requires N training samples for pre-training, we choose N samples from a set N' > samples by filtering out samples with the highest SI scores using proxy model trained
D [di]d. E D')
(5)
Wi E Tra ('),
Pre-training Setup: We the mC4 dataset (Xue et al., 2021) and pre-train an mT5-base model for 200, 000 steps on randomly shuffled data, and use this as the 'Scoring Model (F))' to create the

---

Question Answering
Sentence Pair
Structured
Model
XQuAD
MLQA
TyDi QolG
XNLI
WikiAnn NER
Metries
F1
FI
FI
Accuracy
Span-F1
Cross-lingual zero-shot transfer (models fine- fine-tuned on English data only)
m15-base"
72.33
61.60
49.09
69.98
41.42
m P Sequentiial-reeesee
70.17
59.72
47.92
68.64
37.39
73.40
61.95
51.64
71.22
44.63
Translate-train (models fine-tuned on English data plux translations larget languages)
mt5-base"
78.
65.45
52.75
76.76
80.86
m mTS-base WwwsseeREEENEE-SSSSE reepeeie--eseeenniilreeeessee m P
76.83
64.76
46.01
75.36
79.14
78.96
66.04
57.65
77.74
81.45
Table 2: Performance comparison of using PRESSENCC-SSqqentii to filter out pre-training data. PRESENCE- Sequential filters out noisy pre-train pre-training samples using SI scores and achieves better results than the baselines. Bold shows the best result (discussions Section 3). denotes our reproductions. Discussion in Section 3.1.
filtered dataset. We pre-train an mT5-base model from scratch on the filtered mC4 set for 200, 000 steps by choosing samples with the least SI scores that are theoretically more suitable for model learn- ing. The models are trained with a batch size of 1024, with an input token length 1024 and output token length of 229. Following Raffel et al. (2020), we use a base learning rate of 1.0 with 10000 warm- up steps, an inverse square root learning rate decay schedule, and loss-normalizing factor of 231196. We the first layer of the encoder and the first layer of the decoder in the set K, for TracInSIg.
scores can be used as an indicator of sample qual- ity and can be used for pre-training dataset filter- pre-train ing. To further test our hypotheses, model on data created by removing low SI sam- ples (reverse ranking). We label this model mT5- PRESE-PE--EEEE-CCESSSS
reversse. This model performs significantly worse compared to the base- line, further validating that SI scores are indeed an indicator of the sample quality, and are effec- tive in identifying noisy samples in the large-scale pre-train pre-training corpora.
However, mentioned, PRESENCF-Sequentee
requires different expensive sequential processes: (i) Pre-train a SI scoring model, and (ii) pre-train second model on the filtered dataset. Since pre- training is computationally expensive, we explore joint sample reweighting adaptation next.
Downstream Tasks and Fine-tuning: Follow- ing Xue al. (2021), we utilize datasets across 5 tasks from the XTREME multilingual bench- mark (Hu et al., 2020), including Question An- swering, Sentence-Pair, and Structured Prediction. We evaluate on (i) zero-shot cross-lingual transfer: where the fine-tuning data is only in English, and (ii) translate-traie train: where the fine-tuning data is in English and translated into the target languages for all the downstream datasets. We summarize the datasets used for evaluation in Table We fine- tune all the models on the downstream tasks using a batch size of 128, with a learning rate of 0.001, and dropout rate of 0.1 for 20, 000 steps.
PRESENCE: Sample Reweighting using Self-influence
In this approach, we the SI scores in line joint setting by reweighting samples at the minibatch level. We calculate sample SI scores at each training step and use them to weigh sample gradients before aggregating them for the gradient update. To formulate sample reweighting using SI scores, we consider batch B = { i E n]. where denotes sample. We calculate SI scores using Equation where [S] n, (a.|s. TradnSl [1,
3.1 Results and Analysis
3 for each sample to get array S,
We compare the performance of the model pre-trained filtered web corpora (mT5- bas++PRESEENEF-Seeuennll with the baseline model trained on randomly sampled data in Ta- ble 2. We observe that when we filter out samples with high SI scores, we obtain consistent gains over the baseline models. This indicates that ST
(6) where s; denotes the SI score of z;. We normalize for numerical stability and uniformity,
+ normalize(S)= $ S-
Voz(S)+ +

---

Algorithm 1 Microbatched Training
Traniing Loss
B Batch
Stage Waighing
Stage Welghting
G { Gradient array
>
r < 0
G 0 Gradient initialization
for microbatch by in minibatch do
9: &.)
Training Steps
g GUg
Figure 3: PRESENCE'S two-stage reweighting based on the training loss. We perform direct weighting in the initial stage and inverse weighting next (Section 4.1).
for g. G do G gr
SI scores to limit the impact of noisy and unreliable samples. Two-staged reweighting ensures that the model learns novel information early, and is able to climinate noise a later stage with stable learning. For temperatures 71 0 and 72 O, the softmax temperature training step is given by.
where (1-) and (-) denote the mean and variance, respectively, and is a small number. To calculate relative weights for the samples' gradients, use softmax function, softmax(-) over each 8; with temperature get weights wi
w; softmax(s:,,)
(8)
Ex,es en-
Th i S r2 i>
(10)
Here, wi gives the weight for the gradient of sample 2,. Using weights wi, the gradient G' for the model update is given by,
where denotes the step where we switch stages, We refer to model- online sample reweight- ing strategy using SI scores as PRESENCE. We now adapt PRESENCE the scale of pre-training.
Σ ax g(fo. 2)) ALED
4.1 Two-staged Reweighting for Pre-training A common concern of pre-training the redun- dancy of training data even in massive corpora mined from the web (Raffel et al., 2020). Evidently, training loss decreases exponentially early in the pre-training (Figure 3). Hence, training pro- ceeds, it quickly becomes the case that almost all the computation involved in training spent on concepts the model has already seen many times. High SI scores indicate high gradient norms, which are critical for driving model training (McRae et al., 2022). However, encountering high SI scores from well-trained models is often signal of noise outlier samples (Pruthi et al., 2020). We combine the information about the property of SI scores relative the ability of the model calcu- late them and the nature of the pre-training data to devise a novel two-stage sample reweighting strategy. We utilize the temperature term т when calculating the softmax weights (Equation 8) to for- mulate the two stages. In the first stage, which we call 'direct' weighting, the data samples with higher SI scores more em- phasis, driving the model learning and promoting novelty. the second stage, or 'inverse' weight- ing, where the model has matured, we use T < 0. This de-emmhasszes the data samples with higher
5 Pre-training Adaptation
Pre-training requires large batch size and is prone to instability (Krizhevsky et al., 2017). We thus adapt PRESENCE for pre-training by applying it at the microbatch level. This provides dual ben- efits of regularizing the pre-traine pre-training while being computationally efficient.
5.1 Reweighting Microbatch Gradients using Self-Influence
Microbatched Training Microbatched training enables the use of larger effective minibatch size. involves dividing the full minibatch B into smaller batches, called microbatches, and indi- vidually calculating gradients for each microbatch. These gradients are then aggregated to get the mini- batch gradient G. We present a standard micro- batched training algorithm in Algorithm 1, assum- ing that minibatch B is divided into n micro- batches, i.e. B {bili i E [1.
choose > 0, giving
We first calculate the self-influence for a micro- batch by replacing the individual sample a with a microbatch in Equation to calculate the loss. MicrobatchSIx for microbatch b is.
MicrobatchSll b) = TracinSl (f..
(11)

---

Question Answering
Sentence Pair
Structured WikiAnn NER
Model
XQuAD
MLQA
TyDi QA-GoldP
XNLI
Metrics
F1
FI
F1
Accuracy
Span-F1
Cross monss-lingullancc zero-shot transfer (models fine-tuned on English data only)
mt5-base"
72.92
64.71
49. 19.23
74.21
43.28
PRESSNC
74.12
65,40
53.17
74.47
43.41
m15-large"
64.15
51.61
58.75
67.96
38.66
G.TSlage+PRESENCC
77.78
70.40
62.33
78.54
39.23
Translate-train (models fine-tuned English data plus translations target longuages)
mi5-base
78.76
64.33
59.95
77.56
79.45
1155-bac++PRSSNCC
80.44
65,88
61.75
80,48
80.50
m15-targe*
62.78
53.
66..42
61.80
70.66
PRESSEC 83.15
70.30
69.04
79.72
77.26
Table Performance comparison of mTT5-bass and m1T5--aarge models pre- pre-trained using PRESENCE with baseline pre-trained models. PRESENCE gives consistent gains over corresponding baselines. Bold shows the best result. denotes our reproductions. Note that we use batch-size of 1024 for pre-training mt5- base and a batch-size 512 for pre-training mt5-large. Detailed discussion in Section 6.1.
Model Variant
XQuAD
XNLI
Algorithm 2 Weighted Microbatched Training
lar-larree
73.52
69.40 88.10
+- Batch
G = B <- Gradient array
87.44
= 0 < Self-influence array
+ Gradient initialization
Table 4: Results on the En-only subset for the translate- train setting. Details Section 6.
T + weighting temperature
for microbatch by in minibatch do
9: vi(fo,b.) 4.)
To formulate microbatch level reweighting using their self-inff celf-influence scores, calculate the self- influence using Equation 11 for each microbatch to get array S, where |S| n.
8; 9: gi
G + 9 U
8 + U
S + normalize(S)
W (softmaxí (st, (i, 7) i E S}
for 9.. 10 in G. W do
G G g: Wx
S= (8.18, MicrohatchSI,c(fo.. b.); [1,m]
(12)
where s; denotes the SI score of bg. Using the updated array in Equation 6 and microbatch train- ing strategy (Algorithm 1), we obtain the gradient for the model update G using Algorithm 2 with SI based reweighting in Algorithm 2.
the minibatch into n 8 microbatches and for 1115-large-PRESEEE we divide the minibatch into n microbatches. We select TΙ 1, 52 = 1, and 100, 000 for the two-staged learning. We use the first layer of the encoder and first layer of the decoder as the layer set IC. We use base learning rate of 1.0 with 10, 000 warm-up up steps and an inverse square root decay schedule. pre-training for million steps.
5.2 Training Setup
We use two different variants of the T5 architec- ture (Raffel et al., 2020), namely mT55base and mT5-large for comparisons and pre-train on the mC4 dataset (Xue et al., 2021). We refer to our corresponding reweighted variants as mT5-base- PRESENCE respectively. We pre-train the models with input length 1024 and output length 229. using batch sizes of 1021 for mT5-base and 512 for mar-largee We use loss- normalization during training with a loss normaliza- tion factor 234496 for mT5-base and 117248 for mT5-large. For 115--aae--RRESENEE PRESENCE, we divide
Results and Analysis
Effectiveness of PRESENCE
PRESENCE and
We compare the performance of using PRESENCE with m'55base (nTTS-baac+PRREENCE) and m'T5- large (mIT5-large+PRusSNNC) with random pre- training in Table 3. We observe that for both vari- ants, using PRESENCE helps improve performance on all the datasets considered, This validates the

---

90
mT5-baas
Variant
MLQA
TyDi QA
XNLI
mT8- Dosse+PRESENCE-Sequee
Cross-lingual zero-shot transfer
77.5
m
76.5 77.3
mTS5-base
61.60
49.09
69.98
61.01
45.84
68.78
Evalition
64.6
65.9
61.88
42.94
68.37
STTS-basC+PRESEXCEEE
60.94
45.61
67.62
56.8
52.8
PRESSEEC
61.68
46.47
70.12
62.5
Translate-train
mTS-base
65.45
52.75
76.76
MLQA
TyDi-QA
XNLI
115-basc+PRESENCCE
64.98
55.84
78.80
STTS-base+PRee PRESENRRE
64.88
58.10
79.06
Performance comparison of PRESENCE- Sequential and PRESENCE with mm55-base on translate- train versions of QA and sentence pair tasks. PRESENCE is comparable even better than PRESENCE-Sequental that uses sequential dataset filtering (Section 6.2).
Figure 4:
m PRESEPRREEN
64.65
55.06
78.96
mTS PRSS+PRE 66.32
56.90
79.48
Table 5: Effect of two-staged reweighting compared to only direcr(RESEECE-D) or inverse(PRESENCE: weighting and inverted two-staged reweighting (PRESENCE-I-D) over mT5-base. Bold shows the best result. Discussions Section 6.3.
effectiveness of PRESENCE, indicating that generat- ing SI scores at microbatch level offers smoother and more stable scoring approach as opposed to sample-level SI scoring. The average improvement with the PRESENCE framework more for the large mT5 variants. larger model (mT5-large) potentially generates more reliable SI scores when used for reweighting compared to the mT5- model. We hypothesize these two factors as the key reasons for the significant improvements observed for the PRESENCE approach, particularly for the lare model.
that our online adaptation of microbatch reweight- ing using SI scores is competitive for model pre- training relative to sequential offline dataset fil- tering. One possible reason might be that the online reweighting relies upon the most recent model weights for calculating influence of training samples, providing more suitable signals for data reweighting as compared the offline setting. Fur- ther, the joint online version forms an elegant and computationally efficient alternative to the sequen- tial offline approach, providing an opportunity for scaling reweighting to larger models and datasets.
We also make some interesting observations for zero-shot and translate-train dataset variants, observe
For both mT5-base and mT5-large, more significant gains for PRESENCE when the training data available in the target languages (translate-train). This indicates that reweighting might be beneficial for the model to adapt better across languages as compared to unweighted train- ing. For instance, we consider the English subset from XQuAD and XNLI for translate-train settings. We observe that PRESENCE improves performance significantly for mt5-large experiments (Table 4).
6.3 Impact of Two-staged Learning
We analyze the impact of using our two stage reweighting strategy by comparing its performance with models pre-trained purely with direct, i.c..
= 1 (PRESENCE-D) and inverse, i.e., 7 (PRESENCE-D) weighting. We train all the vari- ants for 200, 000 steps and compare their perfor- mance in Table 5, As shown, observe supe- rior performance of PRESENCE compared the other reweighting strategies. This supports our hypothesis that pre-training probably happens two parts: the model quickly learns new infor- mation in the first stage, after which all new in- formation seems redundant. The second stage is important to stabilize the pre-training. To further test this, perform reweighting in the reverse order, first performing inverse weighting and then direct weighting (PRESENCE-I-D), This strategy causes a degradation in the performance, as the inverse weighting initially may slow down training,
6.2 Comparison with PRESENCE-Sequentia
We compare PRESENCE with the multi-step PRES- ENCE-Sequent ACCE-Sequential explained in Section 3. For fair comparison, we pre-train (TTS-bas-PRREENNCT for 200, 000 steps randomly sampled data and present the results in Figure 4. We observe that even though PRESENCE does not look at the com- plete data at once and operates in joint online setting, it performs comparably, and in some cases, outperforms PRESENCE- PRESENCE-Sequential. This indicates

---

Crawl', publicly-availabel web archive that pro- vides "web extracted text". Raffel et al. (2020) use various heuristics such as retaining lines that end in terminal punctuations, retaining lines and pages based on a minimum number of words, etc to clean Common Crawl. Wenzek et al. (2020) use Kneser-Ney language model (Heafield, 2011) and calculate the perplexity over training data and high quality target domain to extract high qual- ity documents. Multilingual pre-training has also known to depend on language and domain distri- butions in the corpora (Conneau et al., 2020; Du et al., 2022; Hoffmann et al., 2022). Multilingual pre-training involves an additional step of boosting low-resource language data in the corpora using temperature sampling (Conneau et al., 2020; Ari- vazhagan et al., 2019; Xue et al., 2021). DoReMi (Xie et al., 2023) uses a smaller proxy model calculate domain weights of different sources com- prising the mixture of the pre-training data to pre- train larger models. These works either rely on expertly crafted heuristics or require training addi- tional models for dataset selection and filtering.
90
MLQA
TyDi QA
XNLI
Evaluation Metric
75
45
-2
Temperature (I)
Figure 5: Effect of temperature during reweighting on translate-train versions of QA and sentence pair tasks. Discussions in Section 6.4.
while the direct weighting in later stages leads to increased use of noisy samples. However, there are certain datasets where either purely direct or inverse weighting perform better than PRESENCE. We believe that self-influence scores develop corre- lations with multilingual data based on their quan- tity in the corpora, which may cause varied trends in downstream tasks.
Influence Functions and Training Data Attribu- tion Influence functions help to trace model's ss prediction through the learning algorithm and back to its training data, practice commonly known Training Data Attribution (TDA) (Guu al., 2023). Influence functions have been extensively used in deep learning as a means of model interpretability and explainability (Linardatos et al., 2020; Arri- eta ct al., 2020; Guidotti et al., 2018), adversarial learning (Yuan et al., 2019; Salman et al., 2020), federated learning (Kairouz et al., 2021; Geiping et al., 2020), and identifying outliers or mislabeled samples (Koh and Liang, 2017; Ych et al., 2018). 18). TracIn (Pruthi et al., 2020) introduces scalable and general first-order approximation to calculate gradient based influence, and extends the mini- batch level. Bejan et al. (2023) formulates an auto- mated curricular strategy using SI scores for data cleaning and filtering for NLP tasks. Influence functions have mostly been applied for supervised learning with ground truth labels for the data and have generally been explored in an offline setting.
6.4 Scaling Microbatch Gradient Weights
Since our method currently uses two discrete val- ues of temperature r in Equation 8, we probe its ef- fect on single-stage reweighting during model pre- training. These findings can be used to formulate a more granular and automated reweighting strategy using temperature scaling or continuous temper- ature scheduling, which we leave as future work. We pre-train models with 200, 000 steps and evaluate them the down- stream tasks in Figure We observe that increas- ing the magnitude of both positively and nega- tively affects the model performance. A possible reason might be that high positive 7 leads to large variance in the microbatch gradient weights, lead- ing to unstable training, whereas high negative results in much slower convergence compared to the baselines.
(2 1,,1,22) for
7 Related Work
Data selection and online adaptation in super- vised learning Selection functions for supervised learning often leverage training dynamics such as high loss (Jiang et al., 2019; Kawaguchi and
Datasets for Pre-training LMs Language mod- els are generally pre-trained on large-scale corpora scraped from the web (Devlin et al., 2019; Liu et al., 2019; Baevski et al., 2019). The most common source obtaining large-scale data Common
htttpss/ccommonrawl.or/////aaa

---

Lu, 2020) or high prediction uncertainty (Coleman al., 2020) to select "hard" points. Swayamdipta et al. (2020) use the change loss over the course of training rather than each step to also eliminate noisy samples. Removing noisy training samples using offline methods is another direction for se- lecting training data for supervised learning (Chen et al., 2019; Pleiss et al., 2020). Paul et al. (2021) norm of the gradient or self-influence to iden- tify important samples early in the training to heav- ily prune datasets. RHO-loss (Mindermann al., 2022) calculates a heldout loss using a proxy model and uses a combination of model loss and heldout loss to select non-noisy, non-redundant, and task- relevant samples. Ahn et al. (2023) uses per-sample gradient norm to assign importance probabilities, and trains biased model to formulate debiased model training strategy. Ren et al. (2018) and (Jain and Shenoy, 2022) use meta-learning for reweight- ing training samples within a batch for increas- ing robustness and selective prediction respectively. These works operate in supervised settings, requir- ing controlled validation sets or proxy models and adapting them to pre-training is non-trivial.
parameters, particularly for the two-staged learn- ing. Calculation of self-influence score is only done using the first layers of the encoder and de- coder for computational optimization, however, us- ing more layers might lead to more representative weighting information. Even though we believe the training overhead of PRESENCE is significantly lesser compared to the overhead of existing meth- ods such sequential offline filtering, our imple- mentation on microbatches requires a training time higher by 30% compared training the models on randomly sampled data without any reweighting Since the gradients across microbatches are inde- pendent, there can be ways to parallelize the com- putation. Our two stage training strategy currently switches at training step which is chosen based on the total training steps, looking at the loss curve, and following how warm-up steps for learning rate schedules are decided for LM pre-training, which is ad-hoc. This can formalized based on training loss, microbatch self-influence scores, or anchored to the dataset itself, and may lead to more suitable sample reweighting using temperature scaling.
A Acknowledgements
8 Conclusion and Future Work
SC is supported by the Canada CIFAR AI Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant. The authors would like to thank Shubham Mittal for his assistance in compiling and analyz- ing the language- wise and multilingual results, and Dheeraj Rajagopal, Lucas Dixon, Pradeep Shenoy, and Kelvin Guu for insightful discussions on the work. The authors are also grateful to the reviewers and the area chairs for their helpful reviews and discussions.
We introduce PRESENCE method for jointly reweighting samples using self-influence (SI) scores and pre-training. We conduct an in-depth analysis of the relationship between SI scores and sample quality from a pre-training perspective and use them as a filtering objective for pre-training data selection. As sequential filtering is expensive at the scale of pre-training, we formulate PRES- ENCF as joint adaptation for sample reweighting, PRESENCE outperforms baselines trained on ran- domly sampled and Sl-filtered data on 5 datasets across tasks. We believe that PRESENCE is an important first step in the research direction of data sample weighting for pre- treiiran
References
Sumyeong Abn, Seongyoon Kim, and Se- Young Yun. 2023. Mitigating dataset bias by using per- per-sample gradient. In The Fleventh International Conference on Learning Representations.
As future work, we plan to explore relationships between samples in the pre-training corpora and influence functions across languages, data sources, and domains. We also plan to formulate automated reweighting strategies using temperature scaling schedules based on the training step, training loss, and sample influence scores.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. CORR, abs/1907.05019
Limitations
Alejandro Barredo Arrieta, Natalia Dodz-Rodr,,,ez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Garefa, Sergio Gil-López,
As a pre-training strategy, PRESENCE is computa- tionally expensive for finding the optimal hyper-

---

Daniel Molina, Richard Benjamins, et al. 2020. Ex- plainable artificial intelligence (xai): Concepts, tax- onomies, opportunities and challenges toward respon- sible ai. Information fusion, 58:82-115.
Wu, Zhifeng Chen, and Claire Cui. 2022. GLaM: Efficient scaling of language models with mixture- of-experts. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli 2019. Cloze-driven pretraining of self-attention networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP)), pages 3360-5369, Hong Kong, China. Association for Computational Linguis- tics.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting gradients-how how easy is it to break privacy in federated learning? Ad- vances in Neural Information Processing Systems. 33:16937-16947.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 42.
Irina Bejan, Artem Sokolov, and Katja Filippova. 2023, Make every example count: On stability and utility of self-influence for learning from noisy nip datasets.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. 2019. Understanding and utilizing deep neural networks trained with noisy labels. In In- lernational Conference Machine Learning, pages 1062-1070.
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. 2023. Simfluence: Modeling the influence of individual training exam- ples by simulating training runs.
Kenneth Hcafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 197, Edinburgh, Scotland. Association for Com- putational Linguistics.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2020. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representa- tions.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan. Eric Noland, Katherine Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals. Jack William Rae, and Laurent Sifre. 2022. An empirical analysis of compute- optimal large language model training. In Advances in Neural Information Processing Sys- tems.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Nishant Jain and Pradeep Shenoy. 2022. Selective clas- sification using a robust meta- meaa-nc approach.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arxiv preprint ar Xiv:23303.333 03378.
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton. et al. 2019. Accelerating deep learning by focusing the biggest losers. arXiv preprint arXiv:1970.000 1970.0072..
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin. Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu. Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pel- lat, Kevin Robinson, Kathleen Hellser,, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cor- mode, Rachel Cummings, et al. 2021. Advances and

---

open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2)11-22
Geoff Pleiss, Tiany Zhang, Ethan Elenberg, and Kil- ian Q Weinberger. 2020. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044 17056.
Kenji Kawaguchi and Haihao Lu. 2020. Ordered sgd: new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statisties, pages 669-679. PMLR.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influ- ence by tracing gradient descent. Advances in Neural Information Processing Systems, 1e1111992-99999
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 5551.
PMLR.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E, Hin- ton. 2017. Imagenet classification with deep convolu- tional neural networks. Commun, ACM, 60(6):84
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In ICML.
Sameer Kumar, Victor Bitorff, Dehao Chen, Chiachen Chou, Blake Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang, Tao Wang. Yuanzhong Xu, and Zongwei Zhou. 2019. Scale mlperf-0.6 models on google ipu-v3 pods.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury. Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jamnis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garreute, James Lee- Tee-Th Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Maitin-Shepard. Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint 2rX6: 14858
Pantelis Linardatos, Vasilis Papastefanopoulos. and Sotiris Kotsiantis. 2020. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18.
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2022. Same pre- treitrain loss, better downstream: Implicit bias matters for language models.
Hadi Salman, Andrew llyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. 2020. Do adversar- ially robust imagenct models transfer better? Ad- vances in Neural Information Processing Systems, 33:3533-3545.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1662.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lu- cile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, al. 2022. What language model to train if you have one million gpu hours? arXiv preprint 21Xiv:2210.15424
Paul-Aymeric Martin McRae, Prasanna Parthasarathi, Mido Assran, and Sarath Chandar. 2022. Memory augmented optimizers for deep learning. In Interna- tional Conference on Learning Representations.
Sören Mindermann, Jan M Brauner, Muhammed Raz- zak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. 2022. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15600, PMLR.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, Association for Computa- tional Linguistics.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. Deep learning on data diet: Find- ing important examples early in training. In Ad- vances in Newral Information Processing Systems.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In Advances in Neural Information

---

Processing Systems, volume 30. Curran Associates, Inc.
Launingg Ratt
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Iilia Polosukhin. 2017b. Attention is all you need. Advances in neural information processsiinro processing systems, 30.
warmuu
Training Step
Guillaume Wenzek, Maric-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
stops
Figure 6: The learning rate scheduler as described in the Transformers (Vaswani al., 2017b) paper. The learning rate first increases linearly for warmup steps and then decreases exponentially.
square root of the step number. We contrast these types of learning rate schedulers with our two stage reweighting strategy. Increasing the learning rate for a given number of steps warms up the model more by boosting the gradients, and thereafter a de- cay is used to enable the model to reach a minima better. Intuitively, we also aim to achieve similar learning dynamics using our two stage learning: in the first stage of learning, data samples with higher SI scores are emphasized more to drive more learn- ing, while in the subsequent second stage, the data samples with higher SI scores are de-emphasized to limit the impact of noisy and unreliable samples while giving more weight to better quality sam- ples and for more stable training. We believe that as future work, we can use temperature scaling schedulers inspired from learning rate schedulers to automate reweighting curricula.
Sang Michael Xie. Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Lc, Tengyu Ma. and Adams Wei Yu. 2023. Doremi: Optimizing data mixtures speeds up language model pretraining.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Cheng Yang. Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. 2021. Progres- sively stacking 2.0: multi-stage layerwise training method for (bert) training speedup.
Chin-Kuan Yeh, Joon Kim, lan En-Hsu Yen, and Pradeep K Ravikumar. 2018. Representer point selec- tion for explaining deep neural networks. Advances in neural information processing systems, 31.
B Infrastructure
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin L.i. 2019. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions neural networks and learning systems, 30K9):2805-288
We use seqio and T5X (Roberts et al., 2022) to train our models. We use 64 TPU (Kumar et al., 2019) chips for pre-training all the models and use 8 TPU chips for fine-tuning the base variant and 16 TPU chips for fine-tuning the large variant.
A Two-staged Reweighting and Learning Rate Schedulers
C Maturity of Models and SI Scores
We create an analogy between our two stage reweighting and the transformer learning rate scheduler (Vaswani et al., 2017a). The learning rate Ir at step step for a model with input and output dimensionality dmodel and warm-up steps warmau is given by,
Self-influence (SI) scores are calculated using the model gradients for given objective. Their relia- bility, thus depends on the maturity of the model, i.e. how well the model is trained, which is being used to calculate them. Since SI scores are gen- erally used for relative analyses, the models need not be trained till convergence. This characteristic relationship between the model's ability predict correct labels and the reliability of SI scores be- comes an important consideration when adapting SI scores for online adaptations. We have observed
Ir = d mein(step 0.5
(13)
This corresponds to increasing the learning rate linearly for the first warmup training steps, and de- creasing it thereafter proportionally to the inverse

---

that models trained for about 20% of training steps give decently reliable SI scores, however, better strategies such as choosing checkpoints where the loss decreases the most and averaging scores of multiple checkpoints have been proposed related works (Pruthi et al., 2020). We believe that the direct weighting stage of PRESENCE that drives more learning also acts added warmup for the model's ability predict noisy samples for the subsequent inverse weighting stage, enabling it to stabilize the training further. We leave the deeper analyses of scores on training samples early in the pre-training and in later stages as future work.


Reviews:
[{"review_id": "dNEz1YyPxD", "reviewer": "Reviewer_8mAv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude to the reviewers for their thoughtful interaction with our work and for their valuable and positive feedback. We're heartened by their recognition of the importance of the research problem (YoT7), easy to follow and understand (YoT7, nK5X), and promising results (8mAv, YoT7). We are also encouraged that our work was described as novel (nK5X), practicable (YoT7), and impactful (8mAv) with comprehensive experiments (YoT7). We've considered all the concerns that the reviewers have raised, and we've worked to address these issues as discussed below.\n\n**Reviewer's Comment: The experimentation is limited to T5 based models, it will useful to carry out similar experimentation on other LMs.**\n\n**Response:** Thank you for the comments. We used T5 as it is one of the most widely used LMs and it is also publicly available, making development over it easy. We thus decided to experiment with different variants of T5 rather than different LMs as they are architecturally similar (transformer-based).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores the area of training on high quality samples, which helps with improved performance on downstream tasks. Various sizes of the model were used for experimentation to demonstrate impact of the quality framework on the models.", "reasons_to_reject": "The experimentation is limited to T5 based models, it will useful to carry out similar experimentation on other LMs", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper a framework to filter high quality data samples during pre-training process is propossed, this method uses SI score to choose the best samples in the dataset. The experiments were done on various model sizes of T5 and MC4 was used for pre-training. Results were reported for downstream tasks from XTREME benchmark.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xy7l8N861Z", "reviewer": "Reviewer_YoT7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude to the reviewers for their thoughtful interaction with our work and for their valuable and positive feedback. We're heartened by their recognition of the importance of the research problem (YoT7), easy to follow and understand (YoT7, nK5X), and promising results (8mAv, YoT7). We are also encouraged that our work was described as novel (nK5X), practicable (YoT7), and impactful (8mAv) with comprehensive experiments (YoT7). We've considered all the concerns that the reviewers have raised, and we've worked to address these issues as discussed below.\n\n**Reviewer's Comment: The source code has yet to be made available in the reviewed version, preventing me from verifying the effectiveness of this method through the code. I am also uncertain whether the code will be made public after accepting the paper.**\n\n**Response:** Thank you for the suggestion. We would first like to highlight that implementation of PRESENCE is straightforward. The only change required in the training loop of any LM is very similar to the pseudocode following Algorithm 2 on page 6. However, we will also release the code after the acceptance. To give some background, following Algorithm 2, the core implementation involves adding about 15 lines to the trainer.py file of the t5x codebase. Thanks again for this.\n\n\n**Reviewer's Comment: The method's current process is undeniably intricate, entailing the optimization of multiple pipelines. These pipelines encompass sophisticated components, such as a two-phase learning strategy for online adaptation and a two-stage offline filtering procedure. By embracing this sophisticated methodology, we doubt its potential availability in real-world applications.**\n\n**Response:** Thank you for the comment. We would like to point out that PRESENCE (our main proposal) is a single-stage method and not a combination of offline and online stages. This only requires the optimization of the temperature hyperparameter, and we use all the other settings exactly the same as the baseline. We will clarify this more in the camera-ready version to prevent any confusion. \n\nAlso, the two-phase learning strategy is inspired from the loss curve of the baseline model and we present it as an initial starting point. However, as suggested by the reviewer, we discuss this ad-hoc implementation in the limitations and have proposed using temperature schedulers as a possible future work.\n\n\n**Reviewer's Comment: (minor) More than involving three tasks and five datasets is required to verify the effectiveness of this method. I suggest expanding to more General Language Understanding Evaluation (GLUE) tasks.**\n\n**Response:** We followed this experimental setting to replicate the evaluation setup of the mt5 paper and hence chose these datasets and tasks. We believe that this evaluation is extensive considering the brevity of the paper, as it covers varied tasks such as question answering and NER along with NLI and is multilingual in nature. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Data reweighting is an important research problem for the pre-training language models, especially in today's emergence of large LM models.\n2. The experimental study is comprehensive, and the results are promising.\n3. The SI score is used to filter out noisy samples for pre-training dataset selection is practicable.", "reasons_to_reject": "1. The source code has yet to be made available in the reviewed version, preventing me from verifying the effectiveness of this method through the code. I am also uncertain whether the code will be made public after accepting the paper.\n\n2. The method's current process is undeniably intricate, entailing the optimization of multiple pipelines. These pipelines encompass sophisticated components, such as a two-phase learning strategy for online adaptation and a two-stage offline filtering procedure. By embracing this sophisticated methodology, we doubt its  potential  availability in real-world applications. \n\n3. (minor)  More than involving three tasks and five datasets is required to verify the effectiveness of this method. I suggest expanding to more General Language Understanding Evaluation (GLUE) tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a novel approach to data reweighting in task-specific supervised learning and LM fine-tuning. The model-driven reweighting seems essential in the research of pre-training language models. Methods of Filtering Pre-training Data using Self-Influence and Sample Reweighting using Self-influence are relatively simple, and it appears to be effective from the experimental results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The article is easy to read, and the method introduction is easy to understand."}, "extra_text": {}}, {"review_id": "tbtyFtVBhA", "reviewer": "Reviewer_nK5X", "summary": "", "questions": "- A. Is there two-stage reweighting in Section 6.4? If no, is this experiment a reasonable ablation?\n- B. Why is the performance of mT5*-large significantly worse than the performance of mT5 base for Translate-train for XQuAD, MLQA, XNLI, and WikiAnn NER in Table 3? The numbers are also significantly lower than the results from the original mT5 paper. Is there anything wrong during the pre-training in this paper's implementation?\n- C. Why do we need to pre-train another model in PRESENCE-offline, instead of continually pre-training the model used for data filtering?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude to the reviewers for their thoughtful interaction with our work and for their valuable and positive feedback. We're heartened by their recognition of the importance of the research problem (YoT7), easy to follow and understand (YoT7, nK5X), and promising results (8mAv, YoT7). We are also encouraged that our work was described as novel (nK5X), practicable (YoT7), and impactful (8mAv) with comprehensive experiments (YoT7). We've considered all the concerns that the reviewers have raised, and we've worked to address these issues as discussed below.\n\n**Reviewer's Comment: Lack of comparison to other pre-training data filtering methods**\n\n**Response:** We propose PRESENCE as an online reweighting strategy for pre-training LMs rather than a data filtration method. Please note that in the experiments in the paper, PRESENCE is applied on the corpora after the raw corpus has been filtered using numerous filtration heuristics, as described in the mT5 paper. We observe that PRESENCE is able to achieve improved performance even in this pre-filtered setting. Thus, PRESENCE is complementary to existing data filtering methods and can be used in conjunction with them.\n\nGiven this difference in objectives, we feel comparison with data filtering methods is unnecessary. We shall clarify this point in the final version of the paper.\n\n\n**Reviewer's Comment: Significant pre-training computation overhead**\n\n**Response:** We point out that PRESENCE adds a 30% computation overhead (compared to the base model) compared to 100% overhead in PRESENCE-offline filtering. Moreover, in spite of this lower overhead, PRESENCE results in significant performance gains compared to other methods. In summary, PRESENCE gives more improvement while incurring lower overhead. \n\nAs pointed out by the reviewer, we can have offline filtering done once for different models, but considering model-driven methods, this filtration is training dependent. Our focus in the paper is on improving performance through reweighting of a single model training, rather than amortizing the computational overhead of offline data filtering across different pretraining runs. Moreover, through our experiments, we have demonstrated the benefit of online reweighting, compared to multi-stage offline data filtering. Thank you!\n\n\n**Reviewer's Comment: Some experiment results cannot convince me that PRESENCE is good enough.**\n\n**Response:** Our goal in the paper is to present a unified framework where direct and inverse weighting can be controlled using a single parameter in the online reweighting setting. Our results do indicate that online reweighting one way or the other or a combination thereof is usually better than no weighting at all, and we establish this through extensive experiments in the paper.\n\nThrough the results in Table 4, especially in the translate-train setting, we observe that PRESENCE used in combination with the two weighting phases -- Direct followed by Inverse -- results in significantly improved performance much more consistently across all datasets and tasks, as compared to just direct or inverse weighting. Since we only apply PRESENCE to pre-training, we use this to show the effectiveness of two-stage training: it is consistent as compared to just direct or inverse reweighting. Thanks!\n\n**Reviewer's Comment: It is unclear why this paper selects multilingual transferability as the downstream task for evaluation. In the mT5 paper, they also show the results on SQuAD. I wonder if PRESENCE only works for multilingual PLMs and cross-lingual transfer downstream tasks.**\n\n**Response:** PRESENCE is a general online reweighting strategy, and is not restricted to multilingual or monolingual settings. Since we used mT5 as our base model, we thought a multilingual setup would be more appropriate, as well as more representative and challenging. \n\nHaving said that, please note that in our experiments we use English-only finetuning, and many of the evaluation tasks also include English as one of the target languages. We shall include this language-wise performance breakup in the final version of the paper.\n\nWe hope the reviewer will recognize this multilingual evaluation as more desirable and informative rather than a limitation.\n\n**Questions**\n\n**Question: Is there two-stage reweighting in Section 6.4? If no, is this experiment a reasonable ablation?**\n\n**Response:** There is no two-stage training here as we are presenting the dependency of the temperature on the weighting. The aim of this experiment was to present a platform for future work where we discuss the possibility of automating temperature scheduling and give some initial insights into the findings. We will make this explicit.\n\n\n**Question: Why is the performance of mT5-large significantly worse than the performance of mT5 base for Translate-train for XQuAD, MLQA, XNLI, and WikiAnn NER in Table 3? The numbers are also significantly lower than the results from the original mT5 paper. Is there anything wrong during the pre-training in this paper's implementation?**\n\n**Response:** As mentioned in section 5.2, line 355, we use a batch size of 512 instead of 1024 used in the original paper due to compute reasons as pre-training is expensive, and hence the results are worse than mt5-base as well (which effectively sees double the amount of data). We will mention this in the caption of Table 3 as well to prevent any confusion for the readers. Thanks.\n\n\n**Question: Why do we need to pre-train another model in PRESENCE-offline, instead of continually pre-training the model used for data filtering?**\n\n**Response:** Thanks for the suggestion, this experiment sounds interesting. We present PRESENCE-offline not solely as a data filtering method, but rather as a step to strengthen our motivation for our online adaptation. Thus, PRESENCE-offline is not the primary method contribution and we did not perform various ablations. However, to elaborate, in PRESENCE-offline, the base model for filtering is trained on a random subset of the data. Continued pre-training of the same model on the same data will lead to weird inconsistencies as the model would have seen some data once before filtering and subsequently again after filtering. In order to avoid such inconsistencies, we train the model in the second phase from scratch. If we ensure that the dataset used to train the filtration model is completely separate from the data being filtered, we can try using it in a continued pre-training setting as well.\n\n\n**Missing References:** Thank you, this reference is relevant. We will add a discussion about this paper in the related work.\n\n**Typos Grammar Style And Presentation Improvements:** We agree with the suggestions and will incorporate the changes. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Using gradient norm/self-influence for weighting the samples during pre-training is a novel idea for NLP\n- The paper is generally well-written and easy to follow\n", "reasons_to_reject": "- Lack of comparison to other pre-training data filtering methods.\n- Significant pre-training computation overhead.\n     - Detailed explanations for the above two reasons to reject: PRESENCE is essentially a data filtering method, so I expect to see the comparison with some baselines that filter the pre-training data. **The current paper only shows that PRESENCE works, but it does not show it is better than other methods**. As the authors admitted in the Limitation, PRESENCE induces 30% pre-training computation overhead. While the paper says that \"`we believe the training overhead of PRESENCE is significantly lesser compared to the overhead of existing methods such as offline filtering`\", this is not justified by any experiments and numbers. Moreover, offline filtering methods can be done once, and the filtered datasets can be used to train any models, while PRESENCE (the online version) needs to be applied every time when pre-training a model. This makes PRESENCE less unlikely to be applied in reality. \n- Some experiment results cannot convince me that PRESENCE is good enough. The relationship between temperature $\\tau$ and the downstream performance shown in Figure 5 does not seem to justify the two-stage pre-training. It seems that using a positive or negative $\\tau$ can yield improvement over no reweighting. This makes me curious if it is really important to reweight using a positive or negative $\\tau$. The results from Table 4 (cross-lingual zero-shot transfer) also do not show the advantage of using PRESENCE. \n- It is unclear why this paper selects multilingual transferability as the downstream task for evaluation. In the mT5 paper, they also show the results on SQuAD. I wonder if PRESENCE only works for multilingual PLMs and cross-lingual transfer downstream tasks.\n", "questions_to_authors": "- A. Is there two-stage reweighting in Section 6.4? If no, is this experiment a reasonable ablation?\n- B. Why is the performance of mT5*-large significantly worse than the performance of mT5 base for Translate-train for XQuAD, MLQA, XNLI, and WikiAnn NER in Table 3? The numbers are also significantly lower than the results from the original mT5 paper. Is there anything wrong during the pre-training in this paper's implementation?\n- C. Why do we need to pre-train another model in PRESENCE-offline, instead of continually pre-training the model used for data filtering?\n", "paper_topic_and_main_contributions": "Paper topic\n===\nThis paper explores the possibility of using self-influence to reweight the importance of each sample during pre-training. \nThey propose PRESENCE, which uses the gradient norm (equivalent to self-influence) to indicate a sample's importance and reweight the sample's gradient during backpropagation using the gradient norm. They show that the gradient norm is related to the noisiness of the training sample. By devising a two-stage reweighting schedule, they show that the mT5 pre-trained using PRESENCE is better than the model not pre-trained using PRESENCE regarding the performance on downstream cross-lingual transferring tasks.\n\nMain Contribution\n===\nNLP engineering experiment", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Using gradient norms to sample or reweight is not a novel idea. The following paper uses the gradient norm for sampling, while the current paper uses the gradient norm for weighting the gradient. I think the two ideas are very similar, so it is worth mentioning.\n\n@misc{ahn2023mitigating,\n      title={Mitigating Dataset Bias by Using Per-sample Gradient}, \n      author={Sumyeong Ahn and Seongyoon Kim and Se-young Yun},\n      year={2023},\n      eprint={2205.15704},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}", "Typos_Grammar_Style_and_Presentation_Improvements": "- I disagree with the wording in Line 283: `High SI scores result from high gradient norms`. The term \"result from\" is odd, since the SI used in the paper is simply gradient norm, so there is no causal relation between SI and gradient norm.\n- PRESENCE-offline is a two-stage process. But this is not clearly specified before Line 242. This should be mentioned earlier in Section 3."}}]

Author rebuttal:

Meta-review: This paper investigates method for model-driven reweighting of data for LLM pre-training, and propose a two phased learning method that leverages self-influence (SI) scores as an indicator of sample importance. This paper addressed an important research problem for large language models, and proposed a novel idea with good results on using gradient norm for weighting as comments by the reviewers. On the down side, given that this paper proposed a new method for LLM pretraining with data reweighting, it can be made stronger by experimenting on other LMs in addition to the T5 based models, and running evaluations on tasks that are beyond multilingual evaluations as commented by the reviewers. The computation overhead introduced by the proposed method may limit its usage in real-world large scaling training setting.
Decision: Accept-Main