INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models
Authors: H S V N S Kowndinya Renduchintala
Krishnateja Killamsetty
Sumit Bhatia
Milan Aggarwal
Ganesh Ramakrishnan
Rishabh K Iyer
Balaji Krishnamurthy
Venue: EMNLP 2023 Findings (2023)
Keywords: Efficient training
Language Models
Data selection
TL;DR: Selecting highly informative data subsets helps speedup training of language models.

Abstract: A salient characteristic of pre-trained language models (PTLMs) is a remarkable improvement in their generalization capability and emergence of new capabilities with increasing model capacity and pre-training dataset size. Consequently, we are witnessing the development of enormous models pushing the state-of-the-art. It is, however, imperative to realize that this inevitably leads to prohibitively long training times, extortionate computing costs, and a detrimental environmental impact. Significant efforts are underway to make PTLM training more efficient through innovations in model architectures, training pipelines, and loss function design, with scant attention being paid to optimizing the utility of training data. The key question that we ask is whether it is possible to train PTLMs by employing only highly informative subsets of the training data while maintaining downstream performance? Building upon the recent progress in informative data subset selection, we show how we can employ submodular optimization to select highly representative subsets of the training corpora and demonstrate that the proposed framework can be applied to efficiently train multiple PTLMs (BERT, BioBERT, GPT-2) using only a fraction of data. Further, we perform a rigorous empirical evaluation to show that the resulting models achieve up to $\sim99\%$  of the performance of the fully-trained models. We made our framework publicly available at \url{https://github.com/Efficient-AI/ingenious}.

INGENIOUS: Using Informative Data Subsets for Efficient Pre- Pre-Training of Language Models
HS S Kowndinya Renduchintala ; Krishnateja Killamsetty?, Sumit Bhatia
Milan Aggarwal Ganesh Ramakrishnan', Rishabh Iyer", Balaji Krishnamurthy"
1 Indian Institute of Technology Bombay, India
2 University of Texas at Dallas, USA
Media and Data Science Research (MDSR) Lab, Adobe Inc., India
Abstract
however, accompanied by significant costs. For instance, costs an estimated $12 million train GPT-3 (Brown et al., 2020) with roughly 1.2 million pounds of CO2 emissions (Kahn, 2021). Megatron-TTuring NLG (Smith et al., 2022) is a 530 billion parameter PTLM, which is thrice the size of GPT-3 and is trained on 4480 A100 GPUs and yields close 1% performance improvements over GPT-3. By continually increasing the size of PTLMs and pre- pre-training corpora to improve gen- eralization ability, significant additional resources and energy are consumed, resulting dire envi- ronmental consequences (Sharir et al., 2020). Fur- ther, such large-scale resource utilization and the costs associated with PTLMS create an uneven playing field for small organizations and universi- ties, which operate with significant resource con- straints. Hence, crucial step towards develop- ing responsible, fair, and GreenAl (Schwartz et al., 2020) involves minimizing inefficiencies and costs of training these models.
salient characteristic pre-trained language models (PTLMs) is remarkable improve- ment in their generalization capability and emergence of new capabilities with increasing model capacity and pre-training dataset size. Consequently, we are witnessing the develop- ment of enormous models pushing the state- of-the-art. It is, however, imperative to realize that this inevitably leads to prohibitively long training times, extortionate computing costs, and a detrimental environmental impact. Sig- nificant efforts are underway to make PTLM training more efficient through innovations in model architectures, training pipelines, and loss function design, with scant attention being paid to optimizing the utility of training data. The key question that we ask is whether is possible train PTLMs by employing only highly informative subsets of the training data while maintaining downstream performance? Building upon the recent progress in informa- tive data subset selection, we show how we can employ submodular optimization to se- lect highly representative subsets of the train- ing corpora and demonstrate that the proposed framework be applied to efficiently train multiple PTLMs (BERT, BioBERT, GPT-2) using only fraction of data. Further, we per- form rigorous empirical evaluation show that the resulting models achieve up to ~ 99% the performance of fully-trained models. We made framework publicly available at wwwpiituuuboo/ffffffiiieee AiiitttAA/iinnuuus...
Significant efforts toward improving the efficiency of PTLMs have ventured in directions such as op- timizing the model architecture (Chen et al., 2020; Gordon et al., 2020; Zafrir et al., 2021), 1), modifi- cations to the training pipeline (Izsak et al., 2021; Shen al., 2022) and task (Schick and Schütze, 2021), sample efficient masking techniques for improved convergence (Bitton al., 2021) and leveraging contextual knowledge to reduce model size (Kaur et al., 2022). In this work, driven by the observation that the scale of the pre-training cor- pus contributes significantly to the training costs of PTLMs, we explore the feasibility of training PTLMs using highly informative subsets of the corpus. Recent studies have demonstrated the fea- sibility of informative data subset selection for efficient deep model training for images (Mirza- soleiman et al., 2020; Killamsetty et al., 202 2002labbr b,c; Pooladzandi et al., 2022) in both supervised and
1 Introduction
Pre-trained language models (PTLMs) (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2020; Brown et al., 2020; Raffel et al., 2020) have rev- olutionized the field of natural language process- ing (NLP), becoming the default choice for wide array of NLP tasks. The versatility of PTLMs,
"Work done during internship at Media and Data Science Rescarch (MDSR) Lab, Adohe Inc.

---

semi-supervised settings. In light of this, the key question we attempt answer is: Can we effi- ciently pre-train language models using highly in- formative subsets of the training corpus without compromising performance?
Cost Savings
Perfommance Degrodaticc
7.67x
The first step in answering the above question is identifying informative (or representative) sub- sets of the underlying training corpus such that they maximize the representation of the remain- ing samples in the corpus. Intuitively, given a set of sentences, the subsequent addition of sentences similar to existing sentences in the set yields di- minishing returns. More information gains can be achieved by adding diverse, dissimilar sen- tences. While the classical subset selection prob- lem is NP-hard, we can leverage the diminish- ing gains property of submodular functions (Fu- jishige, 2005) and frame subset selection sub- modular maximization problem. Several recent works (Wei et al., 2015; Mirzasoleiman al., 2020; Kothawade et al., 2021; Karanam al., 2022: Maheshwari et al., 2020) have formulated the subset selection problem as that maximizing a submodular objective. However, applying exist- ing subset selection frameworks to PTLMs is non- trivial given the scale of corpora typically used for pre-training (e.g., Wikipedia and Common Crawl consisting of hundreds of millions of sequences and billions of tokens). Most of the existing meth- ods rely on per-sample gradients, which are expen- sive to compute, and to the best of our knowledge, none of the previous works have considered subset selection for such large datasets.
-2-
-1.10%
-0.7%
-2.1%
-1
-2.1%%
INCENIOUS
INGENIOUS
INCENIOUS
INCENIOUS (400K)
(100K) (200)))
(SISK)
PreTriining, Steps
Cost-savings VS Performance tradeoff achieved by INGENIOUS for BERT pre-training: We contrast the accuracy degradation with cost savings compared to the vanilla BERT pre-training on entire dataset. We observe 4.35x cost-savings with 2. 1% ac- curacy drop and 2.33x cost-savings with 0.67% accu- racy drop.
Figure 1:
full dataset. Figure summarizes the cost-savings vs performance trade-off achieved by INGENIQUS for BERT pre-training. We also present thor- ough ablation studies revealing the impact of var- ious design choices and parameters involved. We also evaluate the models trained by INGENTOUS in terms of their knowledge retention capabilities and show how INGENIOUS can be used accelerate pre-training of domain-specifie language models such as BioBERT (Section 4.4). Finally, we dis- cuss the inferences that could be drawn from our work, limitations of our proposed framework and lay out directions for further improvement (Sec- tion 5).
Related Work
Our contributions: We propose the informative data subset selection task for efficient pre-training of PTLMs and present INGENIOUS, framework for subset selection using submodular optimiza- tion (Section 3). We show how to overcome the scalability challenge for typical large-scale pre- training corpora and employ scalable sentence fea- ture encoders to obtain individual data sample fea- tures relevant for subset selection. We also employ various engineering techniques to scalably select subsets from large-scale datasets (Section 3). We use INGENIOUS to pre-train BERT and GPT-2 and evaluate the performance of the resulting mod- els on downstream tasks (Section 4). A rigor- ous empirical evaluation reveals that the models pre-trained with INGENIOUS retain upto x 99% performance the models pre-trained using the
Knowledge distillation and pruning based meth- ods (Sanh et al., 2019; Jiao et al., 2020; Muhamed et al., 2021) 1) pre-train smaller variant of PTLMs (such as BERT) with lesser capacity using the full model as teacher network. Even though lighter versions such as DistilBERT (Sanh et al., 2019) retain z 97% of the performance with up to 60% faster inference, the PTLM still needs to be completely pre-trained initially to be able to distill the lighter version. Thus, the efficiency gains restricted only to the fine-tuning and inference. Other methods prune the architecture through forcing the weights with lesser magni- tude zero value during pre-training (Chen et al., 2020; Gordon et al., 2020) as well as during finc- tuning (Zafrir et al., 2021).

---

Model architecture and training task optimiza- tions: Schick and Schütze (2021) have shown that smaller PTLMs can achieve better performance by formulating the task input in cloze style. Izsak et al. (2021) proposed to optimize BERT pre- training through multiple optimizations related to data, model size, and optimizer choice. Shen et al. (2022) proposed a staged training mecha- nism where they start with training relatively smaller model, which is then used for initializing the full capacity model at a later stage. Yao et al. (2022) identify relevant samples from the pre- training corpus based on their similarity with the task-specific dataset to train task-specific PTLM followed by fine-tuning, thus inherently suffering from the limitation of pre-training separate models for every downstream task.
tion to formulate the problem, followed by details of different steps involved in the framework.
3.1 Notation
We denote the unlabeled dataset for pre-training by U = { トー= consisting of data points each corresponding to a varying length of sequence of symbols {s: [=1 (these symbols could be words character sequences such sub-words). Let S U4 be the subset of the unlabeled dataset on which the language model trained. Let the language model be parameterized by 0. We subscript the changing variables such as model parameters 0, subset S with the timestep / to denote their specific values at that timestep.
3.2 Problem Formulation
In its most general form, subset selection is de- fined as
Curriculum learning based methods employ the sequence length of training samples as proxy for hardness.Typically, shorter (casier) sequences are presented in the initial stages of pre-training followed by longer (harder) sequences at later stages (Nagatsuka et al., 2021; Li et al., 2022). However, such methods have been shown to per- form well only in limited configurations with spect the choice of language models, stage of pre-training, etc..
S: - arg max f(S)
(I)
SCU
where the subset St C U at step / is selected such that it maximizes the function f.
While the above general subset selection problem is NP-Hard, the problem becomes approximable in case the function f is submodular in nature (Fu- jishige, 2005). A set function f 24 - IR is submodular if for æ E U, ((A f(A f(A) 2 S(B U x) S(B), AA C B C u and & B. We pose the data subset selection problem submodular maximization problem since it al- lows for easier optimization by employing differ- ent approximations (Nemhauser et al., 1978; Tyer and Bilmes, 2019). In order to choose a suit- able submodular function, one must understand the characteristics of the subsets that are crucial for the end-goal efficient learning in our case. Previous works in computer vision have demon- strated that commonly used vision datasets con- tain many redundancies, and eliminating such re- dundant data samples does not affect the model's s performance (Birodkar et al., 2019; Toneva et al., 2019; Paul et al., 2021; Sorscher et al., 2022). Fur- ther, one can achieve faster model training by us- ing highly informative and representative data sub- sets (Kaushal et al., 2019; Mirzasoleiman et al., 2020; Sorscher et al., 2022). Please refer to Ap- pendix B for more related work on submodularity based subset selection. Building upon the learn- ings from computer vision research, our primary requirement for the selected subset is that should faithfully represent the training data and have min-
Hardware optimizations for PTLM Training: The suite of Open Pre-Trained Transformers (OPT) (Zhang et al., 2022) require 1/7th of the carbon footprint for pre-training when compared to popular PTLMs such as GPT-3 (Brown et al., 2020) while achieving comparable few-shot gen- eralization. OPTs leverage extensive data and ten- sor parallelism with high-memory GPUs (support- ing large batch sizes), which are usually not easily accessible and can lead to exorbitant costs.
Noticeably different from the aforementioned works, we explore making PTLM training more efficient by utilizing highly informative subsets of the training data. Consequently, our proposal ef- fectively complements other optimization meth- ods that target aspects such as model architecture and hardware enhancements.
3 The INGENIOUS Framework
We now present INGENIOUS an informative data subset selection framework for pre-training language models. We summarize the training pipeline in Figure 2. We first describe the nota-

---

B
Protebilly Dembutorn
baschh
Mink-
Submooutar Fintenng
Probabilltto Sompping
Sw. Aw. N
Min- bakch
Probabil Fammaang
Ppuer
Pr
P
Mr...P
8GD
M Pr..P
Ropeat
for stags
Si steps
01
Pay
Pr
Pojj
Whll Shuffled Ontrnnl
[en/P
M......
PLisip
Paye
F2
Pays
Pr
Fp
Partitoniig Into P blooks
B1
Softmex
partition
B2
Sample nKF samples parthiin
sP
based Pip
Obcain
submodular,
StenrP
01--PP
Subeet of size
Figure 2: INGENIOUS framework for informative data subset selection to pre-train language models. We warm- start pre-training for W steps to enable it to learn useful representations (step A). Owing to the size of pre- pre-training data, we divide the total number of samples (n) into partitioons (step B₁ followed by selecting instances according to submodular gains (step B2) through probabilistic sampling (step C). We obtain subset (of total size k) of representative samples from each partition such that the subset is updated periodically (step D) after R steps of training on selected subset.
imal redundancy within itself.
Lazy-Greedy (Mirzasoleiman et al., 2015). We use the Lazier-than-Lary Greedy optimizer it is the most computationally efficient, along with memoization (lyer and Bilmes, 2019).
3.3 Overview of Approach
In order to select a representative subset as dis- cussed above, we use Facility Location (Salhi, 1991; Krause and Golovin, 2014), a commonly- used submodular function closely related to k- medoid clustering which is defined as
The facility location function utilizes a pairwise similarity kemnel K. (of size [4] x (44)) between the data samples in 2/ to select representative subsets. To estimate the kernel values, compute the co- sine similarity between the feature representations of data samples obtained using the LM itself. To ensure that the extracted representations are mean- ingful during the initial phase, we warm start the model for W training steps as suggested by Kil- lamsetty et al. (2021a,c) (step A in Figure 2). Fur- ther, to ensure that LM sees diverse data samples, we probabilistically sample data points based on submodular ordering obtained from running the greedy algorithm(steps B and C in Figure 2) and update the subset after every Rthh iteration (step in Figure 2).
(A) Σ max Kijj
(2)
JEA
where A the subset being evaluated, IC is pair- wise similarity matrix and K.j is the similarity be- tween the th and yth samples. Thus, our subset selection problem can be represented as:
S arg max (F((S)
(3)
sC
Here, A represents the size of the subset S. We would like clarify that Equation (3) en- ables us to choose diverse samples such that each represents other samples in the corpus, in- stead of selection of similar samples. The op- timization problem in Equation (3) is an in- stance of cardinalit-coonstraieed monotone sub- modular maximization for which approxi- mate solution can be obtained by incrementally building the subset from scratch using algo- rithms such as Naive Greedy (Nemhauser et al., 1978), Lazy Greedy (Minoux, 1978), Stochastic Greedy (Mirzasoleiman et al., 2015), Lazier-than
This re-sampling procedure is repeated till the pre- determined number of steps. Algorithm 1 sum- marises the steps involved and in the following section, we describe the details of each step.
3.4 Methodology Details
Feature Encoders for Similarity Computation: The selection of optimal representative subsets re- quires similarity kernel that captures the intrin- sic relationships between data samples. We ex-

---

Algorithm 1: Pre- Pre-Training using INGE-
kernel, we now describe how select the subsets (steps B and C in Figure 2) as defined by Equa- tion (3), Approximate submodular maximization algorithms such as LazierthanLazy Greedy start with an empty subset and incrementally add data points one by one till the size of the subset equals the budget & set by us. If S represents subset se- lected so far, and represents the next locally opti- mal data sample to be added, the submodular gain value of is defined as f(S Ue) f(S). While running the algorithm, we initially set the budget as the size of the entire data(say MM in order to obtain and store the submodular gain (step B2 in Figure 2) of each data sample at the time of their addition.
NIOUS
Input: Training dalaset: Initial model parametets: 6o. Total of training stops: 7. "Taining steps interval for subset selection: R. Number of steps for warmstart phose: Size of the subset: k. Learning rates: Jez0
eptimizer Ad)mW()
owe Warmstart Phose ouee
repeat
Compuse batches 14 = ((1r, 8b): &.E (1-.. B)) from
for
1i
i 2 W then
bwaak
Compute mask on
4 opeimizersseppl stepp)
until wordd z 2 W
ane Subset Solection 644
gresdyldxs, gains orgmant is (n (S,,,,
probubilities = TaylonSSllmasgginss))
S, r-stmppisgresdylliss probubilliies, k)
regent
Compute batches Su.
((aa, Mo); h E (1 ... BJ) from
S:
The key idea here is to use the submodular gains associated with each data sample as importance score; convert them to probability distribution by using the second order Taylor-softmax oper- ation (de Brébisson and Vincent, 2016) (step C in Figure 2) and then sample a subset of desired size(say k) from the above distribution. Given gains vector {91. 92. .M Taylor-sofumax operation over the vector for converting it to prob-
for
1 B
I 2:2 T then
L break
Compute mask Sya
lct1 = poffimszer.steply
(6%. 0) then
S+41 mrsappoggrooyyyssss protaiiiii k)
beaak
eise
L S.+1 S.
until mntil t 2 T
ability distribution P can be specified as P def 11 0.5gf M
owe Evaluate trained model validation set wee eval evaluate (0z. 17)
return eval, 0r
Lig. 0.5g 1=1
Using the probability distribution P for sampling ensures that samples which have high impor- tance score associated with them are selected with greater probability. However, it also allows the LM to explore the samples with low importance score during training to prevent overfitting. We reuse this probability distribution to sample new subsets of size every R steps by sampling k points without replacement (step D in Figure 2).
plore dense and sparse feature encoders for obtain- ing the feature representation of text samples in U. As a dense feature encoder for text samples, we use the intermediate representations as obtained from the LM that is currently being trained. We compute the representations of an input sequence by averaging the output embeddings of the con- stituent tokens. A question then arises on which layer of the underlying model should be used for obtaining this representation since different lay- ers encode different types of information (Rogers et al., 2020). Another possibility is to use sparse representations such TF-IDF (Aizawa, 2003) owing to its success capturing statistically im- portant lexical features (Robertson et al., 2009). We study the effect of using sparse feature repre- sentations (i.e., TF-IDF) and dense feature repre- sentations obtained from different layers of LM in Section 4.3. Our experiments revealed that dense feature encoders yield the best results.
Recall that we require a similarity kernel of size [4| x [M], the similarity kernels is practically infeasible. We now describe how we scale INGENIOUS to handle size of the pre-training datasets used for LMs.
hence the memory required for storing
Partitioning based Efficient Subset Selection: To minimize the memory consumption, instead of constructing probability distribution over the en- tire unlabeled set directly, we first partition (step B1 in Figure 2) the unlabeled set into Np random blocks of equal sizes (i.e., partition size is Np and construct probability distribution P, over each data block U," |u = Ne
Submodular Greedy Ordering based Data Se- lection: After deciding on the choice of similarity
[4]
We then use the con-

---

structed probability distributions P; over each data block Lff to sample subset of size k/ from the data block without replacement. We compute the final subset using subsets from each partition as follows:
Avg. GLUE
CoLA
Score
Score
Vanilla
82.72
55.98
(IM steps)
Random-Selection (BI)
80.67
51.2
(250K steps)
(-2.05%)
(44.7%%
k NF
Early Stopping (B2)
81.23'
50.93
St = U sample (Uf, Po. Np
(4)
(250K steps)
(-1.49%)
(-5.05%)
Loss-based Sampling (B3)
81 21.
$1.68
(250K steps)
(-1,47%)
(-4.3%)
INGENIOUS
81.37
54.61
The partitioning of the unlabeled set allows to get away with constructing similarity kernels of Np Np. thereby reducing the similarity ker- nel memory usage by around Np² times. We dis- cuss the effect of the partition size in Appendix K. In order to maximize the utilization of available resources, we can construct probability distribu- tions over each block in the data partition in par- allel. As in recent work (Mittal et al., 2022), par- titioned facility location bound of the original objective function.i.e. facil- ity location that is being maximized. should be noted that memory utilization also increases with the number of parallel processes. For example, when Npp subsets are selected from partitions in parallel, the memory usage due to similarity kernel is of the order ON 1M2 ). In our experiments, No
(250K steps)
(-1.15%)
(-1.37%)
Table 1: Comparison of INGENIOUS with vanilla pre- training (full IM steps) and other baselines for BERT. We report fine-tuning performance mark and CoLA task in GLUE averaged over 20 runs. Statistically significant improvements(as measured by one-tailed t-test with 99% significance level) over base- lines B1, B2, B3 are indicated by superscripts 1,2,3 respectively. Numbers in brackets denote difference relative to vanilla variant. We report metrics for IN- GENIOUS and baselines after 250K pre- pre-training steps. Please refer Appendix for task-wise scores and Ap- pendix E for validation set losses during the course of pretraining
size R1l, M4
GLUE bench-
be shown as lower
4.1 INGENIOUS for BERT Pre-training
We consider two leagues of pre-trained models, viz., (i) BERT pre-trained on subsets selected through INGENIOUS and (ii) vanilla BERT pre- trained fully up I million steps. We contrast these by fine-tuning each the commonly used GLUE benchmark (Wang et al., 2019) and re- port the performances of each. Further, we com- pare INGENNOUS against three baselines B1) Random Selection: which is obtained by pre- training BERT on a randomly sampled subset of the same size as that selected by INGENIOUS; B2) Early Stopping: BERT pre-training stopped at 250K steps checkpoint for evaluation; B3) Loss Based Sampling (Loshchilov and Hutter, 2016): which obtained by pre-training BERT on sub- set, of the same size as those selected by INGE- NIOUS, sampled from a probability distribution that is constructed by ranking the losses in de- scending order and allocating the high rank (high loss) samples greater probability than low rank (low loss) samples. We would like to emphasise that choose the baselines B1 and B3 owing to their relevance to making LM pre-training efficient with respect to data optimization. Table I reports the GLUE score averaged over 20 runs on the dev sets obtained after 250K pre-training steps for the pre- pre-trained models obtained by different methods.
we set Npp 100 processes.
4 Experiments and Results
We use BERT (Devlin al., 2019), GPT-2 (Rad- ford et al., 2019) and a domain-specific version BERT BioBERT (Lee et al., 2020) as the underlying LMs. Specifically, we use BERT- Base(110M) and GPT2-SmalI(12MM) For BERT, we use English Wikipedia in conjunction with BooksCorpus the pre-training corpora and em- ploy MLM and NSP tasks for pre-training follow- ing details in the work of Devlin et al. (2019). We perform pre- pre-training using batch size of 1024 for 1,000,000 steps the case of vanilla-BERT. We perform ablations over data subset sizes and num- ber of pre-training steps for INGENIOUS enabled pre-training and find subset size of 25% (Ap- pendix J) with 250,000 pre-training steps (25%) as optimal choice. We set the value of R to 25000 steps. We refer the reader to Appendix G for fur- ther implementation details. For INGENIOUS en- abled pre-training of BioBERT and GPT-2, we dis- cuss the implementation details and experimental results in Sections 4.4 and 4.5, respectively.

---

82.0
VanillaBERRT
83.0
Loss Based Sampling
GLUE Score
81.5
INGENIOUS
GLUE Score
82.5
81.0
80.5
82.0
80.0
Vanilia BERT
79.5
Lass-Based Sampling
INGENIOUS
100
300
400
81.0
400
450
500
550
600
650
700
750
800
850
900
950
1000
PreTraining Steps (in
PreTraining Steps (in K)
82.0
Venille-BERT
Random Selection
Figure INGENIOUS is found to outperform the base-
GLUE Score
81.8
Losx Boax-Baaed Sampling
INGENIOUS
lines even extended training till 1M steps
81.6
81.4
the CoLA task (in Table 1) which is deemed to be most difficult (Geiping and Goldstein, 2022) in the GLUE benchmark. This implies that the sub- sets selected by INGENIOUS are able capture the important and highly informative signals from the underlying data resulting robust performance on challenging tasks as well.
81.0
80.8
80.6
PreTraining Cost (in USD)
Comparison of INGENIOUS with vanilla BERT GLUE performance pre-training steps (top) and cost (bottom) using checkpoints obtained at intermediate pre-training stages.
Figure 3:
Further, to compare different methods at differ- ent stages pre-training, we obtain correspond- ing checkpoints and fine-tune on GLUE tasks. For this particular setting, we present comparison of vanilla BERT pre-training against INGENTOUS in Figure 3. We plot the downstream performance for all the methods and it can be seen that INGENIOUS shows better performance than all the baselines at 250K steps of pre-training and thereafter, be- yond 250K steps, the trend continues consistently (Figure 3 top). Also, pre- pre-training through infor- mative subsets enables BERT to achieve a perfor- mance level at 250K steps which the vanilla pre- training achieves only after over 350K iterations. Similarly, for any given pre-training cost, INGE- NIOUS yields better GLUE score than the base- lines (Figure 3 bottom). Further we observe that INGENIOUS consistenly outperforms the baselines even when extended to million smaxsmma number of training steps prescribed by Devlin et al. (2019) for vanilla BERT pre-training) as shown in Figure 4.
We observe that despite using only a subset of training data and being trained only for 250K steps, INGENIOUS achieves 98.6% performance of the vanilla fully pre-trained BERT. Further, INGE- NIOUS achieves statistically significant improve- ments over the three baselines (B1, B2, and B3). INGENIOUS also outperforms baseline B3, which prioritizes training the BERT model on samples with a high loss rate. Prioritizing high-loss sam- ples may likely result in overfitting, which may explain the poor fine-tuning performance of base- line B3 on GLUE tasks compared to baseline B2. Therefore, INGENIOUS selects informative subsets that not only help improve BERT pre- training convergence but also help retain its gen- eralization capabilities. Further, we observe that extended training of INGENIOUS till 400K steps yields 99.1% 1% performance of the vanilla BERT. We would like to highlight that most of the down- stream task performance achieved by an PTLM is due to the initial stages of pre-training with most of the later pre-training resulting in up to 1% improvement (Smith al., 2022). In this con- text, INGENIOUS helps in achieving later-stage performance gains relatively earlier. Finally, we would like to highlight that INGENIOUS performs significantly better compared to the baselines on
Effectiveness of Importance Sampling: We also evaluated variant where the samples are selected greedily based on submodular ranking instead of importance sampling over submodular gains. In contrast the 81.57 achieved by INGENIOUS, it achieved an Avg. GLUE score of 80.5 after 250K pre-training steps, highlighting the effectiveness

---

Method
Google-RE
T-REx
ConceptNet
SQuAD
Embeddings Representation
Avg. GLUE
BERT,
3,99
25.76
11.48
14.77
Score
1M steps
Kandom Selection
250K
4.1
23
7.87
11.37
Layer 3
81.03
BERT Early Stopping
2.23
24.28
12.41
Layer 6
80.88
250K steps
Bass-B Sampling
Layer 9
81.57
21.66
250K steps
Layer 12
80.93
INGENIOUS
3.39
24.08
11.15
13.71
250K steps
TF-IDF
81.16
Table 2: Knowledge retention of different models as measured by LAMA probe. We report P@ I scores for the four different subtasks LAMA.
Table 3: Ablation study by varying embedding repre- sentation for selecting subsets. We report mean GLUE score to compare INGENIOUS variants.
BCZGMINER)
CHEMPROTIRE)
of importance sampling.
Rueeoo 5c0
4.2 Knowledge Retention with INGENIOUS
Large PTLMs, when trained on a sufficiently large corpus, stores various types of knowledge im- plicitly in their parameters (AlKhamissi et al., 2022). Since INGENIOUS uses only a subset of the whole data for pre-training, it is natural for it to contain lesser knowledge in its parameters but how does it compare with vanilla BERT pretrain- ing and other baseline when comes knowl- edge retention? To answer this question, we use LAMA benchmark (Petroni et al., 2019), a probe designed to analyze factual knowledge present in PTLMs. LAMA is derived from four distinct types of knowledge sources Google-RE, T-REx, Con- ceptNet, and SQuAD from which cloze sen- tences created using facts contained in the spective knowledge sources. The PTLM has to predict the fact tokens place of the mask tokens in cloze sentences. In Table 2, we summarize the results. We note that INGENTOUS suffers minimal loss in knowledge retention with respect to fully pre-trained vanilla BERT on all tasks. Further, the decrease in performance is less as compared to the baselines (for most tasks) which suffer more
vanill
verill
PreTririing time ún
Peeraning
(a) BC2GM (Smith al., (b) ChemProt (Taboureau 2008) Dataset
et al., 201 Dataset
Figure 5: Plots (a) and (b) are the convergence results comparing Avg. Fl score (over three runs) with the Wall clock time for Vanilla BioBERT and BioBERT using INGENIOUS with 25% subset. We observe that INGENIOUS achieves much faster convergence than vanilla BioBERT(í.... Full Training).
and report the performance on GLUE in Table 3. We observe that layer 9 features yield the best re- sults. Further, in Table 3, we compare the effect of using TF-IDF sample representations and con- trast them against dense features (BERT Layer-9). We observe that dense embeddings perform bet- ter than shallow TF-IDF features. We also report effect of subset size and number of partitions Appendices and K.
4.4 INGENIOUS for Domain-Specific PTLM BioBERT
vere decrease in performance. Intuitively, we tribute this to the ability of INGENIOUS select highly informative subsets from the corpus while excluding the redundant information.
We evaluate the performance of Bio-BERT (Lee et al., 2020) pre-trained on subsets selected through INGENIOUS and compare it with vanilla Bio-BERT by fine-tuning on biomedical datasets for the Named Entity Recognition (NER) and Re- lation Extraction (RE) tasks. For vanilla Bio- BERT, we start with a pre-trained BERT model and further pre-train it on the PubMed abstracts datasct for 200,000 steps(as recommended by the original study). Please refer Appendix for fur- ther implementation details. We present the per- formance convergence plots vanilla Bio-BERT vs. training time using INGENIOUS with a subset size of 25% in Figure 5. It shows that during initial
4.3 Effect of Embedding Representations
Different BERT layers have been shown to cap- ture different information lower layers capture word order (Rogers et al., 2020), middle capture syntactic information (Hewitt and Manning, 2019; Jawahar et al., 2019) and the later layers capture task saec-fpe information (Kovaleva et al., 2019;
Hao et al., 2019). We vary the layers (3, 6, and 12) used to obtain features for subset selection

---

ilarity can be potentially constrained by memory limitations and might require high CPU RAM ca- pacity. Further, we acknowledge that our exper- iments are performed on relatively smaller PTLMs compared to GPT-3, OPT or PaLM owing to re- source limitations. We have tried our best to per- form extensive experiments and perform ablation studies to inform our design choices within our re- source constraints.
GUI S0or
INGENIOUS
Vastta-GPP GF12
U6
220 250 350
66
dea
(a) GLUE (Wang et al., (b) BBQ Lite (Srivastava ct al., 2022) Dataset
2019) Benehmark
Figure 6: Comparison of INGENIOUS with vanilla GPT-2 pre-training at different pre-training stages. Pre-training on INGENIOUS subsets enables GPT-2 to achieve hetter GLUE score consistently.
7 Ethical Considerations
We believe that INGENTOUS has a significant pos- itive impact on society since makes pre-training of LMs compute efficient, thereby reducing CO2 emissions and energy costs. Nonetheless, the IN- GENIOUS framework is susceptible to biases and toxic words within the pre-training corpora as it relies standard pre-training datasets. An excit- ing future direction of this research is to investi- gate whether we could use targeted subset selec- tion to filter out toxic words, as well as phrases that promote cultural stereotypes and biases from the pre-training corpora before LM pre-training.
stages of pre-training, INGENIOUS performs sim- ilar to vanilla since the LM is still learning repre- sentations, however once better representations for subset selection are Iearned, INGENIOUS achieves faster convergence than vanilla w.rl pre-training time and achieves the best accuracy around 1.4x faster.
4.5 INGENIOUS for GPT-2 Pre-training
We also pre-train GPT-2 (Radford et al., 2019) us- ing INGENIOUS. We estimate the mean accuracy for GLUE fine-tuning (averaged over 20 runs) and zero-shot accuracy on BBQ Lite generative task. Please refer to Appendix H for implementation details. We plot the performance (see Figure 6) obtained for the above benchmarks against check- points at different pre-training stages (steps). Fig- ure left and right shows that INGENIOUS per- forms consistently better than vanilla GPT-2 pre- training on GLUE and BBQ Lite respectively at different stages of pre-training indicating better convergence.
Acknowledgments and Disclosure of Funding
This work is supported by Adobe Data Sci- ence Award, and by the National Science Foun- dation under Grant No. IIS-2106937 awarded Rishabh Iyer. Any opinions, findings, and conclu- sions or recommendations expressed this mate- rial are those of the authors and do not necessar- ily reflect the views of Adobe or the National Sci- ence Foundation. Ganesh Ramakrishnan is grate- ful to the Adobe Award and Institute Chair Profes- sorship Award at IIT Bombay for supporting this work.
5 Conclusions
We presented INGENIOUS, a framework for effi- cient pre-training of language models using highly informative data subsets, and presented submod- ular optimization based algorithm. We described how it can be scaled for language models and showed its effectiveness using rigorous empiri- cal evaluation. Our future work will explore ex- ploiting external knowledge bases to identify and reduce redundancies the corpus and to study multi-modal training where redundant information can be spread across different modalities.
References
Akiko Aizawa. 2003. An information-thecretic per- spective of tf-idf measures. Information Processing & Managemens, 39(1):45-65.
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. re- view on language models as knowledge bases.
Francis Bach. 2013. Learning with submodular fune- tions: convex optimization perspective. Foundations and Trends in Machine Learning, 6(2-3):145-373
6 Limitations
Francis Bach. 2019. Submodular functions: from dis- crete continuous domains. Mathematical Program- ming, 175(1)4419-55
terms of limitations, the submodular maximiza- tion based on estimation pairwise sample sim-

---

Jeff Bilmes. 2022. Submodularity in machine learning and artificial intelligence.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 143-155, Online. Association for Computational Linguisties.
Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio. 2019. Semantic redundancies in image- classification datasets: The 10% you don't need. ArXiv, ats/aao...110999
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visu- alizing and understanding the effectiveness of BERT. Empir-
Yonatan Bitton, Michael Elhadad, Gabriel Stanovsky, and Roy Schwartz. 2021. Data efficient masked lan- guage modeling for vision and language. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, pages 3013-3028, Punta Cana, Domini- can Republic. Association for Computational Linguis- ties.
In Proceedings of the 2019 Conference ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4143- 4152, Hong Kong. China. Association for Computa- tional Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Sariel Har-Peled and Soham Mazumdar. 2004. On coresets for k-means and k-median clustering. In Pro- ceedings of the thirty-sixth annual ACM symposium Theory of computing, pages 291-300
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Pupers), pages 4129- 4 4138, Minneapolis, Minnesota. Association for Com- putational Linguistics.
Trevor Campbell and Tamara Broderick. 2018. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Ma- chine Learning, pages 698-706.
Rishabh lyer and Jeffrey Bilmes. 2019. A memoiza- tion framework for scaling submodular optimization to large scale problems. In The 22nd International Con- ference on Artificial Intelligence and Statistics, pages 2340-2349 PMLR
Tianlong Chen, Jonathan Frankle, Shiyu Chang. Si- jia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 15834- 15846. Curran Associates, Inc.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. 2021. Submodular combinato- rial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722- 754. PMLR.
Alexandre de Brébisson and Pascal Vincent. 2016. An exploration of softmax alternatives belonging to the spherical loss family. In 4th International Conference on Learning Representations, ICIR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Pro- ceedings.
Rishahh Krishnan Tyer. 2015. Submodular optimiza- tion and machine learning: Theoretical results, unify- ing and scalable algorithms, and applications. Ph.D. thesis.
Peter Izsak, Moshe Berchansky, and Omer Levy. 2021. How to train BERT with an academic budget. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10644-10652, Online and Punta Cana. Dominican Republic. Associ- ation for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186, Minneapolis, Minnesota. Association for Com- putational Linguistics.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computa- tional Linguistics.
Dan Feldman. 2020. Core-Sets: Updated Survey, pages 23-44. Springer International Publishing, Cham.
Satoru Fujishige. 2005. Submodular functions and op- timization. Elsevier.
Xiaoqi Jiao, Yichun Yin. Lifeng Shang, Xin Jiang. Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. 2020. BERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 4163 Online. Association for Computational Linguistics.
Jonas Geiping and Tom Goldstein. 2022. Cramming: Training a language model on a single gpu in one day. arXiv preprint arXiv:2212. 14034.

---

Jeremy Kahn, 2021. A.I'' carbon footprint is big, but easy to reduce, Google researchers say. https://fortun e.com/202 1/0////////11111/ /aiicarrono---ooo htt.haaabboooooootttt--eeee redueerrnt--reduceeenn ironmental- -fe oarmpptt---ffcchgooooo-rr-eerrre resarh--rsearcc--tuuuy. [Online; accessed 19-September-2022],
ceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 131-141.
Suraj Kothawade, Nathan Beck, Krishnateja Killam- setty, and Rishabh Iyer. 2021. Similar: Submodular information measures based active learning in realistic scenarios. In Advances in Neural Information Process- ing Systems, volume 34, pages 18685-18697. Curran Associates, Inc.
Athresh Karanam, Krishnateja Killamsetty, Harsha Kokel, and Rishabh K Iyer. 2022. ORIENT: Submod- ular mutual information measures for data subset se- lection under distribution shift. In Thitty-Sixth Confer- ence on Neural Information Processing Systems.
Suraj Kothawade, Jiten Girdhar, Chandrashekhar La- vania, and Rishabh Iyer. 2020. Deep submodular networks for extractive data summarization. arXiv preprint arXiv:2010.08593.
Jivat Kaur, Sumit Bhatia, Milan Aggarwal, Rachit Bansal, and Balaji Krishnamurthy. 2022. LM-CORE: Language models with contextually relevant external knowledge. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 750-769, Seattle, United States, Association for Computational Linguistics.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCCNPP) pages 4365- 4374, Hong Kong, China. Association for Computa- tional Linguistics.
Vishal Kaushal, Rishabh lyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrish- nan. 2019. Learning from less data: A unified data sub- set selection and active learning framework for com- puter vision. In 2019 IEEE Winter Conference Ap- plications of Computer Vision (WACV), pages 1289- 1299. IEEE.
Andreas Krause and Daniel Golovin. 2014. Submodu- lar function maximization. Tractability, 3:71-104.
Jinhyuk Lee. Wonjin Yoon, Sungdong Kim, Donghyeon Kim. Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformaticx, 16(4):1234-1240)
Vishal Kaushal, Ganesh Ramakrishnan, and Rishabh Tyer. 2022. Submodlib: A submodular optimization li- brary, arXiv preprint arXiv:2202. 10680.
Krishnateja Killamsetly, Guttu Sai Abhishek, Aakriti Lnu, Ganesh Ramakrishnan, Alexandre V. Ev- fimievski, Lucian Popa, and Rishabh K Iyer. 2022. AUTOMATA: Gradient based data subset selection for compute-efficient paraamea tuning. In Thirty- Sixth Conference on Neural Information Processing Systems.
Conglong Li, Minjia Zhang, and Yuxiong He. 2022. Curriculum learning: A regularization method for effi- cient and stable billion-scale GPT model pre- pre-training.
I Loshchilov and F Hutter. 2016. Online batch selec- tion for faster training of neural networks. In Internu- tional Conference on Learning Representations (ICLR) 2016 Workshop Track.
Krishnateja Killamsetty, Durga S. Ganesh Ramakrish- nan, Abir De, and Rishabh Iyer. 2021a. Grad-match. Gradient matching based data subset selection for ef- ficient deep model training. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of P Proceedings of Machine Learning Re- search, pages 5464-5474. PMLR.
Ayush Maheshwari, Oishik Chatterjee, Krishnateja Killamselty, Ganesh Ramakrishnan, and Rishabh Iyer. 2020. Semi-supervised data programming with subset selection. arXiv preprint arXiv:2008. 09887.
Michel Minoux. 1978. Accelerated greedy algorithms for maximizing submodular set functions. In Optimiza- tion techniques, pages 234-243. Springer.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Tyer. 202 Glister: Generalization based data subset selec- tion for efficient and robust learning. Proceedings of the AAAI Conference Artificial Intelligence, 35(9):810-8811
Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. 2015. Lazier than lazy greedy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29.
Krishnateja Killamsetty. Xujiang Zhao, Feng Chen, and Rishabh Iyer. 2021c, RETRIEVE: Coreset se- lection for efficient and robust semi-supervised learn- ing. In Advances in Neural Information Processing Systems.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. 2020. Coresets for data-efficient training of machine learning models. In Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Re- search, pages 6950-6966 69666 PMLR.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,690.
Ashish Mittal, Durga Sivasubramanian, Rishabh Iyer, Preethi Jyothi, and Ganesh Ramakrishnan. 2022. Par- titioned gradient matching- data subset selection for conpute-efficien robust asr training. In Findings of
Katrin Kirchhoff and Jeff Bilmes. 2014. Submodular- ity for data selection in machine translation. In Pro-

---

the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, Abu Dabhi.
Said Salhi. 1991. Discrete location theory. Journal of the Operational Research Society, 42:1124-1122
Aashiq Muhamed, Iman Keivanloo, Sujan Perera, James Mracek, Yi Xu, Qingjun Cui, Santosh Ra- jagopalan, Belinda Zeng, and Trishul Chilimbi. 2021. Cer-bert: Cost-effective knowledge distillation for billion-parameter parameter teacher models. In NewrlPSS Efficient Natural Language and Speech Processing Workshop.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilhert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint AKIVv:9100011
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lunguage Tech- nologies, pages 2339-2352, Online. Association for Computational Linguistics.
Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2021. Pre-training a BERT with curriculum learning by increasing block-size of input text. In Proceedings of the International Conference on Recent Advances in Natural Language Process- ing (RANLP 2021), pages 989-99999 996. Held Online. IN- COMA Ltd.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63( 24:554333..
George L Nembauser, Laurence A Wolsey, and Mar- shall I. Fisher. 1978. An analysis of approximations for maximizing submodular set functions-i. Mathe- matical programming, 14(1):265-25 294.
Or Sharir, Barak Peleg, and Yoav Shoham. 2020. The cost training nip models: A concise overview. arXiv preprint urXiv:2004.0840
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. Deep learning on data diet: Find- ing important examples early in training. In Advances in Neural Information Processing Systems, volume 34, pages 206077 Curran Associates, Inc.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy. 2022. Staged train- ing for transformer language models. In International Conference on Machine Learning. ICML 2022, 17- 23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 19893-19908. PMLR.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Intermational Joint Conference Natural Language Processing LEMNLP-JJCNLP), pages 2463- 2473, Hong Kong, China. Association for Computa- tional Linguistics.
Larry Smith, Lorraine K. Tanabe, Rie Johnson nee Ando, Cheng-Ju Kuo, I.-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M. Friedrich, Kuzman Ganchev, Manabu Torii, Hongfang Liu, Barry Haddow, Craig A. Struble, Richard J. Povinelli, An- dreas Vlachos, William A. Baumgartner. Lawrence Hunter, Bob Carpenter, Richard Tzong-Han Tsai, Hong-Jie Dai, Feng Liu, Yifei Chen, Chengjie Sun, Sophia Katrenko, Pieter Adriaans, Christian Blaschke, Rafael Torres, Mariana Neves, Preslav Nakov, Anna Divoli, Manuel Maña-Lóppez, Jacinto Mata, and W. John Wilbur. 2008. Overview of BioCreative II gene mention recognition. Genome Biology, 9 Suppl 2:S2.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. 2022. Adaptive second order coresets for data-efficient machine learning. In Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pages 77848-17866 PMLR.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18):9
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vi- jay Korthikanti, et al. 2022. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, large- scale generative language model. arXiv preprint arXiv:2201. 1190.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 21(14)):1667.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2022. Beyond neu- ral scaling laws: beating power law scaling via data pruning. In Thirty-Sixth Conference on Neural Infor- mation Processing Systems.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch. Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonoo, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 22X1v:2206.04661.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.

---

Olivier Taboureau, Sonny Kim Nielsen, Karine Au- douze, Nils Weinhold, Daniel Edsgärd, Francisco S. Roque, Irene Kouskoumvekaki, Alina Bora, Ramona Curpan, Thomas Skøt Jensen, Søren Brunak, and Tu- dor I. Oprea. 2011. ChemProt: a disease chemical bi- ology database. Nucleic Acidy Research, 39(Dtabaaee 1s80e):D367-372.
APPENDIX
A Code, Software, and Licenses
The data and code for INGENTOUS is available at the following url: https://github. http://ww..ooomefiiiietttt-eeeee ingenious. We release the code repositories of IN- GENIOUS with an MIT license, which is available for everybody to use freely.
Ehsan Tohidi, Rouhollah Amiri, Mario Coutino, David Gesbert, Geert Leus, and Amin Karbasi. 2020. Sub- modularity in action: From machine learning to signal processing applications. IEEE Signal Processing Mag- azine, -003333
All the code is developed using open-source Hug- gingFace for training LMs with Py as the underlying framework. PyTorch is available un- der the BSD license. HuggingFace is available under Apache 2.0 license. For submodular opti- a library called SUBMODLIB (Kaushal al., 2022), which is freely available at https://github com/deeile teaeeilleeeammmmmmloiiii which is available under the MIT license.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. 2019. An empirical study of exam- ple forgetting during deep neural network learning. In ICLR.
mization,
Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
B Additional Background and Related Work
Kai Wei, Rishabh Iyer, and Jeff Bilmes. 2015. Sub- modularity in data subset selection and active learn- ing. In International Conference on Machine Learn- ing, pages 1954-1999 1963. PMLR.
Submodular Functions: Let 11 denote the unla- beled set of n data points U = 3,, ... and set function f 2 + R. Formallyy function f submodular (Fujishige, 2005; Bilmes, 2022) if for x E 4. f(AUx) f(A) 2 f(BUx) x) f(B), AA C B u and x & B. For a set A C U. f(A) provides a real-valued score for A. A function f is said to be monotone if (A) S S(B) whenever A S B. Further, f is supermodular if ff is sub- modular, modular if it is both, and normalized if f()) 0. Submodularity occurs naturally in var- ious real-world applications (Tohidi et al., 2020; Bach, 2013, 2019; Iyer, 2015) and a number of combinatorial functions, such as facility location, set cover, log determinant, graph cut, etc. (lyer et al., 2021; Iyer and Bilmes, 2019; Kothawade et al., 2020, 2021; Karanam et al.. 2022) are in- herently submodular in nature. Submodularity is particularly attractive due the constant fac- tor 1 (Nemhauser al., 1978) approxima- tion for cardinality-constraee imization, allowing us to solve various combina- torial optimization problems, which are often NP- Hard in nature. Several recent works (Wei et al., 201 401 2015; Mirzasoleiman et al., 2020; Killam- setty et al., 2021b,a,c, 2022; Kothawade et al., 2021; Karanam al., 2022) have formulated the subset selection objective as submodular max- imization problem. Furthermore, variants of the greedy algorithm (Mirzasoleiman et al., 2015; Iyer and Bilmes, 2019) that can maximize submod-
Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. 2014a. Submodular subset selection for large-scale speech training data. In 2014 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pages 3311-3315. IEEE.
Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. 2014b. Unsupervised submodular subset lection for speech data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 4107-4111, IEEE.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Car- bonell, Ruslan Salakhutdinov, and Quoe V. Le. 2020. Xlnet: Generalized autoregressive pretraining for lan- guage understanding.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. 2022. NLP from scratch without large- simple and efficient framework. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pages 25438-2555555 25451. PMLR.
scale pretraining:
submodular max-
Ofr Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. 2021. Prune once for all: Sparse pre-trained language models. arXiv preprint 1..05:2111.05544.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. aixiv preprint 22x::22055.0066

---

ular function in near-linear time have been pro- posed.
E Pre-Training performance of INGENIOUS for BERT
Submodular Data Subset Selection: Submodu- lar optimization has been successfully employed for data subset selection in various applications such as speech recognition (Wei et al., 2014b,a; Mittal et al., 2022), machine translation (Kirch- hoff and Bilmes, 2014), active-learning (Wei al., 2015; Kothawade al., 2021), efficient deep learning (Kaushal al., 2019; Killamsetty et al., 2022: Pooladzandi et al., 2022). Another active area of research is selecting representative subsets data, also known as coresets (Feldman, 2020). A coreset is weighted subset of data closely approximating certain desirable properties of the entire dataset (e.g., the loss function) (Feldman, 2020). Coreset selection has been shown to bene- fit host of geometric problems such k-means and k-median clustering (Har-Peled and Mazum- dar, 2004) and, in recent times, has been used suc- cessfully for efficient bayesian inference (Camp- bell and Broderick, 2018) and improving train- ing efficiency (Mirzasoleiman et al., 2020; Killam- setty et al., 2021a). Such informative data subset selection has shown remarkable promise for effi- cient and robust training of deep models (Killam- setty et al., 2021b,c). We direct the reader survey by Bilmes (2022) for detailed review of submodularity and subset selection for ML.
In Table 7, we show how validation-set losses change for vanilla BERT and INGENTOUS BERT the course of pretraining.
F GLUE Task wise performance of INGENIOUS for BERT
We show task-wise performance on GLUE for BERT trained through INGENIOUS in Table 4. We compare against vanilla LM pre-training and base- lines. We also report the standard deviation for each task along with the mean glue score.
G Further implementation details of pre-training BERT through INGENIOUS
We use Adam optimizer (Kingma and Ba. 2014) with learning rate of le-4, 31 = 0.9, 32 0.99, L2 weight decay of 0.01. We warmstart the model for the first 80K training steps and subsequently train only on selected subsets. Training of all the models is performed on 8 NVIDIA A100-SXM4- 40GB GPUs.
H Implementation details of pre-training GPT-2 through INGENIOUS
For GPT-2, we use Open Webtext(An open-source replication of WebText dataset from OpenAl) the pre-training corpus and employ CLM task for pre-training following details in the work of Rad- ford al. (2019). We perform pre-training using batch size of 256 (achieved using gradient accu- mulation of steps) for 1,000,000 steps case of vanilla GPT-2. With INGENIOUS, we pre-train GPT-2 for 250,000 steps. We set the value of R as 25K steps. We use Adam Optimizer (Kingma and Ba. 2014) with learning rate of le-4, 31 = 0.9, Ba 0.99, L2 weight decay of 0.01. We warmstart the model the first 65K training steps and subse- quently train only on selected subsets. Training of all the models is performed on 16 NVIDIA A A100- SXM4-40GB GPUs.
Datasets
For pre-training BERT, we use English Wikipedia, BooksCorpus datasets. English Wikipedia is 200iB of text containing 6,458,670 articles and BooksCorpus is 5GGB of text containing 24,004,228 lines of text. For GPT-2, use Open- WebText which is open source replication of WebText dataset from OpenAI. OpenWebtext is around 440GB of text around 8,013,769 articles.
Compute Infrastructure
All our pre-trainings were done on Google Cloud Platform (GCP) instances comprising of 8 NVIDIA 1100SXXM-40GGB GPUs for 1 110M parameters) and 16 NVIDIA A100-SXM4-40GB GPUs for GPT- 2(gpt2-small: 124M parameters). In each in- stance, there are 96 CPU cores with total RAM of 680GiB. The costs are estimated using https: www.ooddooogl..cooooooooooo pttm/prruuctsssallllltooo based on the time taken for training.
Implementation details of pre-training BioBERT through INGENIOUS
We use Adam optimizer (Kingma and Ba. 2014)
with learning rate of le-4, B1 = 0.9, 32 0.99.
L2 weight decay of 0.01. For Bio-BERT training using INGENIOUS, we use a subset size of 25%

---

Pre-Training
(USD)
Aog.
Score
SST-2
(malched)
(mismalehael)
Vanilla BERT.
72
35.88110
60.31
92,085a
53.50
81.20
87.
Rundem- Seluction,
stepe
(:785516)
6122um
87.07
82.0
82.70
Vanilla
stopping, steps
(:74.55%)
(-1,49)
06 06.30.,
88.89.00
50.3.
82.30
88.1-
9022..4
67.38mm
Loss Sampling.
stepa
(-74.0598)
$t.25a ao
(-1.47)
31 31.56
80.83
88 Faucn
62.57
00.7
60
INGENIOUS,
stpps
5647.56
(653)
(-1.15)
34.1111
80.8
16
01.01.as
82.16 puuz
Table Comparison of pre- pre-training cost fine-tuning performance on GLUE tasks (averaged over 20 runs) for BERT. We report difference relative to full pre-training of vanilla BERT in brackets for cost and avg. GLUE score. INGENIOUS achieves 98.6% of fully pre-traine cee-rriieed BERT performance, reducing pre- pre-training cost to 28%.
Subset Size
Avg. GLUE Score
K Partitions for efficient subset selection
(% of full dataset)
As discussed in approach, we divide the pre- training dataset into partitions. In Table 6, we analyse the impact of performance on GLUE as the number of partitions is varied. Using fewest partitions (1500) is found to yield optimal perfor- mance. This aligns with the intuition that fewest partitions enable better subset selection since more samples are present in a single partition, allowing to select more representative samples overall.
10 %
81.18
15 %
81.24
20% %
81.10
25 %
81.57
30% %
81.17
Table 5: Ablation study by varying subset size of se- lected subsets. report mean GLUE score to com- pare INGENNOUS variants.
L Few Examples of informative texts sampled by INGENIOUS
Number of Avg. GLUE Score
partitions
We summarize the three types of redundancies that we found in our analysis of selected subsets. More examples can be found at https://gtthu..cmm// com///ffccc/f/ccccc ient Al/ingenious.
1500
81.57
2000
81.19
2500
81.37
3000
81.34
Type 1:
Same information conveyed by multiple sentences in different documents. Sentence 1: "separate sovereign coun- tries but acted as single bloc in for- eign policy and security issues. the pro- posed union was being discussed by a joint scandinavian committee during the winter of 1948 1949, but the cold war tension between the united states and the soviet union, and preparations for western alliance that would result in the north atlantic treaty overshadowed the effort. when it became"
Table 6: Ablation study by varying partition size used during pre-training. We report mean GLUE score to compare INGENIOUS variants.
R value of 5000, model warm-start, i.e., W 0, and trained the Bio-BERT model for 200,000 steps.
J Subset size for efficiency gains
We study the effect the size of the subset se- lected through INGENIOUS that is used for pre- training BERT. In Table 5, we analyse using the following values of subset sizes, viz., 10%, 15%, 20%, 25% and 30% and evaluate the fine-tuning performance GLUE. While lower subset sizes (10-20%) result in inferior performance owing to the fact that the LM is shown less information, op- timal performance is observed when 25% of the pre-training corpus is used, hence, we report cor- responding results in Table 1.
Sentence 2: "they would remain sep- arate sovereign countries but act as a single block in foreign policy and secu- rity issues. the proposed union was dis- cussed by joint scandinavian commit- tee during the winter of 1948 1949, but in the end the cold war tension between the united states and the soviet union and preparations for a western alliance that would result in"

---

Pre-Truining
100K 150K
200K
250K
300K
350K
400K
450K
500K
550K
600K
650K
700K
750K
800K
850K
900K
950K
1000K
Steps
Vanilla
2.29
1.99
1.8
1.78
1.74
1.73
INGENIOUS
BERT
2.29
2.09
2.01
1.95
1.9
1.8
1.85
1.85
1.82
1.8
1.79
1.78
1.76
1.76
1.74
1.74
1.72
1.71
1.7
Table 7: Comparison of validation set losses during pre-training.
INGENIOUS achieves almost similar validation
set loss as compared to vanilla BERT
Type 2: Duplicates in the corpus.
Sentence "after we'd been handed our menus. always get the frozen hot chocolate." frozen hot chocolate? it's really thing? thought they just made that up." no," said, pointing to the spot her menu. see? it's right there." so, do you order anything else?" cake." she looked my deadpanned face and laughed. so, we're"
Sentence 2: "what's good here?" mia asked me after we'd been handed our menus. always get the frozen hot chocolate." frozen hot chocolate? it's really thing? thought they just made that up." no," i said, pointing to the spot on her menu. see? it's right there."
Type 3: Recurring patterns of text. Sentence "according to the united states census bureau, the village has total area of, all land. demographics 2010 census as of the census of 2010, there were 377 people, 159 households, and 101 families residing in the village. the population density was. there were 176 housing units at"
Sentence 2:
"according to the united states census bureau, the village has a total area of, all land. demographics 2010 census as of the census of 2010, there were 801 people. 323 households,
and 225 families living in the village. the population density was. there were 358 housing units at"


Reviews:
[{"review_id": "KCibErzysB", "reviewer": "Reviewer_ETAv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work, the writing and the thorough evaluations. Your constructive feedback is highly appreciated and here are our detailed responses for your suggestions. \n \n**Choice of similarity kernel in Eq. (2):**\n\n> First, something that didn't come across was the importance and intuition behind the choice of the similarity kernel. What types of kernels work best? Are there, e.g., cheap empirical metrics that can effectively estimate the clustering kernel in eq. 2? Could you estimate this similarity kernel through something very simple, such as SentenceBERT embeddings? What features is it capturing that makes it particularly good?\n\nThe similarity matrix in Eq. (2) is vital as the pairwise similarity values help capture \"redundancy\". One of the most common metrics for computing similarity between two pieces of text is the cosine similarity between their vector representations. The critical decision, thus, is to decide on the choice of vector representations. As you noted, one alternative is to use embeddings from pre-trained models (e.g, SentenceBERT). However, note that we deliberately chose not to use them as that would be unfair since otherwise, our proposed (efficient) pre-training method would benefit from pre-training of another language model and would prevent a fair evaluation. To make INGENIOUS self-reliant, we employed embeddings of the underlying language model as it is being pre-trained.\n\nFurther, in Section 4.3, we present an empirical study on the kind of features that work the best. We found that embeddings selected from the middle layer performed better than those taken from the initial/final layers. As noted in the existing literature, initial layers capture word order; middle layers capture syntactic information, and final layers are the most specific to the pre-training objective. Semantic information, however, is spread throughout the model. Hence, based on the empirical evidence described in the paper, we hypothesize that middle-layer features capturing syntactic information are most helpful for our task. We also experimented with TF-IDF representations, although the best performance was from dense embeddings. (All of the above discussion is also presented in the first paragraph of Section 3.4, and also in Section 4.3). \n\n**Comparison to Computer Vision settings:**\n\n>Including a comparison to one of the methods mentioned in the computer vision setting would have been more useful than comparing to, e.g. loss-based sampling. I understand that these are not always applicable and typically require a supervised set-up, but some of them can probably be adapted to language tasks relatively easily.\n\nThere is a fundamental difference between how subset selection techniques are employed in Computer Vision (CV) -- these techniques are used for training for specific downstream tasks (classification, tagging, etc.) and not for pre-training of the model (as we propose in this work). Given the scale of pre-training data in typical LMs, adapting techniques used for CV is practically infeasible, and addressing this challenge is, in fact, one major contribution of our work -- how to make these techniques work for LMs. To explain it further, there are two primary inefficiencies with the current subset selection methods in CV, especially when it comes to pre-training large language models:\n\n1. **Gradient-Based Computations**: Traditional CV subset selection often involves gradient calculations, generally by retaining the model weights while only adjusting the final MLP layer used for classification. This gradient-based approach is computationally demanding as it requires calculating the per-sample loss gradients for the entire dataset. Such a process is more resource-intensive than the simple forward pass our framework uses to compute sample embeddings (Section 3.4).\n\n2. **Need to Select Multiple Subsets for Pre-Training**: For optimal data exploration (for knowledge retention) and to reduce the risk of overfitting (for generalization), it is essential to pre-train large language models on various data subsets, which are updated regularly. With existing CV methods, this demands that the whole subset selection, including the gradient computation, be carried out multiple times during the pre-training phase. This repetitive computation further compromises the efficiency. In contrast, after establishing the initial probability distribution, the INGENIOUS framework can swiftly sample new subsets in virtually no time (Section 3.4).\n\nFurthermore, to our understanding, even within CV contexts, there has not been an exploration of pre-training using the transformer architecture with data subset selection methods.\n\n**Subset size ablation:**\n\n> Not sure I understand why increasing the subset size in some cases actually hurts performance? e.g., Table 4. I think the paper could benefit by elaborating on this, and why some subsets of the dataset are actually harmful to model performance.\n\nGreat question! The size of the subset can impact the model in various ways. Note that we are trying to achieve a delicate balance between the information and \"noise\" that is fed to the model. While intuitively increasing the subset size provides more information to the model, it may also add noise to the model, possibly deteriorating the performance. In Table 4, we only report our empirical observations on the performance with varying subset size. We also note that recent studies ([1], [2]) have found the pre-training corpus quality, rather than just the quantity, to impact the model performance significantly.\n\n[1] Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E. and Launay, J., 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.\n\n[2] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. and Bikel, D., 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper is well written. The experiments are thorough, and I particularly appreciate the conversion of performance metrics to costs, which lets practitioners more easily evaluate trade-offs.", "reasons_to_reject": "There are 3 potential changes that would improve this work:\n* First, something that didn't come across was the importance and intuition behind the choice of the similarity kernel. What types of kernels work best? Are there, e.g., cheap empirical metrics that can effectively estimate the clustering kernel in eq. 2? Could you estimate this similarity kernel through something very simple, such as SentenceBERT embeddings? What features is it capturing that makes it particularly good?\n* Including a comparison to one of the methods mentioned in the computer vision setting would have been more useful than comparing to, e.g. loss-based sampling. I understand that these are not always applicable and typically require a supervised set-up, but some of them can probably be adapted to language tasks relatively easily.\n* Not sure I understand why increasing the subset size in some cases actually hurts performance? e.g., Table 4. I think the paper could benefit by elaborating on this, and why some subsets of the dataset are actually harmful to model performance.\n\n--\n\nEdit: I acknowledge the authors' comments below. Although I still have some questions/concerns about the subset ablations (and hope that the authors will eventually include a more granular analysis of why this happens, and if decoder only models exhibit some type of this behavior), I am moving to increase my soundness score as they have addressed my other comments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main contribution of this paper is a method to choose a subset of the pretraining model data that achieves similar performance as training on the full dataset. Data pruning methods have been extensively studied in the literature, but relatively little is known about optimal strategies for language model pretraining. This work presents an important and interpretable method to do so, and quantifies the trade-offs between cost and model performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GN7S7SQheN", "reviewer": "Reviewer_Ex9w", "summary": "", "questions": "Can you provide a more detailed explanation of how figures 3 and 4 were generated? How exactly are the values in y-axis computed? Additionally, can you clarify which specific glue task was used in these figures?\n\nDo you have any insights into why training on selected data can yield better results compared to a fully pre-trained model? Are there factors beyond noise reduction?\n\nCan you share a pre-training performance comparison between fully pre-trained BERT (e.g., Bert-base) and INGENIOUS pre-trained BERT, without downstream fine-tuning? One suggestion could be something like testing accuracy/loss on the original validation/test set.\n\nWill it be possible for you to share a comparison of the pre-training performance between a fully pre-trained BERT (such as Bert-base) and an INGENIOUS pre-trained BERT, without any downstream fine-tuning? One suggestion on evaluation metrics could be testing accuracy or loss on the original validation/test set.\n\nWill you release the selected subsets that yield your best models?\n\nI will raise my scores if the above concerns can be addressed/clarified by the rebuttal. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere thanks for the very detailed review and appreciating the proposed approach, writing and thorough experiments. Here are tthe detailed response for the clarifications sought.\n\n**Clarification on Figures (3,4) and Tables (1,2,6):**\n\n>First, I’m confused by Figures 3 and 4. In these illustrations, INGENIOUS appears to consistently and significantly outperform the baseline BERT model which undergoes full pre-training, even when it is trained up to 1 million pre-training steps. However, based on the context along with Tables 1&2 and the detailed Table 6 in Appendix F, it becomes evident that the most proficient model pre-trained with INGENIOUS achieves a 98.6% performance compared to the fully pre-trained BERT. This contrast in results is confusing.\n\nIn the Tables 1, 2 and 6, when we say INGENIOUS attains 98.6% of vanilla-BERT’s performance, we intended to convey that INGENIOUS trained for **just 250K steps** is able to attain 98.6% of the performance of vanilla BERT that is pre-trained till **1M steps**. Thus implying that INGENIOUS can lead to significantly faster learning. While Figures 3 (top) and 4 bothconvey the same point, there is a subtle difference.  Here,  we present the performance (measured by GLUE score) achieved by different training methods for the **same** number of pre-training steps (on the x-axis). We see that INGENIOUS achieves higher downstream performance than the vanilla BERT trained for the same number of steps, again implying faster learning given a compute budget.\n\n\n\n**Reasons for INGENIOUS Outperforming Vanilla-BERT:**\n>Do you have any insights into why training on selected data can yield better results compared to a fully pre-trained model? Are there factors beyond noise reduction?\nRecall from Section 3.4 that INGENIOUS employs a probability distribution constructed through submodular gains to sample subsets. This method prioritizes data points with higher importance scores i.e., that are more informative and are thus, more likely to be included in the subset used for training. As a result, the model frequently encounters these high-importance samples, while reducing exposure to redundant data points. We believe this selective exposure to more informative samples is the primary reason INGENIOUS excels. The model achieves better performance than vanilla BERT by effectively reducing redundancy in the training corpus, as INGENIOUS does. The enhancement in LLM performance due to the reduction of redundancy is also supported by recent studies [1] where authors demonstrated that de-duplicating training data significantly improves language model training.\n\n\n**Y-axis in Figures 3 and 4:**\n\n>Can you provide a more detailed explanation of how figures 3 and 4 were generated? How exactly are the values in y-axis computed? Additionally, can you clarify which specific glue task was used in these figures?\n\nSure. As discussed in the paper (Section 4.1), the y-axis in Figures 3 and 4 is the average GLUE score (averaged over 20 fine-tuning runs). In each run, we compute the GLUE score as the average performance across the nine GLUE tasks - CoLA, MRPC, STS-B, RTE, SST-2, MNLI(m), MNLI(mm), QNLI, QQP.\n\n**INGENIOUS vs vanilla-BERT(validation loss comparison):**\n\n>Can you share a pre-training performance comparison between fully pre-trained BERT (e.g., Bert-base) and INGENIOUS pre-trained BERT, without downstream fine-tuning? One suggestion could be something like testing accuracy/loss on the original validation/test set.\n\nGreat suggestion! Following is a table with validation set losses of INGENIOUS and vanilla-BERT during the course of pre-training: \n\n\n\n| Training Steps | 100K | 150K | 200K | 250K | 300K | 350K | 400K | 450K | 500K | 550K | 600K | 650K | 700K | 750K | 800K | 850K | 900K | 950K | 1000K |\n| ---- | ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----| ----|\n| vanilla BERT | 2.29 | 2.07 | 1.99 | 1.92 | 1.88 | 1.86 | 1.83 | 1.8 | 1.79 | 1.78 | 1.76 | 1.74 | 1.73 | 1.73 | 1.71 | 1.7 | 1.69 | 1.69 | 1.68 | 1.68 |\n| INGENIOUS BERT | 2.29 | 2.09 | 2.01 | 1.95 | 1.91 | 1.88 | 1.85 | 1.85 | 1.82 | 1.8 | 1.79 | 1.78 | 1.76 | 1.76 | 1.74 | 1.74 | 1.72 | 1.71 | 1.7 | 1.7 |\n\nAs can be seen from the above table, INGENIOUS achieves almost similar validation loss as compared to vanilla BERT.\n\n\n**Releasing the subsets selected by INGENIOUS**\n\n>Will you release the selected subsets that yield your best models?\n\nYes, definitely! Our code is already public on anonymous github. We will release the pre-processed dataset that is used and the selected subsets in the form of a Google Drive link due to the size. (Not releasing the link now to respect the anonymity guidelines)\n\n[1] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C. and Carlini, N., 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Firstly, the proposed method, INGENIOUS, demonstrates an impressive performance. After fine-tuning, the model pre-trained using INGENIOUS-selected subsets achieves nearly identical performance compared to models pre-trained with the complete dataset, and even surpasses them with reduced training costs (as seen in Table 3).\n\nThen, the experimental design is comprehensive, covering comparisons across various architectures, previous sampling methods, and different downstream tasks. The evaluation is also multi-faceted, assessing both performance and cost considerations.\n\nAdditionally, the paper maintains a well-organized structure. The experiment and analysis sections are good in depth and detail, though the methodology section could benefit from a more detailed elaboration. ", "reasons_to_reject": "First, I’m confused by Figures 3 and 4. In these illustrations, INGENIOUS appears to consistently and significantly outperform the baseline BERT model which undergoes full pre-training, even when it is trained up to 1 million pre-training steps. However, based on the context along with Tables 1&2 and the detailed Table 6 in Appendix F, it becomes evident that the most proficient model pre-trained with INGENIOUS achieves a 98.6% performance compared to the fully pre-trained BERT. This contrast in results is confusing. And if Figures 3&4 is sensible, what is the rationale behind the phenomenon that pre-training on selected subsets can surpass the performance of a fully pre-trained model? Does this suggest that the original pre-training corpus contains harmful noise that interferes with learning?\n\nSecondly,  for a thorough evaluation of INGENIOUS' effectiveness in selecting valuable subsets for pre-training, it is necessary to see a comparison between the pre-training performance of a model using INGENIOUS-selected data and one using the entire dataset, before any downstream fine-tuning.  The clean pre-training performance gains of INGENIOUS-selected data can demonstrate the evaluation process better.\n", "questions_to_authors": "Can you provide a more detailed explanation of how figures 3 and 4 were generated? How exactly are the values in y-axis computed? Additionally, can you clarify which specific glue task was used in these figures?\n\nDo you have any insights into why training on selected data can yield better results compared to a fully pre-trained model? Are there factors beyond noise reduction?\n\nCan you share a pre-training performance comparison between fully pre-trained BERT (e.g., Bert-base) and INGENIOUS pre-trained BERT, without downstream fine-tuning? One suggestion could be something like testing accuracy/loss on the original validation/test set.\n\nWill it be possible for you to share a comparison of the pre-training performance between a fully pre-trained BERT (such as Bert-base) and an INGENIOUS pre-trained BERT, without any downstream fine-tuning? One suggestion on evaluation metrics could be testing accuracy or loss on the original validation/test set.\n\nWill you release the selected subsets that yield your best models?\n\nI will raise my scores if the above concerns can be addressed/clarified by the rebuttal. ", "paper_topic_and_main_contributions": "This paper proposes INGENIOUS, a framework for iteratively selecting informative subsets of the pre-training corpus by optimizing a submodular maximization problem wherein a similarity-based facility location acts as the submodular function. By pursuing this objective, the framework identifies a collection of diverse samples, each representing similar samples within the corpus.\n\nThe authors empirically validate the approach by pre-training language models on the selected subsets. These models are fine-tuned across various glue downstream tasks, which yield performance comparable to models that were pre-trained on the entire dataset and fine-tuned for the same tasks. The INGENIOUS-pretrained models display increased efficiency, achieving the later-stage performance of baseline models at an earlier time.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "e2q0mbEKGD", "reviewer": "Reviewer_m6xK", "summary": "", "questions": "A. Would you provide some examples of informative texts sampled by the proposed method? What differences do exist between the informative texts and the others?\n\nB. In Sec. 4.4, the proposed method achieves final performance on par with vanilla BERT. What does happen if a model is trained for longer epochs? It seems that the performance of the proposed method converges while vanilla BERT keeps improving.\n\nC. The paper reported that 25% of training corpus is optimal, outperforming vanilla BERT when fully trained. Would you explain why 75% of training corpus deteriorates the performance of language models?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging remarks and for appreciating the importance of the problem and the experimental rigor. We appreciate the inquisitive questions which we answer below.\n\n**Determining Useful Information for Text Datasets:**\n\n>Although extracting data subsets work well in computer vision tasks, texts have different properties compared with images in nature. The paper focuses on the size of datasets and lacks discussion about the different modality.\n\nExactly! Text and image data have very different characteristics and this is precisely our argument in the paper that approaches used in Vision have to be adapted to make them work for text data. We focused on selecting smaller subsets for efficiency and also presented discussion in Section 4.3 that delves into details of text-specific characteristics that help us adapt subset selection for LM training. Specifically,  our findings indicate that the ninth layer of BERT-base excels in selecting informative subsets crucial for model pre-training (Table 3)and is more advantageous than initial or later layers. In Section (4.3), we discuss these findings in light of the seminal study[1], which showed that the middle layers (6-9) of BERT predominantly capture syntactic nuances, with layers 8 and 9 showing enhanced subject-verb agreement. Further, the initial layers, like layer 3, focus on word order, while the later layers are tailored to capture the specifics of downstream tasks. It is also worth noting that semantic information is permeated throughout the network. Thus, we present a thorough ablation of what characteristics of language data are beneficial for the subset selection task in contrast to the image data.\n\n**Size of the models:**\n\n>Most experiments are conducted on models with ~110M parameters. It is unclear whether the proposed method consistently improves the efficiency with scaled models.\n\nAs acknowledged in the Limitations section of the paper, we do note that our experiments are performed on moderate-sized language models owing to resource limitations. However, do note that we trained these models from scratch and not just fine-tuned. Given the scale and rigor of our experiments (multiple settings, ablations, and multiple runs for each experiment), even training the three models used in the paper is a significant endeavor. To account for our lack of financial and compute resources (to be honest, very few institutions, except for the largest mega corporations, can afford to train large models from scratch), we ensured a rigorous evaluation scheme to confirm the validity and efficacy of our approach, including thorough ablation studies and multiple runs (20 times).\n\n**Examples of text selected in the subsets:**\n\n>Would you provide some examples of informative texts sampled by the proposed method? What differences do exist between the informative texts and the others?\n\nGreat question! We summarize the three types of redundancies that we found in our analysis of selected subsets here, and will be adding these to the Appendix as well.\n\n**Type-I:** We discovered cases in which the exact same information is conveyed by multiple sentences in different documents, in which case only one of the sentences is assigned a high score to be included in the training sample.\nExample:\n\nSentence 1: \"separate sovereign countries but acted as a single bloc in foreign policy and security issues. the proposed union was being discussed by a joint scandinavian committee during the winter of 1948 – 1949, but the cold war tension between the united states and the soviet union, and preparations for a western alliance that would result in the north atlantic treaty overshadowed the effort. when it became\"\n\nSentence 2: \"they would remain separate sovereign countries but act as a single block in foreign policy and security issues. the proposed union was discussed by a joint scandinavian committee during the winter of 1948 – 1949, but in the end the cold war tension between the united states and the soviet union and preparations for a western alliance that would result in\"\n\n**Type-II:** We discovered many cases (mostly in the Bookcorpus dataset) where there are duplicates in the corpus, in which case only one of the sentence is assigned a high score for inclusion.\n\nExample:\n\nSentence-1: “after we'd been handed our menus. ` ` i always get the frozen hot chocolate.'' ` ` frozen hot chocolate? it's really a thing? i thought they just made that up.'' ` ` no,'' i said, pointing to the spot on her menu. ` ` see? it's right there.'' ` ` so, do you order anything else?'' ` ` cake.'' she looked at my deadpanned face and laughed. ` ` so, we're”\n\nSentence-2: “what's good here?'' mia asked me after we'd been handed our menus. ` ` i always get the frozen hot chocolate.'' ` ` frozen hot chocolate? it's really a thing? i thought they just made that up.'' ` ` no,'' i said, pointing to the spot on her menu. ` ` see? it's right there.”\n\n**Type-III:** We discovered a few recurring patterns of templates in the corpus, in which case INGENIOUS assigns a high score to an example e1 of the template T and a low score to other examples e2, e3,... of the same template T. We posit that this way the redundancy in language is being minimized.\n \nExample:\n\nSentence-1: “according to the united states census bureau, the village has a total area of, all land. demographics 2010 census as of the census of 2010, there were 377 people, 159 households, and 101 families residing in the village. the population density was. there were 176 housing units at”\n\nSentence-2: “according to the united states census bureau, the village has a total area of, all land. demographics 2010 census as of the census of 2010, there were 801 people, 323 households, and 225 families living in the village. the population density was. there were 358 housing units at”\n\nWe will also release the complete pre-processed dataset used and the selected subsets and add many more examples like the above in the Appendix.\n\n**Convergence of Bio-BERT in Tandem with INGENIOUS:**\n\n>In Sec. 4.4, the proposed method achieves final performance on par with vanilla BERT. What does happen if a model is trained for longer epochs? It seems that the performance of the proposed method converges while vanilla BERT keeps improving.\n\nIn Section 4.4, we trained both the standard and the INGENIOUS versions for the full number of epochs as recommended by the original study. INGENIOUS displayed a **faster** rate of convergence, achieving the performance of the fully-trained standard model in approximately **10** hours, compared to the latter's 14 hours. This is the key insight from the result of this experiment. Although the vanilla Bio-BERT model seems not to have converged yet, we expect that with extended training, it too will stabilize, as suggested by the trends observed in its last three checkpoints. We plan to incorporate further experiments with model training for a greater number of steps in the next version of the paper.\n\n**Subset Size Explanation in INGENIOUS:**\n\n>The paper reported that 25% of training corpus is optimal, outperforming vanilla BERT when fully trained. Would you explain why 75% of training corpus deteriorates the performance of language models?\n\nA subtle point to note here. In INGENIOUS, when we mention selecting a 25% subset, it doesn't imply that only 25% of the dataset is used for model training. As illustrated in Figure 2 of the paper and in the pseudo-code in the Appendix C, after every 25K steps, we consistently sample a fresh subset that constitutes 25% of the data, based on the updated probability distribution. This method ensures that the model consistently engages with nearly the entire dataset over time. Moreover, the learned probability distribution prioritizes data points deemed more informative through submodular functions. As a result, more informative data points are sampled more often, while less significant points are sampled less frequently. We argue that in contrast to INGENIOUS, with no subset selection in place in the Vanilla BERT training, the model spends much time on less informative and noisy data points, leading to sub-optimal performance.\n\n[1] Rogers, A., Kovaleva, O. and Rumshisky, A., 2021. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, pp.842-866.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper addresses an important problem of the pre-training cost of language models for efficient NLP.\n\nB. The paper presents a novel approach to remove redundant texts in pre-training corpora in NLP domains.\n\nC. Experimental results that language models trained with informative subsets outperform vanilla models in terms of convergence speed and final performance.", "reasons_to_reject": "A. Although extracting data subsets work well in computer vision tasks, texts have different properties compared with images in nature. The paper focuses on the size of datasets and lacks discussion about the different modality.\n\nB. Most experiments are conducted on models with ~110M parameters. It is unclear whether the proposed method consistently improves the efficiency with scaled models.\n\nC. Lack of analysis and discussion leaves several questions about the reasons of performance improvements.\n", "questions_to_authors": "A. Would you provide some examples of informative texts sampled by the proposed method? What differences do exist between the informative texts and the others?\n\nB. In Sec. 4.4, the proposed method achieves final performance on par with vanilla BERT. What does happen if a model is trained for longer epochs? It seems that the performance of the proposed method converges while vanilla BERT keeps improving.\n\nC. The paper reported that 25% of training corpus is optimal, outperforming vanilla BERT when fully trained. Would you explain why 75% of training corpus deteriorates the performance of language models?\n", "paper_topic_and_main_contributions": "This paper addresses the problem of a large computational cost of pre-training language models due to the size of pre-training corpora. To reduce the pre-training cost, the authors propose to extract a highly informative subset of pre-training corpus. The experimental results in the paper show that the proposed method outperforms early stopping and random extraction methods in terms of fine-tuned GLUE scores.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Yy52uMo34W", "reviewer": "Reviewer_NYV4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our proposed training strategy for the crucial task of efficient LM training.\n\nWe believe there has been a misunderstanding regarding the fundamental argument of the paper. Our key goal is **efficient** training of the LMs. Our proposal is to identify and remove redundancies in the training data. And our rigorous experiments show that handling redundancies and noise and feeding high-quality samples to the model results in faster learning. While bias, toxicity, and values are significant factors to consider for LM training, they are orthogonal to the focus of current work -- efficient training. Our focus is not on measuring or creating high-quality datasets for training.\n\nFurther, our evaluations are not just limited to the GLUE benchmark. While GLUE is one of the most commonly used benchmarks comprising multiple tasks for LM evaluation, we go beyond that and evaluate the knowledge retention capabilities by the LAMA benchmark (Section 4.2), study the performance for domain-specific applications (Section 4.4), perform thorough ablation studies (Section 4.1, 4.3), and use different types of LMs (Section 4.5). Moreover, note that we report the numbers averaged over 20 runs to account for the effects of any random seed. Very rarely, if ever, such stringent evaluation is reported for LM training.\n\nRegarding the model size, as already acknowledged in the \"Limitations\" section of our paper, we used BERT and GPT-2 small models due to resource constraints. However, do note that we were training these models from scratch and not just fine-tuning. To account for our lack of financial and compute resources (to be honest, very few academic institutions, if any, can afford to train large models from scratch), we ensured a rigorous evaluation scheme to confirm the validity and efficacy of our approach, including thorough ablation studies and multiple runs (as also noted by all the other reviewers).\n\nWe have made all our implementations open and free for others to build upon, and we remain confident that the performance enhancements observed in our approach will similarly manifest in larger models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper investigates the issue of resource consumption in training large language models (LLMs) by sampling high-quality information from the training dataset to ensure efficient learning of effective information. This topic is crucial for the efficient training of LLMs. The article provides a clear training strategy and validates its effectiveness on multiple models (but not large enough), which is inspiring.", "reasons_to_reject": "While this paper's research is insightful, it has some limitations. The current issues with training data quality for LLMs go beyond efficiency and information quantity, including concerns about values, toxicity, and biases. Additionally, the paper's validation was limited to BERT and GPT2-small models and evaluated using GLUE metrics, which may not be sufficient to support the conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for improving the training efficiency of pre-trained language models (PTLMs) by selecting highly informative subsets of the training data. The framework employs submodular optimization to select representative subsets of the training corpora and can be applied to efficiently train multiple PTLMs using only a fraction of data. The resulting models achieve up to 90% of the performance of fully-trained models. The contribution of this paper lies in the development of a new training framework that significantly improves the training efficiency of PTLMs while maintaining downstream performance.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper under review offers a novel method, INGENIOUS, focused on the efficient selection of a subset of data for pretraining language models. This approach ensures performance comparable to training on the entire dataset. Leveraging a submodular maximization problem with a similarity-based facility location, the technique identifies representative samples from a broader corpus. When assessed, models pretrained on these subsets showcased remarkable efficiency, often matching or even surpassing models trained on full datasets in downstream tasks. The research stands out due to its rigorous experimental design, innovation in methodology, and addressing a critical issue of training efficiency. While there were concerns related to the clarity of specific results, the main focus on dataset size over other modalities, and the breadth of model validation, it is important to note that some of these criticisms may not directly align with the paper's primary focus.

Pros:
1. The INGENIOUS approach represents a significant advancement in efficient language model pretraining by extracting crucial information subsets.
2. Extensive experimental design appreciated by multiple reviewers, encompassing various architectures, sampling methods, and evaluations beyond just GLUE scores.
3. The research offers profound insights into a pivotal area, potentially paving the way for future investigations.

Cons:
1. Some ambiguities in results, especially related to the advantages of subsetting over full datasets.
2. Results on smaller models may not generalize to larger models
Decision: Accept-Findings