Title: XYZ Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Venue: NeurIPS 2023
Accepted: no
Keywords: data efficiency

training efficiency

foundation model

pretraining

finetuning

language model

GPT-3

BERT

ViT
Abstract: Recent advances on deep learning models come at the price of formidable training cost. The increasing model size is one of the root causes, but another less-emphasized fact is that data scale is actually increasing at a similar speed as model scale, and the training cost is proportional to both of them. Compared to the rapidly evolving model architecture, how to efficiently use the training data (especially for the expensive foundation model pretraining) is both less explored and difficult to realize due to the lack of a convenient framework that focus on data efficiency capabilities. To this end, we present XYZ Data Efficiency, a framework that makes better use of data, increases training efficiency, and improves model quality. Specifically, we propose and combine two data efficiency techniques: efficient data sampling via a general curriculum learning library, and efficient data routing via a novel random layerwise token dropping technique. For GPT-3 1.3B language model pretraining, our work achieves 12.5x less data/time/cost (`$`3.7K if rent on Azure), while still maintaining 95% of model quality compared to baseline with full data and cost (`$`46.3K). For GPT-3 1.3B and BERT-large pretraining, our work can also achieve the same model quality with up to 2x less data/time/cost, or achieve better model quality under same data/time/cost. XYZ Data Efficiency is easy to use and tune, enabling us to easily apply it and verify its benefit on additional tasks including GPT-3 MoE model pretraining and small-scale GPT-2/ViT finetuning.

Abstract

Recent advances on deep learning models come at the price of formidable training cost. The increasing model size is one of the root causes, but another lessemphasized fact is that data scale is actually increasing at a similar speed as model scale, and the training cost is proportional to both of them. Compared to the rapidly evolving model architecture, how to efficiently use the training data (especially for the expensive foundation model pretraining) is both less explored and difficult to realize due to the lack of a convenient framework that focus on data efficiency capabilities. To this end, we present XYZ Data Efficiency, a framework that makes better use of data, increases training efficiency, and improves model quality. Specifically, we propose and combine two data efficiency techniques: efficient data sampling via a general curriculum learning library, and efficient data routing via a novel random layerwise token dropping technique. For GPT-3 1.3B language model pretraining, our work achieves 12.5x less data/time/cost ($3.7K if rent on Azure), while still maintaining 95% of model quality compared to baseline with full data and cost ($46.3K). For GPT-3 1.3B and BERT-large pretraining, our work can also achieve the same model quality with up to 2x less data/time/cost, or achieve better model quality under same data/time/cost. XYZ Data Efficiency is easy to use and tune, enabling us to easily apply it and verify its benefit on additional tasks including GPT-3 MoE model pretraining and small-scale GPT-2/ViT finetuning.

1 Introduction

Recently, large-scale deep learning models are empowering us to achieve more in many ways, such as code generation [17] and text-to-image generation [40, 41]. To keep improving the service quality, deep learning model architecture evolves rapidly, and the model size is also growing at a tremendous speed. The increasing model size leads to unprecedented training cost (especially for foundation model pretraining), which recently grows to 2 months on thousands of GPUs/TPUs [47, 9]. On the other hand, a less-emphasized perspective is that data scale is actually increasing at a similar speed as

BERT

(2018)

GPT-2

(2019)

GPT-3

(2020)

BLOOM

(2022)

PaLM

(2022)

0

200

400

600

800

Billion

Data Scale (Billion)

model scale, and the training cost is proportional to both of them . As plotted in Fig. 1, for several representative language models in the last 5 years both the model and data scales increase at a similar speed. Recent works including Chinchilla [20] and PaLM 2 [18] emphasize the need of increasing data scale at an even faster speed. This demonstrates the importance of improving data efficiency: achieve same model quality with less data and reduced training cost, or achieve better model quality with the same amount of data and similar training cost.

There are two popular research directions among existing data efficiency techniques: Data sampling 37 techniques aim to improve the convergence speed by sampling the most suitable next data batch from 38 the whole data pool; Data routing techniques aim to reduce the computation by routing each data to 39 only a subset of the model components. These techniques improve data and training efficiency, but 40 existing solutions have several limitations: 41

42

43

44

45

46

47

Techniques like curriculum learning improves data efficiency by indexing and sampling training data based on certain difficulty metric [3], and it is recently proved effective on large-scale pretraining tasks [29]. However, implementing different CL strategies for different user tasks can require a lot of code-refactoring, which is time-consuming and error-prone. In addition, existing implementations have less consideration on scalability, which makes it difficult to analyze and index large-scale training data based on different difficulty metrics.

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

Existing data routing techniques such as token drop/bypass/pruning were mostly designed for inference and inapplicable to training. TokenBypass [21], to our knowledge the only data routing technique for foundation model pretraining, skips the compute of part of the input tokens at some middle layers during BERT pretraining, reducing pretraining cost while maintaining model quality. However, it requires several special implementations that may only work for the tested BERT pretraining case, such as the importance score-based token dropping decisions and the whitelist for special tokens. This could limit the possibility and benefit of applying it to other cases.

Although promising data efficiency solutions have been proposed independently, combining multiple methods together for the best outcome is still a laborious process, requiring changes in multiple places in the training pipeline: data loader, data sampler, model architecture, etc. Another challenge is that existing techniques usually add additional hyperparameters but without a clear and low-cost tuning strategy.

To address these above challenges, we present XYZ Data Efficiency, a framework that makes better use of data, increases training efficiency, and improves model quality. Specifically, XYZ Data Efficiency demonstrates the following contributions:

Efficient data sampling via general curriculum learning library. We present a general curriculum learning (CL) library that is both scalable and customizable: it includes a map-reduce based data analyzer that enables scalable analysis and indexing of massive data based on any possible CL metric; it includes a general CL-based data sampler and loader design for users to apply any customized CL strategies. Using this library, we are able to thoroughly explore different CL strategies for GPT-3 1.3B and BERT-large pretraining, and identify the best solution that provides better data and training efficiency than existing CL solution. This library (and the whole XYZ Data Efficiency framework) has been open sourced in a deep learning acceleration library (name hidden for anonymity) that is fully compatible with PyTorch. This will benefit the whole community as a useful tool to apply curriculum learning to their own training tasks.

Efficient data routing via random layerwise token dropping. We present a novel data routing technique called random layerwise token dropping (random-LTD) to skip the computation of a subset of the input tokens at all middle layers. Random-LTD employs a simple yet effective routing strategy and requires minimal model architecture change. It is very flexible to apply random-LTD to various tasks (GPT-3/GPT-3 MoE/BERT pretraining and GPT/ViT finetuning) which the SOTA technique (TokenBypass) does not explore or provides less improvement.

An easy to use/tune framework that maximizes data/training efficiency. XYZ Data Efficiency seamlessly composes the two proposed techniques, and only requires minimal changes on user side. To our knowledge, we are the first to demonstrate that composing data sampling and routing techniques can lead to even better data/training efficiency, especially for foundation model pretraining: For GPT-3 1.3B pretraining, Fig. 2 shows that our approach provides better model quality at all cost budgets, advancing the whole cost-quality Pareto frontier. In particular, we achieve up to 12.5x data/time/cost saving while still maintaining 95% of the model quality (zeroshot eval accuracy) compared to the baseline with full data, while baseline can only maintain 91% of the model quality, a 1.8x higher quality degradation. Based on measured training time, 12.5x would be a cost reduction from $46.3K to $3.7K if renting similar hardware on Azure [2], greatly democratizing research and usage of foundation models for AI community. For GPT-3 1.3B and BERT-large pretraining, we can also achieve up to 2x data and 2x time saving together with better or similar model quality as compared to the baseline training with full data, greatly surpassing state-of-the-art data efficiency solutions as summarized in Tab. 1. Both techniques under our framework are easy to use and tune, and we include a low-cost tuning strategy and a summarized usage guidelines. This enables us to easily apply proposed work and verify its

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

1%

$463

2%

$925

4%

$1850

8%

$3.7K

16%

$7.4K

32%

$14.8K

50%

$23.1K

100%

$46.3K

Consumed data and cost (log scale)

80

85

90

95

100

105

Model quality (%)

Baseline

XYZ Data Efficiency

benefits on additional workloads including GPT-3 Mixture-of-Experts (MoE) model pretraining and small-scale GPT-2/ViT model finetuning.

2 Background and Related Works

Data sampling. For deep learning, the most common data sampling method for minibatch stochastic gradient descent is uniform sampling, where at each step a batch of data is drawn uniformly at random from the whole training data. However, it's potentially beneficial to focus on different kinds of data at different training stages. One example is the curriculum learning technique [3] which aims to improve training convergence speed by presenting relatively easier or simpler examples earlier during training. Building a curriculum learning solution usually requires two components: the difficulty metric (i.e., how to quantify the difficulty of each data sample) and the pacing function (i.e., how to decide the difficulty range when sampling next training data batch). In the NLP area, curriculum learning has been applied on small-scale one-stage tasks and downstream finetuning tasks, such as neural machine translation (NMT) [25, 6, 62, 36, 63] and natural language understanding (NLU) [42, 43, 48, 55]. There are also a few works that explore curriculum learning for language model pretraining [37, 61, 8, 29]. However, one common limitation among existing works is that there does not exist a scalable and customizable curriculum learning library, making it difficult to analyze large-scale data and explore custom difficulty metrics/pacing functions. One evidence is that most of the curriculum learning works for language model pretraining only focus on the sequence length metric due to the difficulty of exploring other metrics on the huge pretraining dataset.

Data routing. In common deep learning training, the model is considered as a whole and all sampled data will be routed to all model components. However, it's potentially beneficial to route each data sample to only a subset of model components, improving the training efficiency. One direction of efficient data routing is to add data bypassing/skipping capability to existing model architectures such as Transformer. Transformer [49] architecture is a stack of transformer layers, each of which has two main ingredients, i.e., the multi-head attention (MHA) and the feed-forward connection network (FFC). Suppose the transformer has l layers denoted as L 1 , . . . , L l . Let X i ∈ R s × d be the output tensor of i -th transformer layer, and x 0 be the input (after embedding) of the transformer. Here s is the sequence length and d is the hidden dimension.

Several token dropping/bypassing/pruning techniques [24, 19, 23, 38, 53] were proposed for BERT inference to reduce the computational overhead, but they are not practical for training. In these works, if a token i ( X j,i ) is decided to be dropped at layer j ( L j ), the compute cost of this token through all remaining layers ( L k where k > j ) is eliminated. As such, the sequence length s i of the i -th layer's input X i -1 will be a non-increasing array, i.e., s 0 ≥ s 1 ... ≥ s l . However, such a configuration has been shown instability for adaptive token-dropping inference [23]. Therefore, [23] utilize the sandwich rule and distillation from [58] to stabilize training and boost accuracy. But these two methods also significantly increase the training cost. Thus, such techniques cannot be applied to speed up the pretraining procedure.

Recently, TokenBypass [21] enabled token dropping for BERT pretraining. It uses several importance scores/metrics to determine the dropped tokens (token frequency and cumulative loss). It proposed two main mechanisms to overcome the training instability issue: (1) the sandwich token dropping rule, where the first ( L 1 to L i ) and the last few BERT layers ( L l -j to L l ) capture all tokens (no token dropping) and only bypass s ′ ≤ s tokens from L i to L l -j middle layers. Particularly, the authors (only) test on the encoder transformer (12-layer BERTbase and 24-layer BERTlarge), and let i = l/ 2 -1 , j = 1 , s ′ = s/ 2 . (2) special token treatment, where special tokens (e.g., [MASK], [CLS], [SEP] ) are never dropped. Compared to TokenBypass, our random-LTD (1) does not require importance score metric, special token treatment, or the sandwich token dropping rule, which dramatically

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

reduces the manual design effort; (2) has been broadly tested on GPT-3/BERT pretraining tasks and GPT-2/ViT finetuning tasks, providing better data/training efficiency than TokenBypass.

3 Design

At high-level, the proposed XYZ Data Efficiency framework has two components as shown in Fig. 3: First we have efficient data sampling, where instead of the baseline's random sampling, we aim to sample the most suitable next data batch from the whole data pool by a general curriculum learning (CL) library. Second we have efficient data routing, where instead of passing all input data to all model components, we aim to efficiently route each data through different components of model by leveraging the proposed random layerwise token dropping (random-LTD) technique. This section presents the design of the two techniques, how we compose them, together with a low-cost tuning strategy and a summarized usage guidelines.

3.1 Efficient data sampling via curriculum learning

To solve the limitations of existing CL solutions as described in previous sections, we design and implement a general curriculum learning library emphasizing the scalability and customizability. It

consists of three components as shown in top part of Fig. 3. First we use a data analyzer to perform the offline CPU-only data analysis which indexes the whole data pool based on any difficulty metric, which could be the sequence length, the vocabulary rarity, or anything defined by user. This data analyzer employs a Map-Reduce scheme: During the Map stage, user provides a function that computes the desired difficulty metric, the raw training dataset, and other configurations such as number of CPU nodes and number of threads per node. Then the data analyzer will automatically splits the dataset based on number of workers, compute the difficulty values in a batched fashion, and write the results to two indexes: one index maps each data sample to its difficulty value, and another index maps each distinct difficulty value to the corresponding samples. During the Reduce stage, the data analyzer will merge the index files produced by all workers. This Map-Reduce scheme is necessary since the training data could be huge thus has to be distributed. For instance, we have 173 million data samples (each with sequence length 2048) for GPT-3 pretraining and 2.5 billion data samples (each with sequence length ⩽ 512) for BERT pretraining. To reduce the memory overhead when analyzing the huge dataset, we write the index files as numpy memory-mapped files. Using this data analyzer we are able to efficiently analyze GPT-3 and BERT pretraining data based on various difficulty metrics. Using 40 CPU threads on a single node with AMD EPYC 7V12 64-Core Processor, we can finish the analysis on one metric within 3/80 hours for GPT-3/BERT data, respectively.

Next, during training, the curriculum scheduler will determine the difficulty threshold for the current step based on a pacing function such as linear, rooted, or any strategy provided by user. Then the data sampler will sample the data with desired difficulty from the indexed data pool. To apply the proposed CL solution to a existing training pipeline, user just need to call an API and provide the raw training data, the difficulty metric index (computed in the offline analysis), and the pacing function configurations. Our framework will then provide a curriculum learing-based data loader that users can simply iterate at each step. Using our CL library for GPT-3/BERT pretraining, we are able to easily analyze and index the huge training data based on 7 difficulty metrics:

Truncation-based sequence length (seqtru), for GPT and BERT. This metric starts with shorter data samples and gradually increases the sequence length during training. To change the sequence length, this metric will truncate the sequences (from the end of sequence) while keeping the number of samples unchanged, thus the number of tokens will decrease. This metric is recently applied to GPT-2 and GPT-3 models and demonstrate decent training efficiency gains [29].

Reshape-based sequence length (seqres), for GPT. This metric is similar to seqtru metric, but instead of truncating we break the original sequences into segments based on the desired new sequence length. Thus we are essentially 'reshaping' the input tensor into more samples and shorter lengths. This metric is proposed in MosaicML Composer as a variant of the seqtru metric [33], but their documentation does not describe which way provides better model quality. We don't apply the seqres to BERT case because unlike GPT data where all tokens are valid, BERT input sequences only include two natural sentences thus each sequence has different 'effective sequence length' and then padded to 512. If we simply 'reshape' BERT sequences, some of the new short sequences may only contain padding tokens.

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

Reorder-based sequence length (seqreo), for BERT. This metric is similar to seqtru metric, but instead of truncating we adjust the sequence length by reordering the training data based on the 'effective sequence length' in BERT training data sequences.

Vocabulary rarity (voc), for GPT and BERT. This metric was proposed in a CL work for neural machine translation [36]. It computes the product of the unigram probabilities for each sequence by -∑ N k =1 log ( p ( w k )) where p ( w k ) is the vocabulary frequency (inside whole training data) of the k th word in the sequence. Lower value indicates that the sequence has more common vocabularies.

seqtru_voc, for GPT and BERT. seqres_voc, for GPT. seqreo_voc, for BERT. These 3 metrics are combinations of above metrics. For seqtru_voc and seqres_voc, we first reorder the training data based on voc metric, then apply seqtru or seqres as a kind of post-processing. For seqreo_voc, we treat it as a single new metric and index the data based on it.

Besides the difficulty metrics, another set of CL hyperparameters is the pacing function: the start and end difficulty ( d s and d e ), total number of CL steps ( T c ), and the kind of pacing function (linear, sqrt, or users can plug in any customized function to the proposed framework). For seqtru and seqres metrics, we set the d s and d e as value-based (e.g., d s = 80 , d e = 2048 ) since the possible values of these two metrics are continuous. For other metrics, we set d s and d e as percentile-based (e.g., d s = 1% , d e = 100% ) since the possible values of these metrics are discrete. For seqtru and seqres we use a linear pacing function ( d t = d s +( d e -d s ) × min ( t T c , 1) ) following the preivous work [29], while for seqreo and voc we use a sqrt pacing function ( d t = d s +( d e -d s ) × min (( t T c ) 0 . 5 , 1) ). This is because seqreo and voc will only sample from a subset of data pool before reaching the end difficulty, and previous work finds that in such case it's beneficial to use a sqrt function to avoid sampling too much easy samples at the beginning [36]. Sec. 3.3 includes low-cost tuning strategy and usage guidelines for our CL solutions.

3.2 Efficient data routing via random-LTD

Layerwise Token Dropping. Existing token dropping methods for inference and training either permanently drop tokens from the compute graph at intermediate layers, or at least make some tokens fully skip a consecutive series of middle layers (Sec. 2). However, several works [50, 31, 51] have shown that MHA focuses on different tokens at different layer depths and the attention map aligns with the dependency relation most strongly in the middle of transformer architectures. Therefore, fully skipping middle layers like TokenBypass [21] may hinder the learnability/generalization of the architecture during pretraining/inference. We conjecture that this might be why multiple first/last layers need to disable token bypassing and the special token treatment is needed.

In order to overcome this problem, we propose a layerwise token dropping (LTD) mechanism. Instead of fully bypassing same tokens over all middle layers, each transformer layer independently drops/retains its own set of tokens. In more detail, recall that the input of ( i +1) -th layer ( L i +1 ) is X i ∈ R s × d . Denote the dropped token index as J i = { j 1 , j 2 , ..., j a i } and the kept token index as K i = { k 1 , ..., k b i } such that a i + b i = s . We have J i ∪ K i = { 1 , 2 , 3 ..., s } and J i ∩ K i = ∅ for each layer. Meanwhile, for any two different layers L i 1 and L i 2 , J i 1 and J i 2 are independent, though the dropped ratios are the same. With this layerwise mechanism, each token rarely bypasses all middle layers. Thus, its dependency on other tokens can be captured by MHA.

Random Token Dropping. Various importance score-based metrics are used to determine the token dropping criterion. Most of them can be categorized in attention score-based or loss/frequency-based metrics. However, both of them introduce challenges that make LTD less practical: For attention score-based metrics, the compute cost for LTD is too high since the metric has to be calculated for every layer; For loss/frequency-based metrics, the accumulated loss or frequency would not be changed within the same iteration, which leads the dropped token to be the same for different layers, breaking the desired LTD mechanism. Instead of importance score, we propose to use purely random token dropping assignment and prove its effectiveness in all our experiments. For each transformer layer, we randomly (uniformly) select a small batch of tokens to proceed with the compute and drop the rest. In more details, assume M i = { m i (1) , m i (2) , ..., m i ( s ) } is a random shuffle of S = {1, 2, ..., s}. Then the dropped token set is J i = { m i (1) , m i (2) , ..., m i ( a i ) } for the input of L i +1 .

Random and Layerwise Token Dropping. Combining layerwise token dropping with random token dropping, we have our final random and layerwise token dropping method (random-LTD), which can efficiently apply token dropping for each individual layer and can capture the attention dependency of each token with other others in middle layers with high probability. As a result, our experiments on BERT pretraining confirm that random-LTD does not require and won't benefit from special token treatment used by the TokenBypass work, further reducing the implementation complexity. Fig. 5

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

Figure 4: random-LTD only requires a few lines of code. hs, khs, and dhs means the full input, kept input, and dropped input. 'gather', 'Layer', 'combine' means the functions for random selection, transformer layer, and order-preserved token combination.

(a) Baseline

(b) Our Random-LTD

Outputtokensof layerl

Outputtokensoflayerl

FFN & MHA

FFN & MHA

（-1）x

outputtokensof layeri

outputtokensoflayeri

FFN & MHA

FFN & MHA

compute

saving

Randomlyassigntokens.

+,

Inputtokensof layeri

Inputtokensoflayeri

...........................

......................

FFN & MHA

FFN & MHA

Input tokens of layer1

Inputtokensoflayer1

Figure 5: Transformer layers for baseline and randomLTD. The dash-line box is repeated by l -2 times.

presents the comparison between standard baseline training and random-LTD. The pseudo-code is given in Fig. 4. For each layer, random-LTD randomly selects (function 'gather') a subset of the tokens and feeds (function 'Layer') them into the transformer layer. Afterward, we combine (function 'combine') the output of transformer layer with the dropped tokens to recover the full sequence length in a order-preserved manner. Thus, the next layer still receives the full sequence and can repeat this process. To apply random-LTD to an existing training pipeline, user just needs to provide the module class name that they want to apply random-LTD (e.g., a TransformerLayer class). Then XYZ Data Efficiency will wrap the module with a new module that includes token dropping capability, and drop some of the input tokens for this module during training.

Layers without Token Dropping. While TokenBypass needs to keep half of the layers in full sequence length training, random-LTD has no such limitation. Thanks to its attention-capture feature, we can apply random-LTD to most of the transformer layers except the first and last layers, enabling further training efficiency gain. Our experiments show that keeping the first and last layers in full sequence length training usually leads to better performance since (1) the first layer directly connects to the embedding, and it can help refine the raw feature; (2) directly connected to the final prediction, the last layer provides a feature realignment for all tokens which could improve the model quality.

Monotonic Sequence Length Growth. In order to reduce the gradient variance introduced by random-LTD, we gradually increase the kept sequence length throughout training with a linear schedule (referred to as MSLG). Thus random-LTD has two hyperparameters similar to CL: starting from a sequence length r s which denotes the size of kept token set K i for each middle layer after dropping, random-LTD will gradually drop less tokens (following a linear function) and eventually stop dropping after T r steps. Our experiments show that MSLG provides better model quality than constant drop schedule under similar data/compute savings. Sec. 3.3 includes low-cost tuning strategy and usage guidelines for random-LTD.

3.3 Composing CL and random-LTD, tuning strategy, usage guidelines

CL and random-LTD are complementary: CL helps to sample the next data batch, and random-LTD helps to decide how to route each sampled data inside the model. XYZ Data Efficiency hides several complexities when composing the two techniques so that users can easily enjoy the compound benefit. As one example, some CL metrics would affect the actual sample sequence length, thus

Case

Guidelines

GPT-3

CL:

d

s

= 80/1% (seqtru/voc),

T

c

= 40% of baseline's total steps

pretraining

random-LTD:

r

s

= 128,

T

r

= 70% of baseline's total steps

BERT

CL:

d

s

= 128/5% (seqtru/voc),

T

c

= 50% of baseline's total steps

pretraining

random-LTD:

r

s

= 128,

T

r

= 100% of baseline's total steps

GPT-2

CL:

d

s

= 32 (seqres),

T

c

= 70% of baseline's total steps

finetuning

random-LTD:

r

s

= 128,

T

r

= 30% of baseline's total steps

ViT finetuning random-LTD:

r

s

= 32/66,

T

r

= 80% of baseline's total steps

inside our framework we make sure the random-LTD's token dropping mechanism is aware of this, and also adjust the calculation of number of actual consumed tokens which are affected by both techniques. This token consumption calculation is also critical to the learning rate schedule: previous CL work [29] finds that if a CL technique reduces the number of tokens on certain steps, it is desirable to use a learning rate decay schedule based on consumed tokens instead of consumed steps. This is because if baseline and CL use the same step-wise LR decay, it leads to much faster token-wise LR decay for CL which hurts model quality. In this work, we apply the token-based LR decay schedule for both CL and random-LTD. To our knowledge this is the first work to apply such LR schedule to token dropping/data routing techniques, and our experiments show that it does help improving random-LTD's performance. Our CL library's general data analyzer/sampler/loader and random-LTD's module wrapping design makes it easy to apply our framework to different model

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

training tasks. And the overall composibility of XYZ Data Efficiency enables us to leverage both data efficiency techniques and achieve even better data and training efficiency (Sec. 4).

Tuning Strategy and Usage Guidelines. Both CL and random-LTD only have two parameters that need user tuning: the starting CL difficulty/random-LTD seqlen ( d s / r s ), and the total CL/random-LTD steps ( T c / T r ). 1 And for both CL and random-LTD we find that it's possible to apply a low-cost tuning strategy proposed in previous CL work [29], where we perform binary search on a very small portion (e.g., 2%) of training to find the smallest d s / r s and largest T c / T r that don't trigger substantial validation loss fluctuations ('whether the perplexity value becomes larger than 1.3x of the previous best perplexity'). For GPT-2 finetuning, given the low training cost we also perform full training of 16 different CL/random-LTD settings which confirm that (1) the low-cost tuning strategy is able to find very good hyperparameters; (2) both CL and random-LTD are not sensitive to hyperparameter choices. Tab. 2 summarizes the usage guidelines based on our tuning results, which we believe can be directly applied to any similar models (at least as a very good starting point for any further tuning).

4 Evaluation

Weevaluate XYZ Data Efficiency by GPT-3/GPT-3 MoE/BERT pretraining and GPT-2/ViT finetuning. Appendix A.5 includes studies of the TokenBypass method on GPT finetuning and pretraining, further demonstrating the advantages of the proposed random-LTD method.

4.1 GPT-3 and GPT-3 MoE pretraining

We use the Pile public dataset [16] to perform the pretraining of GPT-3 1.3B [7] (24 layers, 2048 hidden size, 16 attention heads) model. We also pretrain a GPT-3 Mixture-of-Experts (MoE) 6.7B model (24 layers, 1024 hidden size, 16 attention heads, 64 experts on every other layer) following related work [39]. We then perform 0-shot and 10-shot evaluations on 19 tasks to evaluate the model quality of the pretrained models. Detailed experimental setup is described in Appendix A.1.

Among the 5 CL difficulty metrics we have for GPT-3 model, to find out which metric provides the best model quality we pretrain the model (with 100% data) 5 times (each with 1 CL metric). For seqtru metric (to our knowledge the only metric previously applied to GPT-3 pretraining), we tune the CL hyperparameters d s and T c based on the tuning strategy proposed in previous work [29]. Then for other metrics we use the same hyperparameters without retuning for fair comparison. As presented in Tab. 3 case 1 to 6, results show that all 5 CL metrics provide better model quality than baseline (except (4)CL_voc's 0-shot accuracy), and the (5)CL_seqtru_voc provides the best quality. The extensibility of our general CL library enables us to easily apply different CL metrics to this large-scale model pretraining with huge training data, and identify a new CL metric that provides better model quality than existing solution (2)CL_seqtru. Next we pretrain the model with 67% data, comparing the baseline and the best CL metric we find. Results show that the average 0-shot evaluation accuracy drops from 42.5 to 41.9 when baseline use less data (Tab. 3 case 1, 9). On the other hand, our CL solution (case 10) with 67% data is able to achieve better 0-shot and 10-shot accuracy than baseline with 100% data, achieving a 1.5x data and time saving.

When applying the proposed random-LTD technique, results show similar benefit as CL: better model quality when using 100% data (Tab. 3 case 7), and 1.5x data/time saving while maintaining model quality (case 11). To explore whether composing CL and random-LTD could achieve even better data and training efficiency, first we pretrain the model with both techniques under 100% training data. Results (case 5, 7, 8) show that using both techniques together further improves the model quality, demonstrating the benefit of composability by our framework. Next we pretrain the model with 50% data. Results (case 12 to 15) show that the baseline has worse 0-shot and 10-shot evaluation accuracy under 2x less data. Using CL or random-LTD can only recover part of the accuracy loss. On the other hand, the composed data efficiency solution is able to achieve the same or better accuracy results as baseline with 100% data, demonstrating a 2x data and 2x time saving.

To better understand how the proposed approach influences the model convergence, Fig. 6 plots the token-wise validation perplexity during pretraining. At the beginning of the training the proposed approach has slower convergence since we focus on easier/simpler data samples (CL) and drop more tokens (random-LTD) at the beginning. On the other hand, at the later stage of training the proposed approach is able to provide faster convergence speed than baseline. Our approach with 50% data is able to achieve similar final validation perplexity as baseline with 100% data (while baseline with 50% data cannot). Our approach with 100% data is able to achieve even better final validation perplexity which leads to the highest model quality.

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

(a) Begining 10% of training

0

7.5B

15B

22.5B

30B

Tokens

0

20

40

60

80

100

Validation PPL

Baseline 300B tokens

CL+rLTD 300B tokens

Baseline 150B tokens

CL+rLTD 150B tokens

(b) End of training

0

75B

150B 225B 300B

Tokens

5.50

5.75

6.00

6.25

6.50

6.75

7.00

7.25

7.50

Validation PPL

As presented in Sec. 1 and Fig. 2, we also compare baseline and proposed work when using even less data during GPT-3 pretraining (Detailed accuracy results can be found in Appendix A.1). Results show that our approach provides better model quality at all cost budgets, advancing the whole cost-quality Pareto frontier. In particular, we achieve up to 12.5x data/time/cost saving while still maintaining 95% of the model quality (zero-shot eval accuracy) compared to the baseline with full data. Based on measured training time, this would be a cost reduction from $46.3K to $3.7K if renting similar hardware on Azure [2], greatly democratizing research and usage of foundation models.

Recent work shows that applying Mixture-of-Experts (MoE) to GPT-style model pretraining could lead to better training efficiency while reaching similar model quality [39]. Thus we also pretrain a GPT-3 MoE 6.7B model (350M base model, together with 64 experts on every other layer) to compare baseline and proposed work. Results show that MoE model does achieve similar model quality with less training cost (Tab. 3 case 1, 16). On the other hand, our approach can further improve MoE model's model quality (case 17), confirming its broad applicability.

4.2 BERT-large pretraining

We use the Pile public dataset [16] to perform the pretraining of BERT-large [14] (24 layers, 1024 hidden size, 16 attention heads) model. We then perform GLUE finetuning to evaluate the model quality of the pretrained models. Detailed experimental setup is described in Appendix A.2.

Similar to the GPT-3 case, for CL we first investigate which metric (among 5 metrics we have for BERT model) provides the best model quality by pretraining the model (with 100% data) 5 times. Tab. 4 case 1 to 6 results show that 4 CL metrics provide better model quality than baseline, and the (5)CL_seqtru_voc provides the best quality. Next we pretrain with 67% data, comparing the baseline and our best CL metric. Results show that the GLUE score drops from 87.29 to 87.19 when baseline use less data (case 1, 9). On the other hand, our CL solution (case 10) with 67% data is able to achieve on-par GLUE score as baseline with 100% data, achieving a 1.5x data and time saving.

Tab. 4 case 7, 11, 14 present the case when applying random-LTD only. In terms of data saving random-LTD performs better than CL: it is able to achieve better GLUE score even with 2x less data than baseline (case 14), greatly surpassing the 1.33x data saving by the state-of-the-art TokenBypass method. However, the time saving is less than data saving because the token dropping mechanism adds a computation overhead at each step. Because the BERT-large is a smaller model than GPT-3 1.3B, this fixed latency overhead has a larger relative impact to the training time. However, even with this overhead random-LTD is still a more data/time-efficient solution than baseline/TokenBypass.

Tab. 4 case 8 and 15 present the case when applying both CL and random-LTD. At 50% data, the composed solution further improves the GLUE score from the CL/random-LTD-only cases (case 15), achieving a 2x data and 1.8x time saving while maintaining the GLUE score compared to baseline.

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

Another thing to note is that this case also has more time saving than the random-LTD-only case. This is because CL will first truncate the sequences before random-LTD perform the random token selection, and the shorter sequences reduces random-LTD's computation overhead. At 100% data, the composed solution (case 8) improves the GLUE score from the CL-only case, but is worse than the random-LTD-only case. One hypothesis is that for BERT pretraining when composing the two techniques it's preferable to reduce the CL duration, but exhaustively testing all hyperparameters is out of our resource budget and this work's scope.

4.3 GPT-2 and ViT finetuning

To verify the effectiveness of the proposed work on small-scale tasks, we apply our techniques to PTB finetuning task [30] for an already-pretrained GPT-2350M model checkpoint from Huggingface. Given the much smaller training cost, we focus on improving the model quality under the same amount of data. Detailed experimental setup and hyperparameter tuning are described in Appendix A.3. As shown in Tab. 5, seqres provides the best model quality among the 5 CL metrics (case 3), unlike the two pretraining tasks where the seqtru_voc is the best metric. This is because this finetuning task has much smaller batch size and number of tokens per batch. seqtru will reduce number of tokens per batch, which is less desirable under small-batch training. The small batch also prevents the voc metric to include sufficient number of samples with different vocabulary rarity, limiting its benefit. Applying random-LTD also improves the model quality (case 7). Both CL best metric and random-LTD are able to surpass baseline on all 16 combinations of their hyperparameters, demonstrating that they are not sensitive to the hyperparameter choices. At last we try another 4 seeds for the baseline, CL best metric, random-LTD, and the CL+random-LTD case. The composed CL+random-LTD case (case 8) further improves model quality from random-LTD-only case, but is only on-par with CL-only case. One hypothesis is that for tasks with such small-scale training data, it's less possible to further improve model quality by composing multiple data efficiency techniques.

We also try finetune the vision transformer (ViT) on both ImageNet (with a 12-layer pretrained ViT) and CIFAR (with a 24-layer pretrained ViT). Due to time/resource limitation, we only test random-LTD for this task. Detailed experimental setup is described in Appendix A.4. As presented in Tab. 6, results show that random-LTD is able to achieve 1.3-1.4x data savings while maintaining the model quality, demonstrating its broad applicability.

5 Conclusion

Unlike model scale which could reduce in the future with novel architecture, the amount of available training data will increase continuously and irreversibly. Language model pretraining is one of the first to reach a data scale that even training one full epoch is difficult, but sooner or later all machine learning tasks will face the same data efficiency challenge. In this work we propose the XYZ Data Efficiency framework, which demonstrate the power of composing 2 novel data efficiency techniques together. This enables us to achieve an up 12.5x data/time/cost saving (from $46.3K to $3.7K on Azure) while maintaining 95% of model quality for GPT-3 pretraining, an up to 2x saving for GPT-3 and BERT pretraining while maintaining 100% model quality, or to achieve even better model quality under similar data and cost. XYZ Data Efficiency is easy to use and tune, which enables us to apply it and verify the benefit on additional GPT-3 MoE pretraining and GPT-2/ViT finetuning tasks.

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

References

[1]

Ardavan Afshar, Ioakeim Perros, Evangelos E Papalexakis, Elizabeth Searles, Joyce Ho, and

Jimeng Sun. Copa: Constrained parafac2 for sparse & large datasets. In

Proceedings of the 27th

ACM International Conference on Information and Knowledge Management

2018.

Microsoft Azure. Pricing calculator. https://azure.microsoft.com/en-us/pricing/ calculator/ , 2023.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1533-1544, 2013.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , pages 7432-7439, 2020.

Ondˇ rej Bojar, Jindˇ rich Helcl, Tom Kocmi, Jindˇ rich Libovick` y, and Tomáš Musil. Results of the wmt17 neural mt training task. In Proceedings of the second conference on machine translation , pages 525-533, 2017.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020.

Daniel Campos. Curriculum learning for language modeling. arXiv preprint arXiv:2108.02170 , 2021.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies , 6(4):1-220, 2013.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

, pages 793-802,

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason 480 Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse 481 text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. 482

483

GitHub. Github copilot. https://github.com/features/copilot/ , 2021.

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

Google. Palm 2 technical report. https://ai.google/static/documents/ palm2techreport.pdf , 2023.

Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive wordvector elimination. In International Conference on Machine Learning , pages 3690-3699. PMLR, 2020.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3774-3784, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.

Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003 , 2020.

Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. arXiv preprint arXiv:2107.00910 , 2021.

Tom Kocmi and Ondˇ rej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017 , pages 379-386, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 , 2017.

Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning . Citeseer, 2012.

Conglong Li, Minjia Zhang, and Yuxiong He. The stability-efficiency dilemma: Investigating sequence length warmup for training gpt models. In Advances in Neural Information Processing Systems , 2022.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics , 19(2):313-330, 1993.

Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems , 32, 2019.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018.

MosaicML. Sequence length warmup, mosaicml composer. https://docs.mosaicml.com/ en/v0.11.1/method_cards/seq_length_warmup.html , 2022.

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 , 2019.

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016.

Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M Mitchell. Competence-based curriculum learning for neural machine translation. In NAACLHLT , 2019.

Ofir Press, Noah A Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012.15832 , 2020.

Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409 , 2021.

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-ofexperts inference and training to power next-generation ai scale. In International Conference on Machine Learning , pages 18332-18346. PMLR, 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Mrinmaya Sachan and Eric Xing. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 453-463, 2016.

Mrinmaya Sachan and Eric Xing. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 629-640, 2018.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8732-8740, 2020.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022.

Yi Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4922-4931, 2019.

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. arXiv preprint arXiv:1906.04284 , 2019.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 , 2019.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) , pages 97-110. IEEE, 2021.

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models , 2019.

Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6095-6104, 2020.

Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprint arXiv:1911.07176 , 2019.

Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466 , 2022.

Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1803-1811, 2019.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885 , 2018.

Wei Zhang, Wei Wei, Wen Wang, Lingling Jin, and Zheng Cao. Reducing bert computation by padding removal and curriculum learning. In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) , pages 90-92. IEEE, 2021.

Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv:1811.00739 , 2018.

Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum learning for domain adaptation in neural machine translation. In NAACL-HLT , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces XYZ Data Efficiency, a framework that combines curriculum learning and data routing techniques to improve data efficiency in training recent large models. In detail, authors implemented an efficient difficulty metric calculation method for large datasets by utilizing map-reduce, on top of which authors tried various curriculum learning techniques. Furthermore, by analyzing the limitations of existing data routing techniques, authors developed random-LTD that drops different tokens for different Transformer layers. Finally, authors demonstrate their XYZ Data Efficiency framework leads to achieve the baseline accuracy with less data, or achieve the better accuracy with the same amount of data.\n\nSTRENGTHS:\n1. Developing a general, efficient, and easy-to-use framework for curriculum learning for large models hasn't been explored before to the best of my knowledge. Given the high cost of training recent large models, such library can enable more active research in this field.\n2. Their data routing technique (i.e. random-LTD) is thoughtfully designed, and seems to improve the final performance of large Transformers across different tasks.\n\nWEAKNESSES:\nThis paper touches on multiple aspects of improving training efficiency of large models, especially from the data perspective, but none of them seem to meet the NeurIPS standard.\n1. Firstly, I would argue that random-LTD has almost nothing to do with data efficiency. It looks to me that random-LTD is actually closer to some regularization, particularly dropout [1]. Just because one can achieve the same performance with 2x less data with some regularization techniques (e.g. weight decay), calling them as a \"data efficiency trick\" cannot be justified in my opinion. Since it bypasses some computations of some tokens in some layers, I believe it's closer to a computation efficiency rather than data efficiency trick. From the systems perspective, however, random-LTD consistently hurts the overall throughput as shown in Table 3 & 4 somehow.\n2. I doubt the practical utility of map-reduce-based data difficulty calculation. The metrics used in this paper are all offline metrics in that they can be calculated only once before training and can be reused later. While I understand even such preprocessing can take a painfully long time with recent large datasets (e.g. Pile or C4), I don't think the value practitioners will get from this paper would be not so significant. If they can show their framework can be combined with some online or dynamic metrics (e.g. loss value for each token), I would be more convinced.\n3. XYZ Data Efficiency framework seems to lack the flexibility and/or modularity, a highly important aspect in the framework. For example, the use of certain CL techniques require specific LR schedules to enjoy the maximal improvement. This essentially means that users get a reduced flexibility in choosing their own LR schedulers. Such entanglement between LR schedulers and data sampling strategies can further harm the user experience when they want to implement their custom data sampling strategies. Overall, my impression is that XYZ Data Efficiency doesn't allow much flexibility for users to try out different things, but rather enforces users to follow their predefined pipeline, in this case, composed of random-LTD and several CL strategies.\n\nTo summarize, I find two major framing issues in this paper. First, while CL can be approached from the data efficiency perspective, I believe random-LTD (or data routing) has little to do with data efficiency. Second, I believe XYZ Data Efficiency is more of a combination of two algorithms (i.e. CL and random-LTD) rather than a some general framework due to its lack of flexibility and modularity.\n\n[1] Liu et al., Gating dropout: Communication-efficient regularization for sparsely activated transformers\n\nQUESTIONS:\n1. Can authors provide full train/validation loss curves (from beginning to end)? It's emphasized in the abstract that XYZ Data Efficiency achieves 95% of baseline performance with up to 12.5x less data/time. However, it's generally true that training significantly slows down as it gets closer to convergence. Therefore, it is also possible that the baseline training run also achieves 95% of its maximum performance in the very early stage and takes a very long time to improve final 5%. In this case, comparing time-to-95% would be a more fair metric. Such confusion can be easily resolved by having whole training loss curves.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author proposes XYZ data efficiency framework to improve the data/training efficiency in the foundation model training. The proposed framework mainly consists of two techniques, i.e., (1) the efficient data sampling via general curriculum learning library and (2) efficient data routing via random layer-wise token dropping.\n\nSTRENGTHS:\n* The CL-based library is open-sourced and compatible with PyTorch.\n\n* The proposed method achieves considerable training acceleration with minor or no accuracy degradation.\n\n* The author evaluates their method on several different models, including the large language models.\n\nWEAKNESSES:\n* Missing the full term of “CL” in the introduction section (line 44). The first explanation shows in line 64.\n\n* The author argues that the previous methods require changing the data loader, data sampler etc. However, the proposed method still needs to change then as well.\n\n* Besides the TokenBypass, there are also several data routing techniques for foundation model training. E.g., [1] [2]. The author should also include those works for discussion and comparison.\n\n[1] EViT: Expediting Vision Transformers via Token Reorganizations. ICLR 2022\n\n[2] Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision. AAAI 2023\n\n* I think the previous work [2] also explores the efficiency at the data sample-level and data routing level. So, it is probably not appropriate to claim the proposed method is “the first to demonstrate that composing data sampling and routing techniques can lead to even better data/training efficiency …”\n\n* As the author claims the proposed framework is easy-to-use and admits being open-sourced as one of the contributions of this work, it would be better if the author could submit the anonymous code with the supplementary materials.\n\n* Though the proposed method can achieve considerable overall acceleration, it would be great if the author can provide a discussion about the overhead of data sampling and routing part.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper draws inspiration from the observation of training costs increasing quadratically with data size, leading to a focus on enhancing data efficiency. To address this issue, the paper presents a framework that optimizes data utilization, improves training efficiency, and enhances model quality. The framework introduces efficient data sampling and data routing methods designed to overcome the challenges associated with data size. Extensive experimental results conducted on various foundation models confirm the effectiveness of the proposed methods, validating their ability to achieve improved data efficiency and overall model performance.\n\nSTRENGTHS:\n1. The paper introduces a framework that combines general continual learning (CL) techniques with random layerwise token dropping for data sampling and data routing. This framework aims to address the challenges of CL by incorporating the random dropping of tokens at each layer, enabling efficient data processing and routing during the learning process.\n\n2. The effectiveness of the proposed method is demonstrated through experiments conducted on various foundation models. The results highlight the remarkable data efficiency achieved by the framework, showcasing its ability to handle continual learning tasks effectively while maintaining high performance with limited data.\n\nWEAKNESSES:\n1. While data efficiency is recognized as crucial for various tasks, the paper could provide a more comprehensive study and presentation of how the proposed method enhances models across different data sizes. A more thorough investigation and analysis of the impact of the proposed method on models of varying data sizes would contribute to a deeper understanding of its effectiveness.\n\n2. It is worth noting that the paper's verification of the proposed method is limited to four models. Expanding the experimental evaluation to include a broader range of models would provide a more robust assessment of the method's performance and its applicability across different architectures. This would enhance the credibility and generalizability of the findings.\n\nQUESTIONS:\n1. A direct comparison between the proposed method and Scaling Law [1] is not provided in the paper. It would be valuable to evaluate and discuss how the proposed method differs from or complements the principles and findings of Scaling Law [1] in terms of data efficiency and model improvement.\n\n2. Random layerwise token dropping, as described in the paper, refers to the process of randomly dropping tokens at each layer during the learning process. Drop path [2], on the other hand, is a technique commonly used in neural architecture search where random paths of layers are dropped during training. While both methods involve dropping components during the learning process, how do they differ in terms of the specific units being dropped and affect the performance?\n\n3. The paper primarily focuses on verifying the proposed method's effectiveness on foundation models. It is not explicitly stated whether the method can be applied to tasks with limited data, such as ImageNet. Further investigation and experimentation would be necessary to determine the performance and applicability of the proposed method on tasks with smaller datasets, beyond the foundation models considered in the paper.\n\n[1] Scaling Laws for Neural Language Models\n[2] Deep Networks with Stochastic Depth\n\nLIMITATIONS:\nPlease refer to the weakness and question part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes XYZ Data Efficiency, a framework that makes better use of data, increases training efficiency, and improves model quality. The proposed framework features efficient data sampling, efficient data routing, and an easy-to-use practical framework that is integrated into an existing library.

STRENGTHS:
- The targetted application on improving the data and training efficiency is an important problem, especially in the era of large models. This paper makes a practical step in this direction. 

- I like the writing of the introduction and related works, which provides a structured review and highlights the goal of this paper and its difference from other papers. 

- The description of the proposed method is pretty detailed, which can help the reader to have a detailed understanding of how this framework is implemented and part of the time/computation overhead to run this framework. 

- Overall, I would say this paper makes good engineering efforts to make the

WEAKNESSES:
- My main concern about this paper is the evaluation observations. Specifically, it seems that under lower-budget training settings (e.g., training with less data and training time), the improvement over the baseline actually shrinks. Will this make this method can only be suitable for teams with a large amount of data and computation resources?

- In the paper, the authors mentioned that previous methods for improving data/training efficiency fail to achieve satisfactory performance under large-scale settings. Is there some numerical evidence for this claim other than the one shown in Table 1?

QUESTIONS:
Please refer to weakness.

LIMITATIONS:
The authors may want to address the limitations of this paper in their final version.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces XYZ, a data sampling and routing framework designed to enhance the efficiency of training large transformer models. XYZ incorporates a user-defined curriculum learning metric for data sampling and leverages token dropping to reduce computational overhead. The authors propose random layer-wise token dropping (random LTD) to efficiently apply token dropping per layer, capturing attention dependencies between tokens in intermediate layers with high probability. The framework's effectiveness is validated through experiments on pretraining GPT-3, GPT-3 MoE, and BERT, as well as finetuning GPT-2 and ViT, achieving up to a 12.5x reduction in data/time/cost.\n\nSTRENGTHS:\n1. This paper introduces the random layer-wise token dropping technique, which demonstrates a novel approach to enhance the efficiency of large transformer model training. \n2. The evaluation is across various models of different sizes, including GPT-3, GPT-3 MoE, BERT, GPT-2, and ViT. \n3. The paper provides a comprehensive and detailed account of the training setting used in the experiments. Additionally, the authors present a thorough analysis of the results and observations obtained from the experiments. \n4. The paper shows substantial efficiency gains using the XYZ framework.\n\nWEAKNESSES:\n1. One notable weakness of the proposed framework is its relatively limited performance compared to the baseline when operating at a smaller data scale, as indicated in Figure 6. Further investigation and clarity on the factors contributing to this limitation would be valuable for understanding the framework's practical applicability across various data scales.\n2. Despite claims of open-sourcing the XYZ framework, anonymized code or a link to access the implementation is not provided. \n3. The paper does not explicitly address the limitations of their proposed framework or discuss any potential negative societal impact.\n\nQUESTIONS:\n1. How does the proposed XYZ framework compare to the baseline when pre-training GPT-3 1.3B on a data set of 75B tokens?\n2. Given the observed slower convergence of XYZ compared to the baseline, what are the underlying factors contributing to this behavior? Specifically, does the convergence difference primarily result from the data sampling methodology (curriculum learning - CL) or the token dropping technique (random layer-wise token dropping - rLTD)?\n\nLIMITATIONS:\nThe authors have not explicitly discussed the limitations and potential negative societal impact of their work in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}