Efficient Large-scale Transformer Training via Random and Layerwise Token Dropping
Authors: Zhewei Yao
Xiaoxia Wu
Conglong Li
Connor Holmes
Minjia Zhang
Cheng Li
Yuxiong He
Venue: Submitted to ICLR 2023 (2023)
Keywords: Efficient Training
Large-scale Transformers
Token Dropping
GPT
BERT
ViT

Abstract: Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. 
However, those large models also introduce prohibitive training costs. 
To mitigate this issue, we propose a novel random and layerwise token dropping method (\OURS), which skips the computation of a subset of the input tokens at all middle layers.
Particularly, \OURS achieves considerable speedups and comparable accuracy as the standard training baseline. 
Compared to other token dropping methods, \OURS does not require (1) any importance score-based metrics, (2) any   special token treatment (e.g., \texttt{[CLS]}), and (3) many layers in full sequence length training except the first and the last layers. 
Besides, a new \layertoken learning rate schedule is proposed for pretraining problems that resolve the heavy tuning requirement for our proposed training mechanism. 
Finally, we demonstrate that \OURS can be applied to broader applications, including \gpt and \bert pretraining as well as ViT and \gpt finetuning tasks. 
Our results show that \OURS can save about 33.3\% theoretical compute cost and 25.6\% wall-clock training time while achieving similar zero-shot evaluations on \gptb as compared to baseline.

Under review as a conference paper at ICLR 2023
EFFICIENT LARGE-SCALE TRANSFORMER TRAINING VIA RANDOM AND LAYERWISE TOKEN DROPPING
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. However, those large models also introduce prohibitive training costs. To mitigate this issue, we propose a novel random and layerwise token dropping method (random-LLTD), which skips the computation of subset of the input tokens at all middle layers. Particularly, random-LTD achieves considerable speedups and comparable accuracy as the standard training baseline. Compared to other token dropping methods, random- LTD does not require (1) any importance score- -ased metrics, (2) any special token treatment (e.g.. [CLS] ]), and (3) many layers in full sequence length training except the first and the last layers. Besides, a new LayerToken learning rate schedule is proposed for pretraining problems that resolve the heavy tuning requirement for our proposed training mechanism. Finally, we demonstrate that random-LTD can be applied broader applications, including GPT and BERT pretraining as well as ViT and GPT finetuning tasks. Our results show that random-LTD can save about 33.3% theoretical compute cost and 25.6% wall-clock training time while achieving similar zero-shot evaluations on GPT--31 as compared to baseline.
INTRODUCTION
Large-scale transformers have been demonstrated to have supreme performance on natural language processing (Temney al., 2019; Radford et al., 2019; Raffel al., 2019), computer vision (Dosovitskiy et al., 2020), and other applications (Gong et al., 2021; Guo et al., 1). However, both the pretraining procedure and some downstream finetuning tasks (e-g., long document summary) are time-consuming and sesource-hungry Thus, there is need to speed up the training and reduce the compute cost for large-scale transformer pretraining and finetuning.
Recently, Hou et al. (2022) adopt the token pruning/rroping/yypasingg technique (Kim et al., 2021; Goyal et al., 2020; Kim & Cho, 2020) from BERT inference to BERT pretraining by skipping the compute of part of the input tokens at some middle layers. The results of (Hou et al., 2022) (referred to as TokenBypass) show that can theoretically reduce the pretraining cost by 25% for both BERT base and BERT large without losing accuracy finetuning tasks. Although achieving great speedup, TokenBypass (1) needs an import-score metric to determine the dropped tokens and special token treatment keep important tokens (e.g., [CLS] )), both of which require manual designs; (2) has to keep the first half layers and the last layer (in total, half of the depth) in full sequence length training, which limits its layer-bypassing ability. (3) solely focuses on BERT Masked-LM pretraining tasks and has not been applied to other tasks, e.g., causal-LM. In this work, we address those challenges and introduce our random and layerwise drooping method (random-LTD) In summary, our contributions are as follows:
All tokens are treated equally without any special token treatment or import-score measurement, i.e., no manual design, and are dropped in purely random manner. Meanwhile, instead of fully bypassing the dropped token for all middle layers (Hou et al., 2022), each layer in random-LTD drops tokens independently from the other layers. This helps the multi-head attention in the middle layers capture the dependency relation across different tokens suggested in (Vig & Belinkov, 2019). random-LTD applies token dropping all middle layers except the very first and last layers, which further reduces manual design and increases training efficiency. We also propose a new monotonic sequence length growth method training evolves to (1) reduce the gradient noise introduced

---

Under review as a conference paper at ICLR 2023
by random-LTD for better convergence and (2) close the training and inference (autoregressive generation) gap, since random-LTD breaks the autoregressive manner in middle layers during training, for GPT models.
To reduce the tuning effort for the newly proposed training procedure, we introduce a new LayerTo- ken learning rate schedule, which scales the learning rate based on the sum of consumed tokens of each layer for pretraining tasks.' We show its superb performance for random-LTD on GPT/BERT pretraining compared to the standard lieration-based learning rate schedule.
We extensively test random-LTD on both pretraining tasks, including GPT and BERT pretraining. and finetuning tasks, including causal-LM finetuning for GPT and image classification for ViT. For all tasks, random-LTD achieves similar accuracy the original baseline method with up to 33.3% theoretical cost saving and up to 25.6% wall-clock time saving.
Finally, we show that random-LTTD has a potential regularization effect, which can be used for both pretraining and finetuning problems.
2 BACKGROUND
Transformer (Vaswani et al., 2017) architecture is a stack of transformer layers, each of which has two main ingredients, i.e., the multi-head attention (MHA) and the feed-forward connection network (FFC). Suppose the transformer has / layers denoted as Li...., L. Let X; E Raxd be the output tensor of i-th transformer layer, and zy the input (after embedding) of the transformer. Here is the sequence length and d is the hidden dimension.
Token dropping (or token bypussing/pruning) (Kim et al., 2021; Goyal al., 2020; Kim & Cho, 2020; Press et al., 2021; Wang et al., 2021) was originally proposed for BERT inference to reduce the computational overhead. In this case, if token (X),) decided to be dropped at layer (Iz), the compute cost of this token through all remaining layers (I,g where k > j) is eliminated. As such, the sequence length s; of the i-th layer's input X,-1 will be non-increasing array, i.e., so 2 81 ... 8t- However, such configuration has been shown instability for adaptive token-dropping inference (Kim & Cho, 2020). Therefore, Kim & Cho (2020) utilize the sandwich rule and distillation from (Yu & Huang, 2019) to stabilize training and boost accuracy. But these two methods also significantly increase the training cost. Thus, such techniques cannot be applied to speed up the pretraining procedure. Recently, Hou et al. (2022) extended token dropping from inference to BERT pretraining (referred to as TokenBypass). Hou et al. (2022) use several importance scores/metrics to determine the dropped tokens, c.g., cumulative loss and frequency of each token. To overcome the training instability issue, the authors proposed two main mechanisms: where the first (layer to i) and the last few layers (layer Lı- to L,) of the BERT capture all tokens (i.e., no token dropping) and the middle layers bypass s' S s tokens from L, to Lı-j. Particularly, the authors (only) test on the encoder transformer (12-layer BERT hase and 24-layer BERTaage)) and let i = 1/2 1, = 1, s' = 8/2. (2) special token treatment, where special tokens (e.g.. [MASK], [CLS], , [SEP]) ) are never dropped.
(1) the sandwich token dropping rule,
Compared to TokenBypass from (Hou et al., 2022), our random-LTD (1) does not require importance score metric, special token treatment, or the sandwich token dropping rule, which dramatically reduces the manual design effort; (2) has been broadly tested on pretraining tasks, including GPT and BERT, as well as finetuning tasks, including VIT classification and GPT causal-LM. Meanwhile, we found out that directly applying TokenBypass to causal-LM leads to severe accuracy degradation. Please see the detailed description of random-LTD Section 3 and our extensive evaluation in Section and 5. We also include thorough discussion of other efficient training methods in Appendix A.
3 METHODOLOGY
3.1 RANDOM AND LAYERWISE TOKEN DROPPING METHOD
Layerwise Token Dropping Mechanism. As pointed out in Section 2, existing inference and training token dropping methods either permanently drop tokens from the compute graph at intermediate layers, or at least make some tokens fully skip a consecutive series of middle layers. However, several works (Vig & Belinkov, 2019; Michel et al., 2019; Voita al., 2019) have shown that MHA focuses
Note that the numhers of consumed tokens for different layers are different.
2

---

Under review as a conference paper at ICLR 2023
(a) Baseline
(b) Our Random-LTD
Output tokens of layer!
Outout tokens loyer
if meth == "baseline";
1
2 hs = Layer (hs)
FFN MHA
MHA
if meth "random-177D";
..I....)
k k_hs, d_hs gather (hs)
output tokens of layer
outpul tokens of layer
5 k_hs - layer (k_hs, d_hs) hs = combine khhs, d hs)
FFN & MHA
FFN & MHA
compute 5aving
6
Randomly assign takens...
Input tokens of layer/
Input tokens of layer
random-LTD only requires a few lines of code. Here hs, kas, and das means the full input, kept input, and dropped input. "gather", "Layerr" "combine" means the func- tions for random selection, transformer layer, and token combination.
Figure 2:
FFN & MHA
FFN MHA
Input tokens layer
Input tokens layer
Figure 1: Transformer layers for baseline and random-LTT training. The dash-line box is repeated by / - 2 times (see more in Fig. B.I).
on different tokens at different layer depths and the attention map aligns with the dependency relation most strongly in the middle of transformer architectures. Therefore, TokenBypass used in Hou et al. (2022), i.e., fully skipping middle layers, may hinder the learnabilttygeeeerliian during pretraining//nfereme
of the architecture We conjecture that this might be why multiple first/last layers need to be kept and the special token treatment is needed in (Hou al., 2022). To further verify if this fully skipping middle layer mechanism (Hou et al., 2022) causes any learnability issue, we apply TokenBypass GPT finetuning tasks and observe much lower performance as compared to baseline. See more details in Section 5.1.
In order to overcome this problem, Instead of fully bypassing dropped tokens over all middle layers, each transformer layer independently drops/retains its own set of tokens. In more detail, recall that the input of (i + 1) 1) layer (L++1) is X, E Rsxd, Denote the dropped token index as J, = f2. Ja. J and the kept token index as K; = K (k,,,..,k,. such that a, + b, = s. We have Jj K; = (1, 2..,, s} and Jr n K; 0 for each layer. Meanwhile, for any two different layers Li, and Lig. Ji, and J., are independent, though the dropped ratios are the same. With this layerwise mechanism, each token rarely bypasses all middle layers. Thus, its dependency on other tokens can be captured by MHA.
propose layerwise token dropping (LTD) mechanism.
Random Token Dropping. Various important score- based metrics are used to determine the token dropping criterion. Most of them can be categorized in two ways: attention score related metrics or lossfrrequency-basee metrics. However, both of them introduce challenges that make LTD less practical. Particularly, for attention score-based metrics, the compute cost for LTD is too high metrics, generally
since the metric has to be calculated for every layer; for fosss/frequencc-bass accumulated loss frequency is used and this accumulated metric would not be changed within the same iteration (a.k.a. one forward pass of the network). Therefore, the unchanged loss/frequency metric leads the dropped token to be the same for different layers, making the token dependency not be captured by the MHA of middle layers (Vig & Belinkov, 2019).
To satisfy the independent requirement of LTD, we propose to use purely random token dropping assignment. For each transformer layer, we randomly (uniformly) select small batch of tokens to proceed with the compute and drop the rest. In more details, assume M, = mm, my (2). ... ms)))) is a random shuffle of S =11, 2, .... s). Then the dropped token set is Jj = [m, (1), m, (2). m. (a:)] for the input of L;+1.
Random and Layerwise Token Dropping. Combining layerwise token dropping with random token dropping, we have our final random and layerwise token dropping method (random-LTD), which can efficiently apply token dropping for each individual layer and can capture the attention dependency of each token with other others middle layers with high probability.
The illustration of the comparison between standard baseline training and random-LTDD is shown in Fig. (an additional comparison with (Hou et al., 2022) in Fig. B. The pseudo-code is given in Fig. 2. For each layer, as compared to the baseline, random-LTD randomly selects (function "gather" in Fig. 2) a subset of the tokens and feeds (function "Layer" in Fig. 2) them into the transformer layer. Afterward, we combine (function "combine" in Fig. 2) the output of transformer layer with the dropped tokens to recover the full sequence length. Thus, the next layer still receives the full sequence and can repeat this process.
3

---

Since the dropped tokens of each layer are independent, there is no need for random-LTD treat special tokens (c.g., [MASK], [CLS], [SEP], [PADDING]) differently from other normal tokens, which can further reduce the cost of computing the dropping criterion. Mcanwhile, we show Under review as a conference paper at ICLR 2023
that special token treatment does not bring extra benefits for random-L.TD) on BERT pretraining in Section 5.2.
3.2 DROPPING SCHEDULE OF RANDOM-LTD
Layers without Token Dropping. While TokenBypass (Hou et al., 2022) needs to keep half of the layers in full sequence length training, random-LTD has no such limitation. Thanks to the attention-captur capetten feature of random-LTD, layers except the first and last transformer layers.
apply random-LTD to most of the transformer
Keeping the first and last layers in full sequence length training usually leads to better performance since (1) the first layer directly connects to the embedding, and it can help refine the raw feature; (2) the last layer directly connects the final prediction; feature realignment for all tokens can improve the model quality. We also provide detailed study to show the importance of keeping the first and last layers without token dropping in Section 5.3.
Monotonic Sequence Length Growth. In order to reduce the gradient variance introduced by random-LTTD for better training, we monotonically increase the kept sequence length throughout training (referred to as MSLG) with a linear schedule. Particularly, the dropped token set Jj for the i-th layer gradually shrinks and the kept token set K; gradually grows the training proceeds. Denote the size of J; (K₂) at step t is aiæt (br,)), its final size is O (s), and the total training iterations T. Assume we want to gradually reduce the size of J, to zero at iteration T and the decreasing strength is ddee. Then the decreasing step size is Taee / (ao.t/ 8aee), ). i.e., for every Tace iterations, the size of Jj (K,) reduces (increases) by 8dee. Please see Fig. for an illustration of K; on GPT pretraining. We also show that MSLG outperforms the constant drop schedule with similar compute savings in Section 5.4.
3.3 NEW LEARNING RATE SCHEDULE FOR PRETRAINING
When performing pretraining on language models, we oftentimes use a decaying learning rate schedule based on iteration with a warmup period. Particularly, at the first few thousand or hundred iterations, warming up the learning rate critical for distributed pretraining tasks due its instability (Goyal et al., 2017; Li et al., 2021). However, an iteration- based schedule is not optimal for random-LTD.
First, random-LTD reduces the effective batch size of middle layers at the initial warmup phase. The effective training tokens for dropped token layers become much smaller than the baseline training. Second, for most of our training cases, MSLG does not reach the full length until > 2/3 of training iterations for large compute saving. At such time, the iteration-based learning rate is considerable small. And this small learning rate cannot provide efficient training dynamics for random-LTD. Therefore, stabilize the initial training phase and to have large enough learning rate in the later training phase, we need to increase the warmup iterations and slow down the learning rate decay. Here, we propose called layer-token learning rate (LayerToken LR). Please see Appendix C for the formal and detailed description of LayerToken LR.
learning rate schedule based on the layerwise tokens consumption,
We emphasize that one can always tune the learning rate schedule by increasing the maximum learning rate or the warmup iterations. However, would require lot of engineering effort. Therefore, we propose this LayerToken LR schedule, which is more suitable for our random-LTT than the standard one. We also include a detailed comparison between the standard learning rate schedule and LayerToken LR in Section 5.5.
4
MAIN RESULTS
In this section, first provide the results of random-LTD for pretraining GPT and BERT models. We then extend random- LTD on the computer vision domain to demonstrate its broader applications. Similar to Section 3.3 and Appendix C, we use the LayerToken compute the cost measure the total
4

---

Under review as a conference paper at ICLR 2023
training budget.² We also provide the real training time saving for GPT and BERT pretraining. Kindly note that the real-time saving depends on various factors, e.g., the implementation and hardware.
4.1 GPT PRETRAINING
We train GPy-3-style models with
GPT-350M
GPT-1.38
350 million parameters (GPT-3350M)
4.5
4.5
Banel
Baseline
and 1.3 billion parameters (GPT-31
Ranstim 20um)
4.0
Bandomi (Ount
on PILE dataset (Gao et al., 2020)
Jallsgton Lass
260
and the total number of training to-
3009
140
kens is 300 billion. For random-LTD,
the initial dropped token length for
all middle layers is 1920 (i.e., 128 to-
2.5
2.5
kens are kept for compute), and it de- creases by 16 for every 1.75B train- ing tokens. After 210B training to- kens, random-LTD degrades to stan- dard training procedure with full se- quence length. Theoretically, this can save 1/3 of the LayerToken training budget. See Appendix D.I for more details.
2.0
2.0
Layer-Tokens (x10
Layer-Tokens (x1042)
Figure 3: The comparison of validation curves between base- line and random-LTD on GPT pretraining. Here the x-axis is based on LayerToken consumption. The sequence length is illustrated the inserted figure.
The evaluation loss curves for baseline and random-L.TD) are shown in Fig. 3. As can be seen, for both 350-3 and GPT-3 random-L.TD has similar evaluation losses as the baseline with 1/3 less LayerToken consumption. We also provide the zero-shot evaluation results in Tab. 1. For both GPT-3350M and GPT-3 random-LTD achieves comparable results the baseline, Besides, random-LTD can save 14.3% wall-clock training time on GPT-33300 and 25.6% wall-clock training time on GPT-31.38.
We reiterate that the LayerToken consumption saving ratio cannot directly transfer to GPU wall-clock training time saving ratio due to the implemennationnhardwaar Meanwhile, note that the saving number we reported here is not the maximum potential saving (with fixed implementation/hardwee ctc) since we can reduce the training GPU numbers for random- LTD at the initial training phase, which has shorter effective training sequence length. Also, although random-LTD has the same theoretical compute saving for both GPT-3330M and GPT-313B- the real wall-clock time saving varies lot because GPT-31.3H has larger hidden dim size, which means the model spends more time real computing than other operators, e.g., data movement and gradient communication.
Table 1: Zero-shot evaluation results (last col.) on GPT-3350M and GPT-3 The GPU cost provided here is the used number of A100-40G times the training days. See Tab. 1.1 for 19 tasks.
Table 2: Finetuning results for BERT large The training cost provided here is the used number of A100-40G times the training days. See Tab. E. for full results.
Model
Method
TayerToken Saving GPU Cost (saving) Ave.
Method
LayerToken Saving GPU Cost (Saving) Ave.
GPT-3
haseline
Nore
64x2..9 (0.0%) 38.6
baseline
None
64x5.89 (0.0%)
85.42
random- LTD
33.3%
64x2:22 (14.3% 38.9
rundom-LTD-i
26.2%
64x5.42 (7.96%) 86.95
Baseline
None
64x5.42 (0.0%)
42.7
random-L
31.1%
64x5.21 (11.5%) 86.42
ITT
33.3%
64xx.03325569%
64x4.03
(25.6%)
42.5
4.2 BERT PRETRAINING
We pretrain BERT large on PILE dataset for 2M iterations with batch size 1024 and sequence length
512 following (Shoeybi et al., 2019). We apply random-LTD with two variants, random-LTD-I and random-LTDD- Particularly, for random-LTTD- (random-I.TDD-2,, the initial kept token length is 200 (128), and it increases by 16 for every 48B (38B) training tokens. As such, we save 26.2% (31.1 %) LayerToken consumption for random-LTD-1 (random-LTD-22) We evaluate the trained model on
SSiiilrr to (Hou al., 2022), do not include (1) the final prediction layer and (2) the attention compute difference between the different lengths of sequence for the compute cost comparison.
5

---

Under review as a conference paper at ICLR 2023
four downstream tasks (Shoeybi et al., 2019), i.e., MNLI, QQP, RACE-m, RACE-h. Note that we apply standard finetuning without token dropping to have a fair comparison for both pretrained models from the baseline and random-LTD. Please see Appendix D.2 for more details.
Tab. 2 summarizes the results along with the full results in Tab. E.I. Although Mundom-LTD is slightly worse than baseline while saving 16.2-31.12 % of the theoretical computation overhead in pretraining. Overall, random- LTD-1 achieves 1 1.54 points higher average accuracy over baseline and random-LTD-2 achieves 1 point higher average accuracy over baseline.
certain task (QQP, see Tab. E, it gives much higher accuracy other tasks
Meanwhile, random-L.TD-1 (random-ILTD--) saves about 7.95% (11.5%) wall-clock time as compared to baseline. Note that similar to GPT pretraining, the saving depends on the implementation/hardwww and random-LTD has potentially larger savings if clastic training is performed. Also, although GSTM- and BERT large have similar model sizes as well as similar theoretical compute saving, the final wall-clock training time saving varies by about 3%. This is caused by BERT large having shorter final sequence length (i.e, 512) than GPT (i.e, 2048), and the real compute time for a sentence with sequence length 128 not 1/4 (or 1/16) a sentence with 512 (2048) tokens. This leads the overall compute time-saving for GPT-3350M to be larger than that for BERT large
4.3 VIT FINETUNING
We perform the vision transformer (ViT) on both ImageNet (with a 12-layer pretrained ViT) and CIFAR (with 24-layer pretrained ViT). For random-LTD, the initial sequence length is 66 and linearly reaches the 197 full sequence length at 80% of the total training iterations such that 22.3% layer-token saving is achieved. See training details in Appendix D.3. We summarize the result with standard deviation in Tab. 3 along with the full details in Tab. H.1. As can be seen, random- can achieve comparable results as the bascline on all three datasets. This demonstrates the broader applications of random-LTTD.
Table 3: Finetuning result of VIT on ImageNet. See Tab. H.1 for the results on CIFAR1O/100
Table 4: Ablation study of special token treatment for BERT pretraining with 22.2% LayerToken sav- ing. See Tab. H.2 for all results.
ImageNet datasets on 12-layer VIT
Method
LayerToken Saving
Top-1
Top-5
Keep Special Tokens
Pretraining PPL val/test
Downstream Ave.
baseline
N/A 22.3%
84.6520.04 84.70+0.04
97.41=0.02
97.48-0.02
yes
6.024 6.049 6.018 2018/66 6.040
no
88.52
5 DISCUSSION
In this section, we present several import ablation studies and the potential regularization effect of random-LTD. Besides the three tasks used in previous sections, we also include GPT finetuning on causal-LM problems using the GPT-2350M from Huggingface (Wolf et al., 2019). Please see Ap- pendix D.4 for the training details. Also, we reduce the iterations of BERT pretraining from 2M to 200k due to resource limitations. Please see Appendix D.2 for more details.
5.1 LAYERWISE TOKEN DROPPING VS. BOKENBYPASS
Although TokenBypass (Hou et al., 2022) demonstrates its great ability on BERT pretraining, its skipping policy may still hurt the performance of other tasks, e.g., causal-LM. The reason is mentioned Section 3, i.e., MHA of middle layers focuses on different tokens at different depths. Fully skipping those layers may lead the causal-LM task to lose attention capability. However, random-LTD does not have this issue as it randomly selects kept tokens for each layer.
To verify this, we provide an ablation study the comparison between random-LTD and TokenBy- pass with GPT-2350M finetuning on PTB (Marcus et al., 1993). We make two sets of experiments:
Set 1. Following (Hou al., 2022), we bypass half of the tokens based on their empirically moving average loss from L12 to L2s. Similarly, we apply constant random token drop to the layers from the middle to the second last (L12 to L23).
Set 2. We apply TokenBypass or constant random token dropping for half of the tokens starting from the second layer (T.g) until the second last layer (T2a).

---

Under review as a conference paper at ICLR 2023
The validation curves of two cases shown in Fig. 4. As can be seen, for both cases, random-LTD performs much better than TokenBypass. Particularly, for the Set 2 comparison, the perplexity of Mundom-LTD is about 10 points lower than TokenBypass, demonstrating that random-LTD can be applied to more layers than TopesSypass.
Rand
Bypan 1L12 22318a nd LTD TopenBypase (23)
SP12 on Wicllect-105
G2112 on Wild 1eab-2
Natrod mL
Thore-Srap Rate:
Tekentypasa 20.00
10.0%
87.5%
Netted best FRL:
13.
75.0%
tokertsposs 20.00
33.3
24
LTG 2057
24
20
2
4 5
8
10
12
14
122
23
5.2 9 25
Triiing Epoch
Traning Fpoch
The byye Random-LT
The byee
Figure 4: Comparison between random token dropping and TokenBypass.
Figure 5: Sensitivity of random token dropping of each layer.
5.2 WITH/WITHOUT SPECIAL TOKEN TREATMENT
Different from the GPT pretraining task, which has consecutive sentences/ppragrrahhs, the BERT pretraining data consists of two sentences that could be unrelated. The special tokens [CLS] and [SEP] ] play critical roles for the model in determining the beginning/end of each sentence so that the model can predict the relationship between the two sentences. Thus, there could be potential gain in keeping the tokens for all layers, which has a more detailed discussion in (Hou et al., 2022).
Here we present an ablation study on whether keeping those special tokens helps random-LTD or not. We perform a straight comparison for random-LTDD (1) one with purely random selection and (2) the other with an additional criterion, i.e., keeping the special tokens for all layers. See training details in Appendix D.2
The results of MNLI/QQP are shown in Tab. along with the full details Tab. H.5. As can be seen, for both pretraining loss and downstream finetuning, special token treatment does not provide any benefit for random-LTe LTD. Note that candom-LTD without special token treatment is also more compute-friendly for token dropping.
5.3
WHY WE NEED TO KEEP THE FIRST AND LAST LAYER?
To understand why keeping the first and last layers in full sequence length training, we present single layer sensitivity analysis shown in Fig. 5 for GPT-23s0M finetuning on Wikitext-2 and Wikitext- 103. Particularly, we apply constant token dropping for one layer and keep all the other layers in the standard training mode. After the training, we measure the PPL and use it as the sensitivity metric, i.e., higher PPL indicates high sensitivity and vice versa. The U-shape of both curves implies that the first and last ones most sensitive to token dropping.
Table 5: Comparison of applying random-LTD to different layers on GPT-2350M finetuning and ViT finetuning. See Tab. H.3 with standard deviation.
Table 6: Compare between MSLG and con- stant token dropping schedules. See Tab. H.4 for the full result with standard deviation and the result ViT finetuning.
Apply
rundom-LTD except for Layer
Metric
dataset
None
First
Last
First&Last
dataset
PTB (Metric: perplexity)
PTB
16.00
16.01
16.09
15.92
Token-drop schedules constant constant MSLG
Perplexity
WikiText-2
17.06
17.01
17.01
16.94
LayerToken saving
23.0%
32.1% 33.7%
WikiText-
13.27
13.03
13.23
12.99
Accuracy
Topp
84.47
84.51
84.65
84.70
Performance
18.27
20.76
15.92
To further understand if random-LTD can be applied to all layers when using MSLG, we include other three scenarios, i.e., applying random-LTD to (1) all but not last layer, (2) all but first layer, and (3) all layers. We perform finetuning tasks on both causal-LM and image classification. See the full training details in Appendix D.3 and Appendix D.4. From Tab. 5 and H.3, we clearly see that keeping the first and the last layers intact leads to a substantial improvement (beyond standard deviation) over the rest three scenarios.
7

---

Under review as a conference paper at ICLR 2023
5.4 WHY WE NEED SEQUENCE LENGTH GROWTH?
We now give an ablation study on why MSLG schedule is necessary. Again, We perform finetuning tasks both causal-LM and image classification.
We specially set a constant token dropping rate that matches the token saving of MSLG schedules with all other hyperparameters fixed. We present the results in Tab. 6 and H.4. can be clearly seen that given the almost same amount of LayerToken saving (33% - 35%), the constant dropping schedule has worse performance than MSLG. MSLG schedule can actually be even better or comparable to those constant ones whose saving is 10% smaller.
5.5 LAYERTOKEN LR SCHEDULE EFFECT
To study the effectiveness of LayerToken LR, we compare three training scenarios for GPT--330MM with 300B training tokens (see Appendix D.1 for training details): (1) the baseline training with the standard learning rate, (2) random-LTD with the standard learning rate, and (3) random-LTD with LayerToken LR. The validation curves and their corresponding learning rates with respect to iterations are plotted in Fig. 6. As can be seen, the green curve (random- LTD with LayerToken LR) can achieve comparable validation loss as the baseline, which is better than random-LTD with the standard learning rate. This confirms that the small learning rate introduced by the standard learning rate schedule slows the learning of random-LTV LTD the later training phase. A similar observation made for the BERT pretraining, which will be deferred to Appendix F due to the space limit.
Validation Loss M.t. teration
Learzing Roto st. Reration
doscciie vis havfommi LID (Bryooret de Soodin v.s. Rardoms LTD (Dropstee)
5.0
4.5
10900toon 1000
4.0
Laatmiing Bake
2.5
Uniidettonn 5
2.5
Stnddrt Mastediia
Hathol Deat
Oas zens
Hethed (ool
2.0
0
100 200 300
409 500
600
0
100 200 300 400 500
Layer 16 120
10
Beration (x1000)
Teeation 1o1 B00)
Layor tokons
Figure 6: Comparison between a standard learning rate and our proposed learning rate based on layerwise tokens
Figure 7: Study the regularization of random- LTD on GPT-2350M finetuning the PTB task without dropout (left) and with dropout (right).
5.6 COMPARISON ON GPT-31.3B WITH VARIOUS TRAINING BUDGETS
In this section, perform various training budgets to train GPT-3 1.38 and compare the performance of baseline and random-LTD to verify if random-LTD can consistently save the training cost.
To fully understand the effectiveness of random-LTD, we train GPT-3 3a using baseline with 120B (i.e., 2880 billion LayerToken consumption) to 360B tokens (i.e., 8640 billion LayerToken consumption). For random-LTD, we follow Section 4 setting to save 1/3 of the total training budget and apply to three training budgets, which are equivalent to 2880, 4800B, and 5760B LayerTokens.
The results are summarized in Tab. 7. The first noticeable result here is that the total amount of training LayerTokens directly affects the model's 's final quality. Longer training usually leads to better accuracy. Meanwhile, random-LTD can use 2/3 of the training budget to achieve similar evaluation results the baseline. For example, random-LTD with training budgets 2880, 4800, and 5760 billion LayerTokent can lead to similar accuracy as the baseline with training budgets 4320, 7200, and 8640 billion LayerTokens.
5.7 THE INTERPLAY BETWEEN RANDOM-LTD AND DROPOUT
Token dropping be viewed as special (coarse-grained, grained) case of dropout. As such, it may have the potential to work as dropout for certain tasks or further help dropout for certain tasks. To investigate this, apply random-L.TD with/without standard dropout both pretraining and finetuning tasks.
8

---

Under review as a conference paper at ICLR 2023
BERT targe Pretraining, Please see Appendix D.2 for training details. For random-LTD-3 (random- LTD-4), the initial kept token length for all middle layers is 256, and it increases by 16 for every 3.8 (6) billion training tokens such that eventually save 14.1% (22.3%) LayerToken consumption.
The results are summarized in Tab. with details Tab. H.5 and we include the pretraining perplexity to better understand the dropout effect. Clearly, turning off dropout results in lower evaluation/test perplexity (shown in Tab. 8). Meanwhile, the performance of those no-dropout models (baseline", random-LTD-3*, random-LTD-4*) on MNLI and QQP show obvious improvement over their dropout counterparts (baseline, random-LTD-3, random-LTD-4) However, for RACE finetuning, there is no learning for the no-dropout baseline model, which somewhat is surprising but shows the importance of dropout for pretraining. In contrast, when turning off the dropout for random-LTV we see a compelling better accuracy on RACE, which exceeds the standard baseline pretraining by > % on RACE-m. Thus, random-L.TD) brings not only the efficiency but also the potential regularization effect to BERT pertaining.
Table 7:
Zero-shot results on GPT- 313B with various budgets (by Layer-
Table 8: Study the regularization effect of random-LTD.
We report the average of dev and test for RACE-m and RACE- h. means no dropout. Please see Tab. H.5 for the full result
Token). See Tab. 1.2 for all 19 tasks.
Method
Budget (B)
Ave.
with standard deviation.
2880
41.0
(Laycr-token saving)
Pretraining PPL val/test
Downstream Ave.
4320
41.7
baseline (Nonc)
5.78/5.80
80.14
Baseline
5760
42.5
baselinc* (Nonc)
5,45/5.46
56.76
7200
42.7
Bandom-LTD-3
6.37/6.40
78.72
8640
43.1
random-LTD-3" (14.1%)
5.79/5.80
81.01
2880
42.1
random-L.TD-44222.. (22.3%)
6.52/6.58
78. 16
random-LTD
4800
42.5
random-LTDD-4 (22.3%)
6.02/6.04
80.49
5760
43.1
GPT-23300 Finetuning. Let us further study the potential regularization effect of random-LTD on GPT-2250M finetuning tasks. We train two sets of models, one without dropout and one with dropout (the default rate is 0.1). We also apply random-LTD with three initial sequence lengths- 128, 256, and 111-ssch that they reach the full sequence 1024 at the same epoch (12). We present their validation curves in Fig.
We see that the bascline training without dropout quickly overfits, as shown in the left of Fig. 7 (black curve). And its best validation PPL is worse than the baseline with dropout (block curve in right figure). However, for all three cases of random-LTD, they achieve similar PPL as the standard baseline with dropout. Even with the default dropout in the right of Fig. 7, its validation still faces non-negligible overfitting issue after the middle of training. In contrast, random-LTD with MSLG introduces potential (extra) regularization effect. As can be seen, the validation curves of random-LTI are flattening and they maintain in a considerably small value regime towards the end of training.
Summary. We do not claim that random-LTD can replace dropout. In fact, the two can work well (note that both are turned on for the results in GPT pretraining in Section 4) with each other as dropout and random-LTD focus on different granularity. We see that Fandom-LTD with dropout achieves lower perplexity for the small finetuning task (shown in the right of Fig. 7). Thus, the additional potential regularization effects introduced by random-LTu could be complementary to dropout and play helpful role in those small-dataset tasks with critical overfitting issues. Also, we see that Landom-LTD without dropout achieves better downstream finetuning accuracy for low-budget BERT large pretraining. This may indicate random- LTD can be potential regularization method for lud-btdget pretraining.
CONCLUSIONS
In this work, we propose a novel random and layerwise token dropping algorithm Candom-LTD) along with dropping schedules and new learning rate schedule. We demonstrate the efficiency of random-LTi on both GPT and BERT pretraining problems as well as GPT and ViT finetuning tasks. In addition, probe all ablation studies to verify the effectiveness of each single algorithm component used random-LTD. Furthermore, we also show the potential regularization effect introduced by random-I.TD. For the discussion the limitations and future work, see Appendix G.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ardavan Afshar, Ioakeim Perros, Evangelos E Papalexakis, Elizabeth Searles, Joyce Ho, and Jimeng Sun. Copa: Constrained parafac2 for sparse & large datasets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 793-802, 2018.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL https: www.. aclweb. org/ wnwhhhogoo//aaaa (arlooygyy11-a/ 11160.
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the are dataset. arXiv preprint arXiv: 00358, 2018
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:/905. 10044, 2019.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4): 1-200, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: / oopnnrveeveeeeeoeeeeeeeeeeeeeeee repieee .net// / fto.mm Mide-YiccFNNTTT
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint htt.::2011.00022 2020.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pp. 2337-2346. PMLR, 2019.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint (eXiv:2104.0177 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: arXiv:/706.02677 2017.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference Machine Learning, pp. 3690-3699, PMLR, 2020.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3784-3784, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18633/v1/202.. 26 2. URL https: / //acl anthooooyy...oo.oooo org/ /2lgg/222222 02222022222222222 -occclllon 1on.lllo......... 266.
10

---

Under review as a conference paper at ICLR 2023
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems. 32, 2019.
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question pairs, 2017.
URL htlps://aata. quora. com/First Larust-Quor--eeee Daliaaat-asttteeeeeeee wwwee-eeeeeeeeeieoo----eee Question-Pairs 2017.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint
2017.
Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003, 2020.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. arXiv preprint arXiv:2107.009// 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:/704.0444 2017.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. Shallow-to-dee to-depp training for neural machine translation. arXiv preprint arXiv:2010.037.7 2020.
Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum learning: regularization method for efficient and stable billion-scale gpt model pre-training. arXiv preprint arXiv:2108.06084, 2021.
Yahui Liu, Enver Sangineto, Wei Bi. Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 34: 28888-3330,, 2021a.
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu. Faster depth-adaptive transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13424-13432. 2021b.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. Computationnal Linguistics, 19(2):31-330, 1993. URL https: ttt/://wwwwwwb..//////////// http:///...//e..oooog/////----00
Regatron-DeeepSeer
http://gghtbbbooomnccccoo///nnns
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL htttps://ppet
review net/ Eorum ? Bid-By
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training, arXiv preprint arXiv:1710.03740 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02777 2018.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 1606.6603333 2016.
11

---

Under review as a conference paper at ICLR 2023
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susamnah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2019.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 3505 -3066, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8732-8740, 2020.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. arXiv preprint arXiv:2203.062/1 2022.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ing Training 2019.
parameter language models using model parallelism.
arXiv preprint 1909v
Sandeep Tata and Jignesh M Patel. Piqa: An algebra for querying protein data sets. In 15th International Conference on Scientific and Statistical Database Management, 2003., pp. 141-150. IEEE, 2003.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. 1xXiv:9055005500 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention transformer language model. arXiv preprini arXiv: 1906.01966 2019.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint 1rXiv:1905.00441. 2019.
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on Highh-Prfformance Computer Architecture (HPCA), pp. 97-110. IEEE, 2021.
Ross Wightman. Pytorch image models, mage-models, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint 17X1v:1004.05422, 2017..
Adina Williams, Tristan Thrush, and Douwe Kiela. Anlizing the adversarial natural language inference dataset. arXiv preprint arXiv:2010. 12729, 2020.
12

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the- ahe-art natural language processing. arXiv preprint arXtv:1910003771, 2019.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprini 1rXiv:1911.07770 2019.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1803-1811, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 2019.
Minjia Zhang and Yuxiong He. Accelerating training of transformer- basedd language models with progressive layer dropping. Advances in Neural Information Processing Systems, 13:14011- 2020.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprchension. arXiv preprint arXiv:1810. 12885, 2018.
13

---

Under review as a conference paper at ICLR 2023
A OTHER EFFICIENT TRAINING APPROACHES
Mixed-precision training (Micikevicius et al., 2017), different parallelism schemes (Shoeybi et al., 2019; Huang et al., 2019), and memory-effciint system design (Rasley et al., 2020; Rajbhandari et al., 2020) are the most commonly used system-related efficient training methods to train large-scale transformer models.
Besides those system-level optimizations, researchers and practitioners also investigate various efficient training methods. Gong al. (2019): Li et al. (2020) propose layer stacking speed up BERT training by gradually increasing the number layers. Zhang & He (2020); Liu et al. (2021b) extend this idea by adaptively changing the depth of the training. Following this, Shen et al. (2022) further incorporates the growth of the width along with depth for language modelings. Rae et al. (2021) tile the weights from small model to larger ei al. (2021) introduce curriculum learning to stabilize the training and get faster convergence of GPT models. Liu al. (2021a) a) propose auxiliary self-supervised task enable ViT be effectively trained on small datasets.
reduce the training cost for larger models.
Despite the remarkable success, those methods usually only demonstrate their capability on one specific application and do not show their generalizability across the wide usage of transformer models. In contrast, we extensively test random-L.TD on both pretraining and finetuning.
B COMPARE BETWEEN BASELINE, TOKENBYPASS AND RANDOM-LTD
We include the illustration of the comparison between baseline, TokenBypass, and random-LTT Fig. B.I.
(a) Baseline
(b) Our Random-LTD
(c) Token Bypass
Output tokens of l
Output tokens of layer
Output tokens layer
FFN & MHA
FFN & MHA
FFN MHA
x.
X.
Output tokens of layer
Output tokens of layer
Input tokens of layer
T
1
FFN & MHA
FFN MHA
1/2
Randomly essign Input tokens of layer
output tokens of layer
Input tokens layer
& MHA
FFN & MHA
FFN & MHA
Input tokens layer
Input tokens layer 1
Group 1 tokens
Group tokens
Divide into groups based
takenns' accumulative loss
o anseemmenn x x 2)
Output tokens of layer
& MHA
Input tokens of layer
FFN & MHA
Input tokens of layer
Figure B.1: Illustration of the transformer model for the baseline training (left), TokenBypass training (right) and random-LTD training (middle). Compared to TokenBypass, random-LTD requires no criterion on the dropped tokens and trains well for all middle layers. The box with dash line is repeated block. For both (a) and (b), the block is repeated by times, while for (c), the block is repeated by 1/2. the box, "Output tokens of layer i" is the same "Input tokens of layer 1"".
14

---

Under review as a conference paper at ICLR 2023
C FORMAL LAYERTOKEN LR DESCRIPTION
Formally, recall that the number of drop or kept tokens is ai,t or bs.t for layer L, at iteration f. Since have the same size of J; and K; for all layers, drop the subscript for simplicity. At iteration i, the total LayerToken consumed by the entire network is 2s + (1 2)b; (random-LTD is applied to all middle layers). Then the total consumed tokens is 2sT + Σ E 2)6r.
Suppose for the iteration-base baaed, the warmup iterations is Twanupp which means the consumed tokens for warmup in all layers is sl'Twarmpp Then, the warmup iterations using our LayerToken LR would be Tiwammup such that
Teieomm
2sTLTwarmup LTwarmup +
2s
Σ (1 2)b+ sIT warmup--
Similarly. for an peeration-based learning rate schedule, the decay learning schedule (e.g., linear decay or cosine decay schedules) is based on the length of T T warmup. For our LayerToken LR schedule, the corresponding length is
TETramap
2T + En- 2S LTwarmup
Σ (1- 2)b2.
Note that this proposed schedule would reduce standard learning rate if we keep b, for all layers (without random-LTDD LTD). Please see Section 5.5 for the effectiveness of our LayerToken LR as compared to standard learning rate schedule and see Fig. 6 for the learning rate schedule illustration.
D DETAILED EXPERIMENTAL SETUP
D.1 EXPERIMENTAL SETUP FOR GPT PRETRAINING
We use DeepSpeed (Rasley et al., 2020) and Megatron-Deeppppeed to train GPT models with 350 million parameters (GPT-3150M) and 1.3 billion parameters (GPT-31.35)
repository
The pretraining data are from PILE dataset Gao et al. (2020) and the total training tokens are 300 billion without extra explanation. For random-LTD, the initial dropped token length for all middle layers is 1920 (i.e., 128 tokens are kept for compute), and it decreases by 16 for every 1.75 billion training tokens. That is to say, after 210B training tokens, random-LTD degrades to standard training procedure with full sequence length. Theoretically, this can save 1/3 of the layer-token training budget. All models are trained with 64 A100-40G GPUs.
We evaluate our results on 19 zero-shot evaluation tasks, including 19 accuracy evaluation tasks (i.e., HellaSwag (Zellers et al., 2019), LAMBADA (Paperno al., 2016), TriviaQA (Joshi al., 2017), WebQS (Berant et al., 2013), Winogrande (Sakaguchi et al., 2020), PIQA (Tata & Patel, 2003), ARC (Challenge/Easy) (Boratko et al., 2018) 8), ANLI (R1/R2/R3) (Williams et al., 2020), OpenBookQA (Mihaylov et al., 2018), RACE-h (Lai et al., 2017), BoolQ (Clark et al., 2019), Copa (Afshar et al., 2018). RTE (Dagan et al., 2013), WSC (Levesque et al., 2012), MultiRC (Yaday al., 2019), and ReCoRD (Zhang et al., 2018)).
D.2 EXPERIMENTAL SETUP FOR BERT PRETRAINING
Similar to GPT pretraining, we use the -eeerpon- repository to train our BERT large models with 336 million parameters (24 layers) with sequence length 512. The pretraining recipe follows (Shoeybi et al., 2019), but the pretraining data is public, the same GPT pretraining.
For the main results in Section 4.2, the total training tokens is 512 x 1024 x 2 x 100. Here 1024 is the global batch size and 512 is the sequence length. We pretrain with 2 million iterations. We trained on 64 A100-40g GPUs (16 batch/GPU), For random-LTD-2 (random-LTD-I), the initial kept token length for all middle layers is 128 (200) for compute, and it increases by 16 for every 38 (48) billion training tokens such that we eventually save 31.1% (26.23%) layer-tokens. We take the last checkpoint to perform downstream tasks the popular stable benchmarks. including MNLI (Williams et al., 2017), QQP (lyer et al., 2017), and RACE (middle and high difficulty) (Lai al., 2017), of which the fine-tuning configurations (same (Shoeybi et al., 2019)) respectively
15

---

Under review as a conference paper at ICLR 2023
10, 12 and 3 epochs with batch-size 128, 128 and 32 and learning rate 1e-5, Se-5 and 2e-5. We report the median (best) of five repeated runs (random seeds 1234-1230 1238) in Tab. 2. Note that we apply the standard training (no random-LTTe LLD) in the downstream tasks in order to have fair comparisons.
For the ablation study of BERT pretraining in Section 5.2 and 5.7, we shorten the pretraining iteration to 0.2 million (due to limited resource) and set the maximum learning rate be 4e-4. We start with the initial sequence 256 and increase it by 16 for every billion training tokens such that we eventually save 22.3% layer-tokens. To compensate for the short-time pretraining, make the standard fine-training epoch much longer (30 epochs) for MNLI, QQP, and RACE and their batch-size (learning rate) are 128 (5e-5), 128 (5e-5), and 64 (2e-5) respectively.
D.3 EXPERIMENTAL SETUP FOR VIT FINE-TUNING
We apply random-LTT to the vision transformer (ViT) (Dosovitskiy et al., 2021) on fine-tuning tasks in order to demonstrate the broader applications of our method across different domains. We use the pretrained models published in (Wightman, 2019) and focus mainly on the two small image recognition benchmarks- CIFAR1O and CIFAR 100 (Krizhevsky et al., 2009), and one large-scale Imtaset-ImageNel (Deng et al., 2009). For ImageNet (CIFAR 10/100), we use the 12-layer (24-layer) pretrained ViT with an input resolution 224 x 224 in which each patch of size 16 x 16 such that the sequence length becomes 196++1 1 (the extra token is for position). ImageNet (CIFARIO/ 10/100) is trained on an 8-GPU (1-GPU) A100-40G machine such that the batch size is 32 (128) images per GPU. The training budget for all three datasets is 14 epochs and a small constant learning rate is used based on grid search. Particularly, the best learning rate for ImageNet is 5e-5 and CIFAR1O/CIFAL 100 is le-4. For ImageNet (CIFAR), the sequence length is started with 66 (32) and linearly reaches to the 197 full sequence length at 80% of the total training iterations such that 22.3%(30.9%) layer-token saving achieved.
D.4 EXPERIMENTAL SETUP FOR GPT FINETUNING
For the language fine-tuning tasks, we directly take the existing pretrained GPT model (350M, 24-layer) published HuggingFace (Wolf et al., 2019) and fine-tune on the three dataset: Penn Treebank (PTB) (Marcus et al., 1993). Wiki'Text-2 and WikiText-103 (Merity et al., 2017). These fine-tuning tasks are trained on a single V100 GPU with a batch size of 32 and a constant learning rate 5e-5. We trained for 15 epochs for PTB, 4 epochs for WikiTeext 103 and 10 epcoh WikiTexx-2. As for random-LTD in Tab. 5, the sequence length is started with 128 (64) sequence with a linear increase to the 1024 full sequence length at 80% (70%) of the total training iterations for PTB (WikiText-103/-2)
E STANDARD DEVIATION FOR BERT LARGE DOWNSTREAM TASKS
We show the standard deviation for BERT lurge downstream tasks finetuning in Tab. E.
Table E.1: The comparison between baseline and random-LTD on finetuning for BERT largo Comple- mentary to Tab. 2, we report the mean and one standard deviation over five independent runs with the same hyperparameters.
Method
MNLI-m/mma
QQP
RACE-m (dev)
RACE-m (test)
RACE-b (dev)
RACE-b (test)
Baseline
09,09=0.05/89.52220 52:0.11
92.2940.11 12
84,58+0.25
82.8820.57
80.54:0.29
79.01=0.45
random-i.II-I 39.75=11099999994 93
92.19=0.08
85.67+0.32
84.51
82, $2.7840.28
81.4740.33
candom-LTD-2 1/8110../897441..111
91.9620.11
86. 1410.29
84.9620.27
82.0840.19
80.7420.16
F BERT LARGE PRETRAINING WITH/WITHOUS LAYERTOKEN LR
In Section 5.5, we have seen the benefits of using LayerToken LR over the standard learning rate for GPT pretraining. Here present the additional results for BERT pretraining. The training budget is 0.2 million iterations, and the training details are given in Appendix D.2. The results are presented in Tab. F.1. which further confirms the dominant benefit of LayerToken L.R.
16

---

Under review as a conference paper at ICLR 2023
Table H.3: Comparison of applying random-LTT different layers on GPT-23500 finetuning and ViT finetuning. This is the full result of Tab. 5.
Random-LTD applied all layers except for the following
Method
dataset
None
First
Last
First and Last
PTB WikiText-2
16.00+0.02
16.01=0.03
16.09=0.02
15.92+0.02 16.9440.01
Perplexity
17.0640.02
17.01=0.02
17.01=0.02
WikiText-103
13.27#0.01
13.03+0.03
13.23=0.04
12.99+0.01
Accuracy
ImageNet-Topl
84.4720.08
84.51+0.08
84.65+0.04
84.70+0.04
Table F.1: Results of BERT pretraining with 0.2 million iterations for random-LTD with 14.1% layer-token saving. Random-LTD-34 in Tab. is the same as the last row (LayerToken LR).
Learning Pretmining results method
DownStream finetuning results
ppl(val) ppl(test)
MNLI-m/ man
QQP
RACE-m (dev)
RACE m (lest)
RACE- (dev)
RACE- (lest)
Stundard LR
5.92
86.46/86.53
91.82
72.09
73.79
73.50
71.04
LokerToken LR
5.79
5.80
87.14/87.29
91.95
78.05
77.84
73.47
71.30
G LIMITATIONS AND FUTURE WORK
We believe it is critical for every work to clearly state its limitations, especially in this area. An important limitation of this work is that we keep the dropped token ratio for all intermediate layers the same. We design it in such way to reduce the parameter tuning effort. However, each layer may have its sensitivity (see Fig. 5). Therefore, an automated dropped ratio could help here. Another limitation is that our MSLG is based self-adaptive schedule could further improve the efficiency and convergence behavior. Finally, in this work, we found out random-LTV LTD can have the potential regularization effect as the standard dropout. However, to improve its generalizability. there are still a lot of experiments to be done, which is out of the scope of this paper. We leave this as future work.
linear increasing manner. This might not be optimal, and
H FULL RESULTS USED IN MAIN TEXT
We include the full results used in main text in this section.
Table H.1: Finetuning result of ViT on ImageNet and CIFAR. This is the full result of Tab. 3.
ImageNet datasets on 12-layer VIT
CIFAR datasets on 24-layer VIT
LayerToken Saving
Top-1
Top-5
LayerToken Saving
Top-1 (CIFARI00)
Top-1 (CIFARIO)
baseline random-LLT
N/A 22.3%
84.6520.04
97.4110.02 97.4840.02
NIA 30.9%
93.93=0.30
99.3240.05
84. 84.7040.04
94.02a0.40
99.3010.03
Table H.2: Ablation study of special token treatment for BERT pretraining with 22.2% LayerToken saving. This is the full result of Tab. 4.
Keep
Pretraining results ppl(test)
DownStream finetuning results
Special Tokens
ppl(val)
MNL.Im//mm
QQP
yes 6.024 6.018
6.049
86. 70/860/6.97
91.83
no
6.040
86.66/86.92
91.97
I FULL ZERO-SHOT EVALUATION OF GPT-STYLE MODELS
We include all zero-shot evaluation results for all GPT models Tab. I.1 and 1.2.
17

---

Under review as a conference paper at ICLR 2023
Table H.4: Compare between MSLG and constant token dropping schedules. This is the full result
of Tab. 6.
datasets
CIFARIO (Metrie: Top-1 accuracy)
(Metrie: perplexity)
Token-drop schedules
constant
constant
MSLG
constunt
MSLG
LayerToken saving
16.5%
23.6%
30.8%
32.3%
23.0%
32.1%
33.7%
Performance
99.33+0.01
99.28+0.01
99.2610.08
99.3240.03
18.2740.08
20.76+0.06
15.92+0.02
Table H.5: Study the regularization effect of random-LTD. We report the average of dev and test for RACE-m and RARCE-h. * means no dropout.
Mothod
Premraning results pplival)
DownStream finetuning results
(Laycr-token saving)
ppl(tett)
MNLI-m/-mm
QQP
RACE-m (dov)
RACE-m (tesl)
RACE-h (dev)
RACE-b (test)
bascline (Nonc)
5.78
5.80
86.44/86.51 51
92.11
75.85
75.50
73.78
70,78
(None)
86.4/86.93
32.67
madom-LTTD-3 (14.1%)
6.37
6.40
85.91/85.97
91.84
74.72
73.22
70.95
68.43
ranom-LLD--3 (14.1%)
5.79
5.80
87.14/87.29
91.95
78.05
77.84
73.47
71,30
midom-1170-4 (22.366)
6.52
6.58
85.69/85.66 64
91.74
73.65
rundom tundom-LTD-4* LTD-4* (22.3%)
6.02
6.04
86.66/86.92
91.97
76.78
76.56
73.38
71.15
Table I.:: Zero-shot evaluation results of baseline and random- on GPT-3330M and GPT-31.3.
Baseline
random-LTD
Tasks
GPT-3
GPT-31.38
1.3B
GPT-3350M
GPT-31.3R
HellaSwag
39.3
52.1
40.2
51.8
LAMBADA
52.3
61.2
52.3
62.2
TriviaQA
3.6
6.3
3.14
6.05
WebQs
1.82
2.21
1.58
1.92
Winogrande
53.3
55.7
50.9
58.0
PIQA
67.0
71.1
67.2
71.0
ARC (Challenge)
25.2
29.4
24.9
28.2
ARC (Easy)
46.0
53.4
44.4
53.2
ANLI R1
32.7
32.9
33.1
33.5
ANLI R2
32.6
33.7
33.0
32.4
ANLI R3
33.8
35.1
33.7
34.6
OpenBookQA
28.8
33.4
29.6
32.6
RACE-h
29.3
33.4
31.8
34.2
BoolQ
55.6
56.4
58.4
62.7
Copa
67.0
71.0
68.0
72.0
RTE
51.6
56.7
53.1
52.7
WSC
36.5
43.3
36.5
36.5
MultiRC
0.84
0.84
0.84
0.84
ReCoRD
76.3
82.3
76.4
82.9
Average Acc
38.6
42.7
38.9
42.5
18

---

Under review as a conference paper at ICLR 2023
Table 1.2: Zero-shot evaluation results of baseline and random-LTD on GPT-31.BB with various
training budgets. Here, "Budget" in the first column means the training layer-token the final budget.
Baseline
random-LTV
Tasks/Budgets (B)
2880
4320
5760
7200
8640
2880
4800
5760
HellaSwag
46.4
49.2
51.9
52.1
52.9
49.1
51.8
53.4
LAMBADA
57.1
59.0
60.2
61.2
61.5
60.2
62.2
62.8
TriviaQA
5.36
5.58
7.73
6.3
7.2
4.79
6.05
7.46
WebQs
2.31
2.02
1.33
2.21
2.26
1.97
1.92
1.72
Winogrande
54.7
55.1
57.5
55.7
56.6
56.4
58.0
60.1
PIQA
70.1
70.1
71.1
71.1
72.3
70.5
71.0
70.7
ARC (Challenge)
26.0
26.6
27.8
29.4
28.8
26.2
28.2
28.9
ARC (Easy)
51.1
52.0
52.5
53.4
54.6
51.6
53.2
53.2
ANLI RI
33.0
31.3
32.7
32.9
32.1
33.4
33.5
33.2
ANLI R2
32.6
32.2
35.7
33.7
33.4
32.7
32.4
33.0
ANLI R3
35.2
33.8
36.9
35.1
35.8
34.8
34.6
32.9
OpenBookQA
31.2
31.2
33.0
33.4
33.6
33.6
32.6
34.2
RACE-h
32.8
34.4
34.0
33.4
36.6
34.3
34.2
35.7
BoolQ
62.0
58.9
61.5
56.4
59.9
61.7
62.7
62.8
Copa
71.0
72.0
70.0
71.0
74.0
73.0
72.0
71.0
RTE
52.0
58.1
54.2
56.7
56.3
56.7
52.7
56.3
WSC
36.5
36.5
36.5
43.3
36.5
36.5
36.5
37.5
MultiRC
0.94
2.62
0.94
0.84
1.26
0.84
0.84
0.84
ReCoRD
79.5
81.2
82.2
82.3
83.2
81.6
82.9
83.4
Average Acc
41.0
41.7
42.5
42.7
43.1
42.1
42.5
43.1
19


Reviews:
[{"review_id": "usqeEe_hXO", "reviewer": "Reviewer_kbJK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the idea of random layerwise token dropping method. Extensive experimental restuls empirically demostrate that randomly dropping tokens during pre-training preserves the task accuracy but saves the computational cost for pre-training large language models.  ", "summary_of_the_review": "This paper does have technical contributions, considering its extensive and thorough experiments on layerwise random token dropping in both BERT and GPT pretraining.\n\nI am not sure about the significance of this work. This work may be beneficial to some practitioners who want to pretrain large language models with lower costs. However, I think more extensive analysis and discussions beyond ablation studies are further required.\n\nConsidering its practical contributions, I rate this paper as a weak rejection.", "strength_and_weaknesses": "### Strengths\n\n- Simple but effective idea; The idea is really simple. The authors empirically verify their simple algorithm works well on both BERT and GPT pre-training.\n- Thorough analysis and ablation studies; This work conduct thorough analysis and ablation studies on their proposed method. It is beneficial because practitioners may gain wide insights and confidence in its feasibility from such extensive experiments.\n\n### Weaknesses\n\n- Limited significance and novelty; This paper has practical contributions to the field since the authors extensively validate their method. However, the idea of token dropping during training is already proposed in a previous work TokenBypass [1]. Although the authors already discussed and compared against TokenBypass [1] a lot in the main paper, they conduct limited comparison against TokenBypass [1] only in  GPT-2 (Figure 4). I think it is better to include the comparison also in the BERT-like model.\n- Limited Analysis on “why” it works; The authors empirically demonstrate that pretraining with random-LTD significantly lowers pretraining's computational costs while maintaining the language model's task performance. But does it apply to all models and architectural sizes? Large language models used in the experiments by the authors include BERT-large and GPT-3 1.3B. There is no proof, yet, that this approach can scale up to much larger models like the GPT-3 135B. Then, how do practitioners choose to pre-train their huge language models using this method? I don't recommend that the authors do experiments on all language model sizes; rather, I believe they should provide some explanations (or at least conjectures) for why and how the random token drop method is effective.\nConsider dropping tokens depending on their token (or word) frequency as an alternative to dropping them at random. It is assumed that the (random) token dropping works well since it eliminates some types of duplicate training during the pre-training stage if such a technique (e.g., token frequency-based dropping) outperforms random dropping.\n\n### Suggestions\n\n- Questions\n    - What is the LayerToken? I fail to find any definition on LayerToken. As far as I understand, LayerToken indicates the sum of the number of tokens used in every layer. Am I right? If not, please clarify this term.\n    - What is the baseline in Figures 3, 6, 7, and Tables 1, 2, 7, 8? Does it indicate the pre-training without any token dropping?\n- Typos\n    \n    page4: we use the LayerToken ‘compute’ the cost to measure the total → to compute?\n    \n    page8: Again, ‘We’ perform → Again, we perform\n    \n\n[1] Hou et al., Token Dropping for Efficient BERT Pretraining, ACL 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7c1XQMrmSWo", "reviewer": "Reviewer_YEPB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents random-LTD (Layer Token Dropping), a method that randomly skips the computation of certain tokens during pretraining, and evaluates it on BERT and GPT pretraining, and ViT fine-tuning.  The paper includes several ablations to show that the design decisions behind its method are necessary.", "summary_of_the_review": "This paper presents experiments showing that a token dropping method is effective, however despite being fairly extensive the experiments leave do not adequately support the claims of random-LTD's superiority to TokenBypass.  The novelty of the proposed method is also quite limited.", "strength_and_weaknesses": "Strengths\n\nThe paper's method is simple to understand, and the paper includes many experiments.  The methods are appropriately described in the context of the most similar prior methods, and both the proposed method and the prior ones are described clearly.  In the experiments, the proposed method does offer meaningful improvements over standard training, and unlike previous work this paper shows that it is effective on both BERT and GPT.  The ablations isolate interesting aspects of the design.\n\nWeaknesses\n\nThe most significant weakness in this paper is that the experiments do not adequately demonstrate the value of the technique over previous work.  We don’t see a comparison against TokenBypass on any end tasks, which given that the paper is claiming that TokenBypass performs poorly on GPT and the proposed method performs better, is a serious limitation.  The only comparison against TokenBypass on GPT is in a very limited PennTreebank perplexity evaluation, where the best TokenBypass achieves 1.3 points worse perplexity than the submission’s method.  We don’t see a comparison on end tasks with BERT or GPT pretraining on TokenBypass.  Finally, the pretraining experiments appear to involve only a single run of the proposed system and the baseline, so it is somewhat unclear how reliable the conclusions are.\n\nThe paper also claims that its method requires far less manual design effort than TokenBypass, but I was somewhat unconvinced about this claim.  While it's true that the paper's method does not require importance weights, rules for special tokens, or the “sandwich rule”, I'm not sure these differences are that significant.  This paper has its own somewhat complex design details (as the ablations show): e.g., the fraction of dropped tokens linearly decreases as training proceeds, which is important for performance.  Likewise, the authors have to invent an associated learning rate that accounts for the number of retained tokens.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "359ZMCl7RB", "reviewer": "Reviewer_1ZdE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel random and layer-wise token dropping method (random-LTD), which skips the computation of a subset of the input tokens at all middle layers. The new random-LTD method does not require any importance score-based metrics but just random selection, and hence saves computational cost. In addition, it does not require many layers in full sequence length training except the first and the last layers. This method is evaluated in several scenarios of both pre-training and finetuning using GPT-3 and BERT. ", "summary_of_the_review": "This paper proposes a novel random and layer-wise token dropping method (random-LTD), which skips the computation of a subset of the input tokens at all middle layers. The new random-LTD method does not require any importance score-based metrics but just random selection, and hence saves computational cost. In addition, it does not require many layers in full sequence length training except the first and the last layers. This method is evaluated in several scenarios of both pre-training and finetuning using GPT-3 and BERT. However, it lacks competitive baseline methods such as TokenBypass and different token selection methods (coreset based token selection) in the experimental section to justify the performance of the proposed method.", "strength_and_weaknesses": "Strength:\n1. This paper is well-written and clearly describes a novel methodology for token pruning that can be used for pre-training and fine-tuning. \n2. It evaluates the method in several scenarios of both pre-training and finetuning using GPT-3 and BERT for both NLP and vision domain. \n3. Moreover, it gives various ablation studies such as comparing random-LTD with SOTA TokenBypass and investigating if LTD-random requires special token treatment. \n\nWeakness: \n1. Insufficient baseline methods: this paper describes his main novelty in comparison to TokenBypass method (Hou et al., 2022). However, it only compares its random-LTD with a baseline model without any token pruning in the main results and the comparison with TokenBypass in the ablation studies is insufficient. In particular, it would be interesting to see the comparison results in the following setting. (a) At the same (or similar) level of computational cost, how random-LTD is performed with TokenBypass (in terms of accuracy) in both pre-training and finetuning with GPT and BERT. (b) try different values of computational cost (i.e., different values of kept token length), at each value, how random-LTD is performed with TokenBypass. It is expected that, as the computational cost becomes smaller (more tokens are pruned), accuracy drops from the baseline model (which does not have any token pruning) becomes larger. But it will be good to see that random-LTD has smaller accuracy drop from the baseline model without any token pruning, in comparison to the TokenBypass method. \n\n2. Lack of justification on using random method to select tokens. The token pruning method consists of three major parts: (1) how many token to prune at each layer (which is a essentially hyper-parameter in the random-LTD), (2) given the number of tokens to prune, which tokens are kept and which ones are pruned (i.e., token selection method), and (3) mechanism to recover full sequence length that will benefit pre-training and other fine-tuning task that require full sequence length for final prediction. It is understandable that the proposed random-LTD cannot be directly compared with inference only pruning methods such as Pyramid-BERT or PowerBERT. However, given (1) and (3) being fixed, we can compare token selection method between random method with other token selection methods such as the coreset based token selection method in Pyramid-BERT (https://arxiv.org/abs/2203.14380) or attention score based token selection method in PowerBERT (https://arxiv.org/abs/2001.08950). I.e., under different values of computation cost, replace the random method in random-LTD by coreset and attention score based token selection method.\n\nAlso, another naive baseline is given the number of tokens to prune k, truncating the first k tokens and see how it compares with the random method. Because of the dummy tokens appended at the end of each sequence to make the uniform input sequence length, truncating the first k tokens sometimes can give better performance as it simply just removes the unimportant dummy tokens. \n\nSome questions:\n1. Will the `combine` function in figure 2 preserve the original sequence order? \n2. For GLUE tasks, is the embeddings of CLS token on top of the transformer used for final classification or the pooling of embeddings from all tokens in the top layer? \n3. For random selection method, will random-LTD conduct random selection based on real tokens only or based on all tokens including dummy ones? It would be good to have those details in the paper.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1mDWDCjdeF", "reviewer": "Reviewer_JdQ2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient method to train large-scale transformer models by random and layerwise token dropping, called random-LTD. This method skips the computation of randomly selected tokens in intermediate layers accepte the first and the last layer. The schedule of sequence length and its corresponding learning rate are used for stable and efficient training. Experiments on GPT pre-training, BERT pre-training, and ViT fine-tuning demonstrate that random-LTD achieves similar (or better) performance compared to standard training while saving computational cost about 20-30%.", "summary_of_the_review": "Overall, the proposed method sounds reasonable and the justification of each component is verified with ablation study. The method is not difficult to implement. Considering the exhaustively expensive costs of training large transformer models, this paper is practically useful. I think the paper could benefit from the revised writing. \n", "strength_and_weaknesses": "random-LTD achieves reasonable computation saving while keeping accuracy. For my understanding, random-LTD is more like a regularization method as the authors also mentioned though itself also gives efficiency gain during the training. It would be great if this two effects could be differentiated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject