Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an efficient method to train large-scale transformer models by random and layerwise token dropping, called random-LTD. This method skips the computation of randomly selected tokens in intermediate layers accepte the first and the last layer. The schedule of sequence length and its corresponding learning rate are used for stable and efficient training. Experiments on GPT pre-training, BERT pre-training, and ViT fine-tuning demonstrate that random-LTD achieves similar (or better) performance compared to standard training while saving computational cost about 20-30%.
strength_and_weaknesses: random-LTD achieves reasonable computation saving while keeping accuracy. For my understanding, random-LTD is more like a regularization method as the authors also mentioned though itself also gives efficiency gain during the training. It would be great if this two effects could be differentiated.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. However, the terms are overly defined in some cases while some terms are not exactly defined. For example, gather and combine operations in Figure 2 should be explained with equations. I understood what the term LayerToken means but I coudn’t find the formal definition.

I likes the comprehensiveness of ablation studies on all components of the proposed method in the paper.

I am curious about the performance of models trained with random-LTD with token dropping at inference time. I believe the trained model will be robust to token dropping at inference time.

summary_of_the_review: Overall, the proposed method sounds reasonable and the justification of each component is verified with ablation study. The method is not difficult to implement. Considering the exhaustively expensive costs of training large transformer models, this paper is practically useful. I think the paper could benefit from the revised writing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel random and layer-wise token dropping method (random-LTD), which skips the computation of a subset of the input tokens at all middle layers. The new random-LTD method does not require any importance score-based metrics but just random selection, and hence saves computational cost. In addition, it does not require many layers in full sequence length training except the first and the last layers. This method is evaluated in several scenarios of both pre-training and finetuning using GPT-3 and BERT. 
strength_and_weaknesses: Strength:
1. This paper is well-written and clearly describes a novel methodology for token pruning that can be used for pre-training and fine-tuning. 
2. It evaluates the method in several scenarios of both pre-training and finetuning using GPT-3 and BERT for both NLP and vision domain. 
3. Moreover, it gives various ablation studies such as comparing random-LTD with SOTA TokenBypass and investigating if LTD-random requires special token treatment. 

Weakness: 
1. Insufficient baseline methods: this paper describes his main novelty in comparison to TokenBypass method (Hou et al., 2022). However, it only compares its random-LTD with a baseline model without any token pruning in the main results and the comparison with TokenBypass in the ablation studies is insufficient. In particular, it would be interesting to see the comparison results in the following setting. (a) At the same (or similar) level of computational cost, how random-LTD is performed with TokenBypass (in terms of accuracy) in both pre-training and finetuning with GPT and BERT. (b) try different values of computational cost (i.e., different values of kept token length), at each value, how random-LTD is performed with TokenBypass. It is expected that, as the computational cost becomes smaller (more tokens are pruned), accuracy drops from the baseline model (which does not have any token pruning) becomes larger. But it will be good to see that random-LTD has smaller accuracy drop from the baseline model without any token pruning, in comparison to the TokenBypass method. 

2. Lack of justification on using random method to select tokens. The token pruning method consists of three major parts: (1) how many token to prune at each layer (which is a essentially hyper-parameter in the random-LTD), (2) given the number of tokens to prune, which tokens are kept and which ones are pruned (i.e., token selection method), and (3) mechanism to recover full sequence length that will benefit pre-training and other fine-tuning task that require full sequence length for final prediction. It is understandable that the proposed random-LTD cannot be directly compared with inference only pruning methods such as Pyramid-BERT or PowerBERT. However, given (1) and (3) being fixed, we can compare token selection method between random method with other token selection methods such as the coreset based token selection method in Pyramid-BERT (https://arxiv.org/abs/2203.14380) or attention score based token selection method in PowerBERT (https://arxiv.org/abs/2001.08950). I.e., under different values of computation cost, replace the random method in random-LTD by coreset and attention score based token selection method.

Also, another naive baseline is given the number of tokens to prune k, truncating the first k tokens and see how it compares with the random method. Because of the dummy tokens appended at the end of each sequence to make the uniform input sequence length, truncating the first k tokens sometimes can give better performance as it simply just removes the unimportant dummy tokens. 

Some questions:
1. Will the `combine` function in figure 2 preserve the original sequence order? 
2. For GLUE tasks, is the embeddings of CLS token on top of the transformer used for final classification or the pooling of embeddings from all tokens in the top layer? 
3. For random selection method, will random-LTD conduct random selection based on real tokens only or based on all tokens including dummy ones? It would be good to have those details in the paper.



clarity,_quality,_novelty_and_reproducibility: The quality of the paper is valid as it clearly describes the methodology and conduct multiple evaluations on its proposed method under different scenarios. However, it lacks competitive baseline methods to justify the performance of the proposed method. For details, see `Strength and Weakness` section.

This paper contains all the details of experiment setting such as number of runs and hyper-parameter configurations, which provides helpful information for other researchers to reproduce the work. 
summary_of_the_review: This paper proposes a novel random and layer-wise token dropping method (random-LTD), which skips the computation of a subset of the input tokens at all middle layers. The new random-LTD method does not require any importance score-based metrics but just random selection, and hence saves computational cost. In addition, it does not require many layers in full sequence length training except the first and the last layers. This method is evaluated in several scenarios of both pre-training and finetuning using GPT-3 and BERT. However, it lacks competitive baseline methods such as TokenBypass and different token selection methods (coreset based token selection) in the experimental section to justify the performance of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the idea of random layerwise token dropping method. Extensive experimental restuls empirically demostrate that randomly dropping tokens during pre-training preserves the task accuracy but saves the computational cost for pre-training large language models.  
strength_and_weaknesses: ### Strengths

- Simple but effective idea; The idea is really simple. The authors empirically verify their simple algorithm works well on both BERT and GPT pre-training.
- Thorough analysis and ablation studies; This work conduct thorough analysis and ablation studies on their proposed method. It is beneficial because practitioners may gain wide insights and confidence in its feasibility from such extensive experiments.

### Weaknesses

- Limited significance and novelty; This paper has practical contributions to the field since the authors extensively validate their method. However, the idea of token dropping during training is already proposed in a previous work TokenBypass [1]. Although the authors already discussed and compared against TokenBypass [1] a lot in the main paper, they conduct limited comparison against TokenBypass [1] only in  GPT-2 (Figure 4). I think it is better to include the comparison also in the BERT-like model.
- Limited Analysis on “why” it works; The authors empirically demonstrate that pretraining with random-LTD significantly lowers pretraining's computational costs while maintaining the language model's task performance. But does it apply to all models and architectural sizes? Large language models used in the experiments by the authors include BERT-large and GPT-3 1.3B. There is no proof, yet, that this approach can scale up to much larger models like the GPT-3 135B. Then, how do practitioners choose to pre-train their huge language models using this method? I don't recommend that the authors do experiments on all language model sizes; rather, I believe they should provide some explanations (or at least conjectures) for why and how the random token drop method is effective.
Consider dropping tokens depending on their token (or word) frequency as an alternative to dropping them at random. It is assumed that the (random) token dropping works well since it eliminates some types of duplicate training during the pre-training stage if such a technique (e.g., token frequency-based dropping) outperforms random dropping.

### Suggestions

- Questions
    - What is the LayerToken? I fail to find any definition on LayerToken. As far as I understand, LayerToken indicates the sum of the number of tokens used in every layer. Am I right? If not, please clarify this term.
    - What is the baseline in Figures 3, 6, 7, and Tables 1, 2, 7, 8? Does it indicate the pre-training without any token dropping?
- Typos
    
    page4: we use the LayerToken ‘compute’ the cost to measure the total → to compute?
    
    page8: Again, ‘We’ perform → Again, we perform
    

[1] Hou et al., Token Dropping for Efficient BERT Pretraining, ACL 2022
clarity,_quality,_novelty_and_reproducibility: - Clarity

Overall, the paper is well-written and easy to understand. However, there are some parts that are hard to understand due to the lack of explanations (see Suggestions above).

- Quality

The quality of the experiments is impressive. They conduct extensive experiments and ablation studies in a range of experimental settings. However, more discussions on the method (e.g., how and why it works even with the random policy? or more comparisons against TokenBypass) should be further made.

- Novelty

The idea is not entirely novel. However, they empirically show that such a simple algorithm works well on both BERT and GPT pretraining. It is clear that this method has not yet undergone thorough research.

- Reproducibility

I cannot find the reproducibility statement in the main manuscript. The authors do not provide any codes or software to reproduce their results. However, the authors do provide some details on the implementations and experimental setups in the Appendix.
summary_of_the_review: This paper does have technical contributions, considering its extensive and thorough experiments on layerwise random token dropping in both BERT and GPT pretraining.

I am not sure about the significance of this work. This work may be beneficial to some practitioners who want to pretrain large language models with lower costs. However, I think more extensive analysis and discussions beyond ablation studies are further required.

Considering its practical contributions, I rate this paper as a weak rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents random-LTD (Layer Token Dropping), a method that randomly skips the computation of certain tokens during pretraining, and evaluates it on BERT and GPT pretraining, and ViT fine-tuning.  The paper includes several ablations to show that the design decisions behind its method are necessary.
strength_and_weaknesses: Strengths

The paper's method is simple to understand, and the paper includes many experiments.  The methods are appropriately described in the context of the most similar prior methods, and both the proposed method and the prior ones are described clearly.  In the experiments, the proposed method does offer meaningful improvements over standard training, and unlike previous work this paper shows that it is effective on both BERT and GPT.  The ablations isolate interesting aspects of the design.

Weaknesses

The most significant weakness in this paper is that the experiments do not adequately demonstrate the value of the technique over previous work.  We don’t see a comparison against TokenBypass on any end tasks, which given that the paper is claiming that TokenBypass performs poorly on GPT and the proposed method performs better, is a serious limitation.  The only comparison against TokenBypass on GPT is in a very limited PennTreebank perplexity evaluation, where the best TokenBypass achieves 1.3 points worse perplexity than the submission’s method.  We don’t see a comparison on end tasks with BERT or GPT pretraining on TokenBypass.  Finally, the pretraining experiments appear to involve only a single run of the proposed system and the baseline, so it is somewhat unclear how reliable the conclusions are.

The paper also claims that its method requires far less manual design effort than TokenBypass, but I was somewhat unconvinced about this claim.  While it's true that the paper's method does not require importance weights, rules for special tokens, or the “sandwich rule”, I'm not sure these differences are that significant.  This paper has its own somewhat complex design details (as the ablations show): e.g., the fraction of dropped tokens linearly decreases as training proceeds, which is important for performance.  Likewise, the authors have to invent an associated learning rate that accounts for the number of retained tokens.

clarity,_quality,_novelty_and_reproducibility: The basic description of the methods is quite clear.  However, other parts of the paper were less clear.  In particular, I found it hard to understand the reasons why random-LTD would be expected to outperform TokenBypass on GPT, and some of the material on the experiments seemed to list large amounts of detail while sometimes not making it explicitly clear what the key question being answered by the experiment was, and why the experimental setup was justified.

Regarding novelty, the methods are essentially a small tweak on TokenBypass; the experimental measurements could be valuable but the methodological novelty is very limited.

Reproducibility of the experiments seems adequate to me.
summary_of_the_review: This paper presents experiments showing that a token dropping method is effective, however despite being fairly extensive the experiments leave do not adequately support the claims of random-LTD's superiority to TokenBypass.  The novelty of the proposed method is also quite limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents random Layer Token Dropping, a method that randomly skips the computation of certain tokens during pretraining. Experiments on GPT and BERT pre-training, and ViT fine-tuning demonstrate that similar or better performance compared to standard training while being more efficient. The reviewers found the work to be well motivated, and found to work to be valuable in saving compute while not hurting performance. However, reviewers raised several concerns regarding lack of depth in experiments including baselines and  lack of comparisons with highly relevant work. 
justification_for_why_not_higher_score: Most reviewers complained about lack of baselines and missing a highly relevant prior work.
Decision: Reject