Beyond Next Token Prediction: Patch-Level Training for Large Language Models
Authors: Chenze Shao
Fandong Meng
Jie Zhou
Venue: ICLR 2025 Spotlight (2025)
Keywords: large language models
patch-level training
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces patch-level training to reduce the number of text units for training LLMs, where every consecutive K tokens are aggregated into a patch unit.

Abstract: The prohibitive training costs of Large Language Models (LLMs) have emerged as a significant bottleneck in the development of next-generation LLMs. In this paper, we show that it is possible to significantly reduce the training costs of LLMs without sacrificing their performance. Specifically, we introduce patch-level training for LLMs, in which multiple tokens are aggregated into a unit of higher information density, referred to as a `patch', to serve as the fundamental text unit for training LLMs. During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data at a significantly reduced cost. Following this, the model continues token-level training on the remaining training data to align with the inference mode. Experiments on a diverse range of models (370M-2.7B parameters) demonstrate that patch-level training can reduce the overall training costs to 0.5$\times$, without compromising the model performance compared to token-level training. Source code: \url{https://github.com/shaochenze/PatchTrain}.

Published as a conference paper at ICLR 2025
BEYOND NEXT TOKEN PREDICTION: PATCH-LEVEL TRAINING FOR LARGE LANGUAGE MODELS
Chenze Shao, Fandong Meng", Jie Zhou
Pattern Recognition Center, WeChat AI, Tencent Inc, China {chenzeshao, fandongmeng, witht owzhou) etecentt..tcccoooommm com
ABSTRACT
The prohibitive training costs of Large Language Models (LLMs) have emerged a significant bottleneck in the development of next-generation LLMs. In this paper, we show that it is possible to significantly reduce the training costs of LLMs without sacrificing their performance. Specifically, we introduce patch-level training for LLMs, in which multiple tokens are aggregated into unit of higher information density, referred to as 'patch', to serve as the fundamental text unit for training LLMs. During patch-level training. we feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data at a significantly reduced cost. Following this, the model continues token-level training on the remaining training data to align with the inference mode. Experiments on a diverse range of models (370M-2.7B parameters) demonstrate that patch-level training can reduce the overall training costs to 0.5x, without compromising the model performance compared to token-level training. Source code:
ctm https//ww.....eescccceeee//aattt
INTRODUCTION
Large Language Models (LLMs, Achiam et al. 2023 Touvron et al. 2023a b Team et al. 2023 Bai
al. 2023) have achieved remarkable progress in language understanding and generation, which are primarlly attributed to their unprecedented model capacity and the corresponding growth in the volume of training data they require (Kaplan et al., 2020 Hoffmann et al., 2022). However, this scaling up comes with substantial rise in computational costs, making the training efficiency of LLMs a critical concern. Despite the ongoing efforts on efficient L.LMs (Wan et al. 2023), remains formidable challenge to reduce training costs without compromising the model performance.
Specifically, the amount of compute (FLOPs) required for training LLMs is approximately propor- tional to both the number of model parameters N and the number of text units (i.e., tokens) D in the training data. This relationship can be expressed as:
C 6ND.
(1)
Therefore, strategies for reducing training costs can target either the reduction in the number of model parameters N or the number of text units D. One prominent approach to reduce the parameter size N is called model growth (Gong al. 2019 Yang al. 2020 Chen al. 2022). Rather than equipping the model with full parameters from the beginning. it advocates for a progressive expansion of the model's parameter size throughout the training phase, thereby reducing the average parameter size during training. Nonetheless, a model's performance hinges on an adequate number of extensive knowledge and develop intricate reasoning capabilities. The inadequacy of parameters inherently limits the scope of knowledge and capabilities a model can develop, rendering it challenging to match the performance of training with the full parameter set.
parameters
The second pathway to lowering training costs is reducing the number of text units D within the training data. This direction remains largely unexplored but intuitively holds more promise, as the knowledge embedded within training data is sparsely distributed across numerous tokens, with each token encapsulating a minimal amount of information. This sparse distribution of information results
"Corresponding author.

---

Published as a conference paper at ICLR 2025
cost
cost
token-level
token-lavel
1
patch-level
1
K
K
epoch
A
epoch
Figure 1: Visualization of overall training costs with patch compression for fraction À of training data and patch size K.
from scratch
patch-level training
from patch-level
2.7
Test NLL Loss
2.5
2.4
2.3
50
100
150
200
250
300
350
Training Tokens (Billions)
Negative log-likelihood (NLL) loss on test set the number processed tokens during the training of 370M-parameter Transformers.
Figure 2:
in scenario where, despite the substantial computational costs incurred during each learning step, only small fraction model parameters that are relevant the current token are effectively updated. By increasing the amount of information the model processes at each learning sep-thar is, by augmenting information density of text units and thus reducing the number of text units D-we could potentially boost training efficiency significantly without sacrificing model performance.
Building on these insights, this paper introduces patch-level training for large language models, in which multiple tokens are aggregated into unit of higher information density, referred as 'patch', to serve as the fundamental text unit for training LLMs. Specifically, we divide the training process into two stages: patch-level training and token-level training. During patch-level training, feed the language model shorter sequences of patches and train it to predict the next patch, thereby processing the majority of the training data a significantly reduced cost. The resulting parameters used to initialize the token-level model, which then continues training on the remaining data to adapt the knowledge gained during patch-level training to the token-level.
Figure I illustrates the efficiency advantage of patch-level training, where the area of the shape represents the overall training costs. With patch size of K, the amount of compute required for patch-level training is K of that required for token-level training. When fraction À of the training data is compressed into patches, the overall training costs are reduced to A/K + original costs. For instance, to halve the training costs, one could set the patch size K 4 and conduct pateh-level training on 2/3 the training data.
times the
Employing the above settings (K 4, 2/3), we train series of LLMs of varying sizes (370M- 2.7B parameters) on the Pile dataset (Gao et al. 2020). Figure illustrates the trend of NLL loss against the number of training tokens for the 370M model. After initialization with patch-level training, the model experiences a rapid decrease in loss as continues token-level training on the remaining data. Remarkably, it achieves while reducing training costs by half. By further adjusting the hyperparameter settings, even higher acceleration rates can be achieved, with only slight sacrifice in model performance.
lower loss in comparison with training from scratch,
2

---

Published as a conference paper at ICLR 2025
patch =
Σ
-logpxx))
Hogp()e)))
Hogp(Xax)
-ogp(xen
loap(xer.))
next patch prediction
model
Sequence Model
patch embedding
Po
Pr-1
average
average
token embedding
%
X1
Xx-1
KXT-K
MXT-K41
XKT-1
Figure 3: Overview of patch-level training. Every consecutive K token embeddings are averaged to form the patch embedding. The sequence model fed the patch sequence and trained to predict the next patch. The cross-entropy loss is computed based on each patch prediction vector and all the subsequent K tokens in its next patch.
PATCH-LEVEL TRAINING
In this section, we outline the patch-level training approach for large language models, as illustrated in Figure Initially, the token sequence is transformed into patch sequence by compressing every K consecutive tokens into a single patch. This patch sequence then fed into the sequence model, and the model is trained to predict all tokens in the next patch. The knowledge acquired during patch-level training is subsequently transferred to the token-level model. Specifically, we use the parameters obtained from the patch-level model initialize the token-level model, and then proceed with teken-level training on the remaining data.
While formulating the patch-level model structure, goal is to minimize the discrepancy between patch-level and token-level models, thereby ensuring that the knowledge gained during patch-level training can be smoothly transferred to the token-level model. In practice, we observed that is crucial for at least one end (input or output) of the patch-level model to remain consistent with the token-level model, which acts as an anchor align their representations. The detailed ablation is presented in section 3.7
Given the context length for token-level training, we set the context length for patch-level training as KT,, which is then compressed to a patch sequence of length T to maintain consistency with the subsequent token-level training. To avoid introducing unnecessary parameters during token-to- s--patcc compression, represent the patch embedding as the average of its associated token embeddings. Let pı the i-th patch, JıK the k-th token the i-th patch, and E the embedding function. The patch embedding is:
1 K-1
E(p.)
Σ F(TaK+)). k=
K
The patch-level model trained through next patch prediction, i.e., predicting the K tokens in the next patch. The simultaneous prediction of multiple tokens has been explored in speculative decoding, which typically employs multiple output heads and each head is responsible for predicting distinct token (Cai et al. 2024 [in et al. 2024). However, this approach would also entail additional parameters that may be unfavorable for the subsequent knowledge transfer. Instead, we maintain single output head and make its prediction cover all tokens in the next patch. Specifically, calculate the cross-entropy loss for all the subsequent K tokens based on the same patch prediction P(.lpes). resulting in the following loss function:
T K-1
Spatch =
Σ
Σ
log P(zaktslpes).
(3)
1=1
0=
3

---

Published as a conference paper at ICLR 2025
Since the model finally works at the token-level, it essential to reserve some training data adapt the patch-level model to token-level. Specifically, we conduct patch-level training on a fraction À of the training data, and then the resulting parameters to initialize the token-level model. Following this, the token-level model continues training on the remaining data to adapt the knowledge gained during patch-level training the token-level. As illustrated in Figure [ the overall training costs reduced to A/K K + 1-111 - À times the original costs of token-level training. When the amount of training data is limited, this approach can also be utilized for efficient multi-epoch training. For example, given a budget of N epochs, we can conduct patch-level training on the first N X epochs, and then switch to token-level training for N (1 X) epochs.
EXPERIMENTS
3.1 SETUP
Datasets, We evaluate our approach on standard language modeling tasks, using the Pile dataset (Gao et al. 2020) containing 360B tokens for training ! We assess the performance LLMs from multiple aspects, including their perplexity, zero-shot accuracy, and instruction-following ability. Perplexity is
calculated on the WikiText- 103 test set (Merity et al., 2017). We evaluate the zero-shot capabilities of language models 6 NLP benchmarks, including MMLU (Hendrycks et al., 2022I) HellaSwag wag
(Zellers et al. 2019). PIQA Bisk et al.] 2020), WinoGrande (Sakaguchi et al. (020). ARC-E, and ARC-C (Clark et al. 2018
For the pre-trained LLMs, conduct instruction fine tuning using the
Alpaca dataset by GPT4 (Taori et al. 2023), and then evaluate their cestructiin-followiii abilities on
MT-Bench (Zheng al. 2024).
Models. We use the Transformer backbone (Vaswani et al. 2017) and adopt most of the architecture designs from LLaMA (Touvron et al.) (202a)). We apply pre-normalization us- ing RMSNorm (Zhang Sennrich 2019), use the SwiGLU activation function (Shazeer) 2020), and rotary positional embeddings (Su et al., (021). We also apply FlashAttention- (Dao) 2024) to accelerate attention computation. We scale the model demension and obtain 4 different sizes of Transformers: Transformer-370M (hidden_size=1024, intermedi- ate 2tc_i hidden layers=4, attention 16) Transformer-780M mhideen_size=155 inter- Beediate_size-412 hidden layers,, attention Transformer- 1.3B (hidden (hidden_size=204., in- tenmediate size=5504, hidden layers=24, attention heads=16), Transformer- (hidden size=2560, intermediate size=6880, hidden layers=32, attention heads=32).
Implementation Details. Unless otherwise specified, the patch size K is 4. The context length for token-level training 2048. For patch-level training, the context length is the patch size K * 2048, The global batch size is 2M tokens, and the total number of training steps is N 180000. For patch-level training, the number of training steps is NA, and then the model proceeds with token-level training for NV x) steps. After patch-level training. only the obtained model parameters are used for initialization, and all other states like the optimizer and learning rate scheduler are reset. We use the tokenizer of I.LaMA2, whose vocabulary size is 32000. Our models are optimized by the AdamW optimizer (Loshchilov & Hutter 2019) with 31 0.9, .32 0.95, 1e 8. The learning rate is 3e 4 and the cosine learning rate schedule is applied with warmup of 2000 steps. We use a weight decay of 0.1 and gradient clipping of 1.0, and no dropout is applied during training.
3.2 MAIN RESULTS
We train a series of LLMs of varying sizes (370M-2.7B parameters) on the Pilc dataset. We employ patch-level training with the settings of K = 4, 2/3, which theoretically reduces the training costs to 0.5x. Please refer to Appendix B for the actual speed measurement. For the Transformer- 370M, we also explore other choices of À to evaluate its impact. Table I presents the performance comparison of the resulting models. Remarkably, our approach consumes only half of the compute
Previous works generally refer to the Pile dataset having 300B tokens, but actual measurement is 360B. The discrepancy likely due differences tokenizers; we use the L.L.MA2 tokenizer, which has relatively small vocabulary, possibly resulting in more tokens. The perplexity scores are also incomparable with models using other tokenizers. "haps://gituub.com wwtpeauww.oo//eeeeeeeee A Tlt.uuthrrr//mmmcaaaaaionnaaeeee
4

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison of Transformers trained on the Pile dataset. À denotes the proportion of training data used for patch-level training, with the patch size K fixed at 4. 'PPL' represents the perplexity score on the WikiText- 103 test set. For zero-shot evaluations, we report the normalized accuracy across 6 NLP benchmarks. 'Average" means the average shot accuracy.
Model Type
Cost
PPL
MMLU
HellaSwag
PIQA
WinoG
ARC-E
ARC-C
Average
Transformer3370M
1.0x
10.9
22.9
67.5
53.1
44.3
24.7
42.2
Patch 1/2)
0.625x
10.6
23.5
42.0
67.9
52.1
46.1
25.6
42,9
Patch (A 2/3)
0.5x
10.7
23.7
41.1
68.0
51.9
46.0
24.2
42.5
Patch (A 1/5)
0.4x
11.0
23.3
40.5
67.5
51.7
44.9
24.5
42.1
Transformer-
1.0x
9.2
24.4
48.5
69.0
55.4
26.7
45.5
Patch (A 2/3)
0.5x
9.1
24.1
49.1
70.6
54.8
51.3
28.2
46.3
Transformer- I.3B
1.0x
8.2
23.9
71.2
57.3
28.9
Patch (A 2/3)
0.5x
8.2
24.3
54.1
71.6
57.8
55.6
30.4
49.0
Transformer- 2.7B
1.0x
7.1
25.3
62.2
74.3
61.5
61.2
34.3
53.1
Patch 2/3)
0.5x
7.2
25.4
61.9
74.9
62.4
61.9
34.6
53.5
Writin
triting
Wirting
kekeyby
Humoniiees
Ralrg
Hurcnntte
kslepbby
Foleyday
STOH
Roseo
STEM
0.5 2,9255
STEP
00.1152253
Racaon
STEM
nooet
Direction
Math
Ivtrction
Math
Extretionn
Meth
Dorect
Math
Transformnn3770M avg anonemt -epatch (A=2/3), avg score=1.31
Transfarmen- aug Acomem +patch (A=2/3), avg scare-1.92
Transformee-1 1.38, avg soonem ppatch (A-2/3), avg score-2.08
Transformen-2 78, aug scorwe2 ppttch (A- evg score-2.88
Figure 4: Instruction-folloiing abilities evaluated on MT-bench, a multi-tun turlii-uur question set.
and incurs almost no performance loss. It matches the baseline model in terms of perplexity and even demonstrates consistent gain zero-shot evaluations, raising the average accuracy by approximately 0.5%. The model performance is also influenced by the choice of X. Within the range of values we set, a smaller À leads to better model performance but also entails larger training costs. A more detailed study the hyperparameter À will be presented in Section 3.6
We further conduct instruction fine-tuning using the Alpaca dataset by GPT4 to examine the impact of patch-level training the model's enstrctioonfolloviii ability. We evaluate our models using MT--Bench, a multi-turn question set, and present the experimental results in Figure 4 As can be seen, our approach maintains a similar instruction-folloiing ability the original models, with some experiencing a score decrease (Transformer=370M, Transformer-1.3BB and others showing an improvement (Transformer-780M. Transformer-2.7B) which can be viewed regular variations.
Our primary motivation for patch-level training is to enhance the model's knowledge acquisition efficiency. Interestingly, experimental results show that this approach can sometimes lead per- formance improvements, which is beyond our initial expectation. We initially thought that the extended context length in patch-level training contributes to the improvements. However, when we decreased the context length during patch-level training from KT 8192 to T 2048 for Transformer-370N, (A = 1/2), the model performance only experiences a slight decline (PPL 1 0.06, zero-shot accuracy J 0.2), yet still surpasses the baseline, implying that context length is not the primary factor. We hypothesize that two other factors might be responsible for these improvements: firstly, the patch-level initialization could potentially serve form of regularization; secondly, by compressing consecutive tokens into patches, the model might more effectively recognize and capture long-range dependencies due to the reduced token distance.
5

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of Transformers trained on 60B tokens for 6 epochs.
Model Type
Cost
PPL
MMLU
HellaSwag
PIQA
WinoG
ARC-E
ARC-C
Average
Transformer-3700
1.0x
11.0
23.6
40.8
66.5
50.8
44.8
25.2
42.0
Patch (A 1/2)
0.625
10.4
23.9
43.3
67.5
55.6
44.4
26.1
43,5
Patch (A 2/3)
0,5x
10.5
24.7
42.4
67.9
51.9
45,3
24.7
42.8
Patch (A 4/5)
0.4x
10.7
23.0
41.5
67.0
52.0
45.1
25.4
42.3
3.3
MULTI-EPOCHI TRAINING
Given that patch-level training consumes training data more rapidly, it is more data-hungry compared to token-level training. Consequently, it is essential to consider scenarios where training data is relatively limited and assess the performance of patch-level training when training data is reused for multi-epoch training (Muennighoff et al. (2023). We randomly extract subset of 60B tokens from the Pile dataset and increase the number training epochs N 6. In this way, the model first trained on patch-level for NÀ epochs, followed by N(1 X) epochs of token-level training.
The results presented in Table demonstrate that patch-level training maintains its superiority in terms of training efficiency and performance on multi-epoch training. Intriguingly, when both consuming 360B tokens, patch-level training for multiple epochs even outperforms its single-epoch variant in Table 1] This unexpected advantage may stem from the synergistic effect of integrating patch-level and token-level training on the same data, which likely enhances model regularization. also suggests that approach can be data-efficient by initializing the model with patch-level training for one or multiple epochs, offering a promising direction for boosting model performance.
3.4 SCALING PROPERTIES
In the above, we have validated the effectiveness of patch-level training across several model sizes (370M-2.7B). using training set of 360B tokens. However, state-of-the-an art LLMs are generally trained on model sizes and datasets that are least an order of magnitude larger than our settings. Therefore, it is crucial to know the scaling properties of patch-level training, i.e., how it performs when applied to larger training datasets and models.
In Table D notice a trend related to the model size: the performance advantage of patch-level training appears to decrease as the model size increases. describss indicating that the model with patch-level training experiences smaller performance gains from the increase in model size. On the other hand, [ublle/ppreentt the changes of cross-entropy loss when maintaining constant model size and varying the size of the training data. As the data size increases, the performance of patch-level training improves at a faster rate compared to the baseline model.
describes this trend more precisely,
Table 3: Test losses when scaling the model size from 370M to 2.7B and training the Pile dataset (360B tokens). '4' indicates the loss reduction compared to the previous model size.
Model Size
370M
780M
1.3B
2.7B
Transformer
2.392
2.217 (JO.175)
202 (40. 115)
1.961 (10. 141)
Patch (A 2/3)
2.368
2.208 (J0.160)
2.108 (40.100)
1.980 (10.128)
Table Test losses of Transformer-3707 when scaling the size training data from 45B to 360B. "J" indicates the loss reduction compared to the previous data size. The batch size is adjusted to maintain a consistent number training steps.
Data Size
45B
90B
180B
360B
Transformer
2.526
2,460 (10.066)
2.423 (J0.037)
2.392 (J0.031)
+ Patch (A 2/3)
2.553
2,468 (J0.085)
2,413 (J0.055)
2.368 (J0.045)

---

Published as a conference paper at ICLR 2025
K=2
K=4
Test NLL Loss
3.0
K-8
K=16
2.8
Wo initialization
2.4
10
20
30
40
Training Tokens (Billions)
Figure 5:
Test losses of Transformer-370M w.r.t the number of processed tokens. Models are initialized by patch-level training with patch size K.
This phenomenon can be explained from the perspective of knowledge transfer. As the data size increases, more training data is employed adjust the model from patch-level token-level, facilitating smoother knowledge transfer process. However, an increase in model size implies greater number of model parameters be transferred to the teken-level, which raises the level of transfer difficulty and necessitates more training data, Based on this explanation, patch-level training better suited for scenarios with abundant training data.
Note that the above conclusions are drawn under the settings of K 4, = 2/3, which may vary with changes in the patch size K and the patch-level data fraction X. At present, we have not identified general scaling law for patch-level training that incorporates K and X. Instead, we have made some observations regarding their effects on model performance, which will discussed in the following.
3.5 EFFECT OF PATCH SIZE
We investigate the effect of patch size under the settings of 90B training tokens, 370M model parameters, a batch size 51 512K, and X = 1/2. The results are shown in Figure Across different patch sizes, the loss curves for patch sizes K and K are nearly indistinguishable, while further increasing the patch size to 8 or 16 results in a certain performance decline. Despite this, these models still exhibit significant performance improvements when compared the model trained from scratch, which does not benefit from the initialization of patch-level training.
Overall, the patch size of performance. Considering that larger patch sizes can process more data at the same cost, we also experiment with patch-level training using K = on 45B tokens. Following this, both models proceed with token-level training on 45B tokens, and coincidentally, their loss curves are nearly identical. In this context, advantage of K lies in its data efficiency, as it achieves similar performance while consuming less data.
strikes a favorable trade-off between training efficiency and
8 on 90B tokens, which costs similar compute as
3.6 EFFECT OF À
The hyperparameter allocates the ratio training data between patch-level and token-level training. A larger À results in more tokens being compressed into patches, leading to higher acceleration rate, but it may also leave insufficient data to adjust the model to the token-level. In this section, we investigate the effect of À under the settings of 370M model parameters, batch size of 512K, and patch size of K = 4. We consider two scenarios:
1. Unlimited computational budget: We assess the impact varying À while keeping the data size constant (90B tokens). The results are shown in Figure
2. Unlimited training data: We identify the optimal under a fixed amount of computational budget (tokens patches 56.25B). For example, when À 1/2, the size of training data should be 90B tokens, with 45B tokens being compressed into 11.25B patches. The results are shown in Figure
7

---

Published as a conference paper at ICLR 2025
13.0
12.25
12.5
12.00
PPL
PPL
12.0
11.5
11.50
11.0
518
11.25
0
1/2
273
3/4
4/5
Figure 6: Effect of varying À while keeping the data size constant.
Figure 7: Effect of varying À while keeping the computational cost constant.
Figure 6 shows that the model performance initially rises and later falls as À increases, with turning point near 1/4. The performance improvements when À < 1/4 can be attributed the inherent benefits of patch-level training, as analyzed in Section 3.2 When À exceeds 3/4, further increasing leaves insufficient data to adjust the model the token-leved leading to a rapid decline in performance. Figure on the other hand, shows that when computational budget are limited, the optimal value for À is around 2/3. Note that these conclusions are specific to the current settings and should be used as reference only. The optimal À may vary depending factors such as data size and patch size. To determine the optimal value of any scenario, it essential to establish the scaling law for patch-level training.
3.7 EFFECT OF ARCHITECTURE
In the current setup, the architecture of the patch-level model is identical that of the token-level model, facilitating smoother model transfer but also leading to some concerns. For instance, the token-to-patch transformation and the prediction of the next patch do take into account the order of tokens within a patch. One may conjecture that patch-level models might benefit from adopting architectures that are better suited for representing and predicting patches, where the additional parameters introduced in such architectures could simply be discarded in the next stage.
We evaluate this strategy under the settings 90B training tokens, 370M model parameters, a batch size of 512K, and À 1/2. Specifically, we incorporate linear projection layer at both the input and output sides of the model. On the input side, the conversion of token embeddings into patch embeddings is facilitated through linear projection Win E IR Kdxd, On the output side, linear projection Wout E d Kdd is employed to transform patch-level representations back into token- level representations, followed by K softmax layers obtain the probability distribution of each token. The effects of these two modules are detailed Table
Table 5: Impact of architecture modifications in patch-level models. InputProj" and '+ OutputProji denote the incorporation of linear projections at model's input and output, respectively. 'Ptthh PPL and "Token PPL are perplexities of the patch-level and token-level model, respectively.
Model
Transformer
InputProo
+OutputProj
Both
Patch PPL
159.17
146.93
99.48
86.50
Token PPL
11.46
11.63
11.50
12.33
Overall, while these modifications are effective reducing the patch-level loss, they do not translate into benefits for the subsequent token-level training. Particularly, when linear projections are applied at both the model input and output, the performance of the subsequent token-level model significantly declines. It suggests that it is crucial for at least one end (input output) of the patch-level model to remain consistent with the token- level model, which acts as an anchor to align their representations. It also shows that there is no direct correlation between patch-level loss and the final performance of the token-level model, so a large loss during petch-level training does not imply ineffective learning. Therefore, opt preserve the original Transformer architecture for patch-level training.
8

---

Published as a conference paper at ICLR 2025
K=1-
K=2-
K=4-
K=8
K=16
Layer
Figure 8: Percentage of activated neurons for models of different patch sizes. Output neurons of each model layer (FFN output) with an absolute value greater than 0.5 are classified as activated.
3.8 NEURON ACTIVATION
In this section, we quantitatively explain why patch-level training leads better learning efficiency from the perspective of neuron activation. The training of LLMs is essentially process of embedding knowledge from the training set into the model's parameters. During this process, the model employs all of its parameters to encode every token and updates the relevant parameters based on the gradient feedback. We argue that this is an inefficient process for large models, as the knowledge encapsulated in each token is only associated with small subset of model parameters, resulting in limited number of effectively activated and updated parameters.
We substantiate this by measuring the percentage of activated neurons for models of different patch sizes, as depicted in Figure the token-level model (K =1), only small proportion of neurons are activated, suggesting that the model has enough capacity to handle text units with higher information density, thus indicating significant room for improvement in training efficiency. By grouping multiple tokens into patch, the information density processed at each step is increased, which is manifested as increased neuron activation rates. From this observation, becomes evident that patch-level training makes more comprehensive utilization of the model's capabilities, therefore demonstrating higher training efficiency.
4 RELATED WORK
Model Growth. Our approach draws inspiration from transfer learning, reducing training costs by transferring knowledge acquired at a lower training cost (patch-level) to model with higher training cost (token-level). similar strategy has been employed in studies of model growth, which train large models at relatively lower cost by progressively increasing the model size during training. For example, Gong et al. (2019): Yang et al. (2020) improve the training efficiency by transferring knowledge from low model to deep model, where model layers are progressively stacked during training. Gu et al. (202T) further proposes progressive compound growth, where the model grows multiple dimensions during training, including the context length, model width, and the number of layers. Subsequent studies primarily focus on the initialization problem during the model growth process, i.e., how to expand the small model into large one. Chen et al. (2022); Yao al.) (2024) aim to achieve function-preserviie
growth (Chen et al. (2015) that the post-growth model have the same function as the pre-growth model, which intuitively ensures smooth knowledge transfer. Wang et al. 2023); Pan et al. (2023) introduce learnable línear operators that linearly map the parameters of the small model initialize the large model. Compared model growth, patch-level training is more flexible and generalizable as it does not necessitate specialized model architectures or carefully crafted model mapping strategies. More importantly, patch-level training merely alters the information density of text units while preserving the full set of model parameters. Therefore, it enables the model to develop its complete capabilities without compromising its performance.
Multi-Tokes Prediction. Our approach improves training efficiency by concurrently predicting all tokens in the next patch. Similar attempts of multi-token prediction have been made in the past to improve the inference efficiency, including non-ac non-autoregressivee generation (Gu et al., 2018) and speculative decoding (Stem et al. 2018 Leviathan al. 2023 Chen al. 2023). Non-autoreeresssiee

---

Published as a conference paper at ICLR 2025
generation reduces the number of decoding iterations by generating all tokens at once, involving
techniques such as knowledge distillation (Kim & Rush [0166] Shao et al. 2022), training objectives
(Shao et al. 2019 2020 2021 Ghazvininejad et al.][20200 Du al. [22 Shha & Feng. [022 Ma
et al. 2023 Shao et al. 2023) ) latent modeling (Kaiser et al. 2018 Ma et al.
2019 [Shu et al. (020), iterative decoding (Lee al. 2018 Ghazvininejad et al. [209 Gu et al. (219)) and expressive model
architectures (Libovický & Helel 2018; Huang et al. [022] Gui et al. 2023). Despite the reduction
in decoding iterations, the computational demands (FLOPs) do not decrease and may even rise for
certain structures, leading to increased training costs. Speculative decoding is novel decoding paradigm for accelerating LLM inference. During each step of decoding, it drafts several future tokens efficiently and then verifies them in parallel. The model for generating draft tokens can either be a relatively small model (Leviathan et al.] [023] Chen et al. 2023) or a non-autoregressive model that generates multiple tokens in parallel (Cai et al. [224]] Lin et al. 2024, Fu et al.) 2024) Li et al. 2024). Recently. Gloeckle et al.) pointed out that besides accelcrating the inference, multi-token prediction also serves as an auxiliary training task to enhance the training signal. The key difference between our approach and these works lies in our utilization of multi-token prediction for reducing sequence length during training, with the goal of improving training efficiency. Regarding the model structure, we avoid introducing extra parameters and employ a single head for multi-token prediction.
Patch-Level Models. The concept of handling input data at the patch-level has emerged as a pivotal strategy for enhancing computational efficiency and capturing local features. Convolutional Neural Networks (CNNs. Lecun et al., 1998) are perhaps the earliest attempt of patch-level processing, utilizing kernel filters to extract local features from images. More recently, Vision Transformers (Dosovitskiy et al.) (021) have revolutionized image processing by employing CNNs to encode an image into non-overlapping image patches, thereby enabling Transformers to efficiently capture both local and global features. Similarly, speech models also rely on CNNs to compress high-frequency waveforms into hidden state representations (Baevski et al., 2020: Hsu et al., 2021). which can be interpreted as speech patches. DDoovviskkiy et al. (2021) suggested that the Vision Transformer should be fine-tuned at higher resolution, a strategy similar to ours, which improves data precision during the later phases of training. In subsequent studies, the impact of patch sizes has also garnered attention. Beyer et al. (2023) proposed to randomize the patch size at training time, which creates models capable of handling diverse patch sizes and adapting to different compute budgets. Further exploration by Anagnostidis et al. (2024) revealed that training with larger patch sizes is generally more computationally efficient. This insight led to training strategy that progressively decreases the patch size, enabling more efficient training of vision Transformers. For textual data, characters, the basic building blocks of text, can be downsampled into more compact representations (Clark et al.. 2022) or merged into tokens (Sennrich et al., 2016 Kudo & Richardson 2018). Recently, there have been attempts to further compress tokens into patches, with the model directly processing the patch sequence (Nawrot et al. 2022, Mujika 2023 Yu et al. 2024 Ho et al. 2024). However, it remains necessary to upsample the patch representation and input it into a token-level autoregressive model for the likelihood inference.
CONCLUSION
This paper introduces patch-level training, an efficient training approach for large language models, in which multiple tokens are aggregated into a unit of higher information density, referred to as 'patch', to serve as the fundamental text unit for training LLMs. During patch-level training, the model reads training data in patches and learns to predict the next patch. Following this, a small amount of training data is utilized to adjust the model to the token-level. Experimental results show that this approach can cut L.LM training costs by 50% while maintaining comparable performance.
Yet, our exploration of patch-level training is still in its infancy, and advancements in the following directions could further enhance this methodology: assessing the scalability of patch-level training by evaluating its performance on larger models and datasets; establishing empirical scaling law for patch-level training, ideally incorporating both K and X; developing advanced training techniques to accommodate larger K and X, thereby pushing acceleration rates to higher level; further investigating the potential of patch-level training in multi-epoch training; exploring the applicability of patch-level training other data modalities, such as images, speech, and video.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774 2023.
Sotiris Anagnostidis, Gregor Bachmann, Imanol Schlag, and Thomas Hofmann. Navigating scaling laws: Compute optimality in adaptive model training. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 1511-1530. PMLR, 21-27 Jul 2024. URL https:// html
proceedings wtt..p.e.//223.3 http:///w../annnossssiiaaaa22///
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 33:12449-12460) 2020.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint (ttt::2309.1660000 2023.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14496-14506 2023. doi: 10. 00.1109/CVPP52222..223.000
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-744) 7439, Apr. 2020. doi:
62399. URL
https://ojjs www.:ossaaaaa...... www.eaaaaaa.aaaieeeeeeeeeeeeaeee wwg//orggg.. php/ php/AAATI
AAAT Cat.A//rrrccle///e
6239
Tianle Cai. Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Bean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Cheng Chen, Yichun Yin, Lifeng Shang. Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang. Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2BERT: Towards reusable pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2134-2148, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18.18/ al/2022.acl-Iong. 151. htthttts:://////. UBLLattpee//acl..antaooo.y..y. wttp://orrr22222.c111oooo. acl-long. 151
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint 1rXiv:1511.05641,, 2015.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre-training an effi- cient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91 2022. doi: 10.1162/tacl 00448. URL https tacl-1.5
//acllanchllogy.orrggg
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.0:800. 2018.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In
The Tweifth International Conference on Learning Representations, 2024. URL https?
/openreview www.:/wwwwee..oreeeeee http://frrrmm11mmmmm2xxxxhh

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL /tttps://peerrvvieee
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressivv machine translation. In International conference on machine learning, pp. 2849-2859. PMLR, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Yichao Fu, Peter Bailís, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. Forty-firsi International Conference on Machine Learning, net wwwp:eeet/ooroee
2024. URL /ttttp:::/ppeerevii.seee http://wwwee..ettttrreeeeeaoo--. http://orrnnnnddiiidseeeeee
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002 2020.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6112-6121, 2019. URL https:/ /www
ac www.wb.rggg....... org/ antho lachholooo wtwp/oy//////// 9- 633
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non con-autoregressive machine translation. In Proceedings of the 37th International Conference Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3515 3523. PMIR, 2020. URL http1 //proceedings. mlr preespprsss... penpess/vl99) http//llll//g/// http:///t//aaaiiiinineeeeeeeee jad20a. html
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of BERT by progressively stacking. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 2337-23466 PMLR, 09-15 Jun 2019. URL https: http::::///pcceedingg. mlr. www.pprss//viie
19a.ng999.... html
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. International Conference on Learning Representations, 2018. URL
ht httpps://pperre/e
net. /tt////orommmmm/o/..eeee 1Cb
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11179-11189, 2019. URL wwwphhttp/sceo / httpt:ttt/p/oooooo http:::::////oceeeingg. neurips wctpappe//0199 9/hash/ 7755982062665beee
1b5 900490046666 Abstract html
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive BERT training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5174-51 5180, Online, June 2021. Association for Computational Linguistics. doi: 18.18653/v httl////200 1/202 .naacl main. 406. URL https://eet antho ogy. org/2021 naacl -ma i 406
12

---

Published as a conference paper at ICLR 2025
Shangtong Gui, Chenze Shao, Zhengrui Ma, xishan zhang, Yunji Chen, and Yang Feng. Non-autooregressive machine translation with probabilistic context-free grammar. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 5598-5615. Curran Associates, Inc., 2023. URL httpp::/pproceeeings....c neurip http://wwwprppp.ppppepeeeeeeeeee wwtppccreppppappceeee
wttrr/fflls/pape//2222 FLL/ALefIaallG338Bbbaaaa33333313 Papee Conference. pdf
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence Learning Representations, 2021. URL httpttts///peeeeeeeeeeeee net/ forum?id htwpppnnweeeeee.eeeoorutttt....
d7KB 3Gm@.
Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thome, and Se- Young Yun. Block transformer: Global-to-local language modeling for fast inference. arXiv preprint arXiv:2406.02657, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv: 15556, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3400 2021.
Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. Directed acyclic transformer for non-autoregressive machine translation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9410-9428. URL https: //tp::/:///cccediiggs waagsssmmll.... www..press/vlll https//wwlohhuaagg220
PMLR, 17-23 Jul 2022.
html
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2390-2200300 2399, StockholmsmA&xssn Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: httt:::/ppocceee.nne..n www.angss............m v880/aaiser html
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint urXiv:2001.0836/ 2020.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653W1/D16-11333
(tRthttss: www. a web.
/Dl D11--116110 61199
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1111116//1DDS 2012.. URL Dig 2012
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278 22274, 1998. doi: 10. 17995799..76
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173-1 182, Brussels, Belgium, October-Nover November 2018. Association for Computational Linguistics. doi: 10. 11111865511/D/S 149. URL https: :ttss://www /www
acl web. htt.oogg//aattaa htttthhhhooy/111111111
13

---

Published as a conference paper at ICLR 2025
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, Pp. 19274-19286. PMLR, 2023.
Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Zhuang Liu, Dong Li, Jinzhang Peng, Tian, and Emad Barsoum. Amphista: Accelerate II inference with bi-directional multiple drafting heads in a autoorrrrggrrsivv style. arXiv preprint arXiv:2406.13170, 2024.
Jindfich Libovický and Jindfich Helel. End-to-end antreurrrerresivve neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3016-3021, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/D18-1D1
URL https
77 wwww. acr.aallwen web. wewebborgg/attee otgg/anhhologgyiee-ee11
Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, and Rong Xiao. Bita: Bi-directional tuning for lossless acceleration in large language models. arXiv preprint arXiv:2401. 12522, 2024.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL htttps://oppereeeeeet net. /t///ooooo forum? id=
BkG6RicqY7
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4282 4292, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18655/v1/DD11-1433
URL https:://///mm www a c web. wwt.bbrgo//tooooogoyoooooooooo/e 9-1437
Zhengrui Ma, Chenze Shao, Shangtong Gui, Min Zhang, and Yang Feng. Fuzzy alignments in directed acyelic graph for non-autoregressive machine translation. The Eleventh International Conference on Learning Representations, 2023. CRLhttps:: httt:p:://pperevleee net/ forum?
id=LSz=90ydd0zE
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. In International Conference on Learning Representations, 2017. URL https://
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
https://openrevial httpe/rwwiwwwweeeeeee
Asier Mujika. Hierarchical attention encoder decoder. arXiv preprint arXiv:2306.01007, 20233
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1559-1571, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10. 10.18653/w122022 URL htttts:://acaaaeeeetttt
ogy- orgg 202g.2222..00 fi ngs www.nngnnnnnnnnnmmmmmm haw.:/ooon..iiiiinnnnaaaa.......
Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang. Xin Jiang, and Qun Liu. Reusing pretrained models by multi-linear operators for efficient training. In seirtt- Conference on Neural Information Processing Systems, 2023. http/httts://ppeev//ee /tpp///eeeee..e.//////e/ee net. forum?
Id=RqNXKIrWyU
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Proceedings of the AAAI Conference Artificial
Intelligence, 34(05):8732 Apr. 2020. doi: 10. w34.000 URL https:
//ojs.aasi.org/indeee wwtp:/oww..goiioieeeeeemoio
/lojs.aast
6399
14

---

Published as a conference paper at ICLR 2025
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 12.18863/11/116-11
URL https: ac antho wtw.eaanonooooooooooooo.o... ogg..oogooooooo..... 6162
Chenze Shao and Yang Feng. Non-monotonic latent alignments for etc-based non-autoregressive ma- chine translation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 8159-8173. Curran Asso- ciates, Inc., 2022.
cttpppr.ccppppppppppppppppp paper files/paper
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. Retrieving sequential information for non-autoregressive neural machine translation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 3013-3024, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1//199 1/P19/ 1288. URL https: www acl org/ anthology Paaoolloyy/PS3 1
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressive neural machine translation. In The Thirty-Fourthh AAAI Conference on Artificial Intelligence, AAAI 2020. New York, NY, USA, February 7-12, 2020, pp. 198- 205. AAAI Press, 2020. URL http::ttp//// //aaai.:///aa..... wwt.ooog//ooooo.oooooooo wwwpxiiixxx//co///ee wttpp//wwwpp.//.t//atiiaic/c/
vi ew 5351
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou, Sequence-Leve Training for Non-Autoregresssiv Neural Machine Translation. Computational Linguisticss, 47(4):891-925. 12 2021. ISSN 0891-2017. doi: 10.1 10.1162/coli.a 00421. htLhhtps::/ http://////.....0.11111.10.. coli 00421
Chenze Shao, Xuanfu Wu, and Yang Feng. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human I Language Technologies, pp. 1. Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 www.l/aaaacaaaaaa maaeel-mai.277.. URL ht https: /tRK:ttss///acaatttoooogg/ 2022. www.:anclllaaa -aaacl-main.. 277
Chenze Shao, Zhengrui Ma, Min Zhang, and Yang Feng. Beyond MLE: Convex learning for text generation. In shirth- Conference on Neural Information Processing Systems, 2023. URL
https: htttps:://peereeview net. http://eeeeeteeettto/eeee forum? fomummdd=sp dd=iddlallil 7V80uWA
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using delta posterior. In The Thirty-Fourth
AAAI Conference on Artificial Intelligence, AAAI 2020, New York. NY. USA, February 7-12, 2020. pp. 8846-8853. AAAI Press, 2020. URL
owg.aio.rg/jjo.. /ind///xxx//e/xxx////ee// wwwppppppat..... /AAI/
article/view/en 6413
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2 104. (00.:2/04.09644, 2021.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnstruction-followinn llama model. https: httpbbb.tbtbttt/// com/ tom/taa Lab/ st anford. alpaca, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, fean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk. Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
15

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092 09288888, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023.
Peihao Wang. Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learning Representations, 2023. URL
net. htt.:eeww...eeeooooomdooooooooo.
Cheng Yang. Shengnan Wang, Chao Yang. Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert training speedup. arXiv preprint arXiv:2011.13635,, 2020.
Yiqun Yao, Zheng Zhang. Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre-training. In The Twelfth International Conference on Learning Representations, 2024.
URL httpss://cperrevieet net
n htt////orummiiiee wod.:mrdddd
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.07800 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https
/openreview. net htte://oommmiddrdrppprr
Rb
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2025
A PSEUDOCODE
Model input
num_patchhs - seq_length // self. helf.atthh_izz
inputs_embeds inputs_embedds view bbatch_size, nun_patches, self.
patch_size, -1) .mea (2)
position ids position ids(:, saum_patches
Model output
9 shift_logits = Logits(.... :-1, :1 reshapo (-1, saff.coofig.vocaabe
shift_labels = labels salf.patch_size: r (-1, self. self.patch_siiae
loss =
log_probs ss.oo_softmxsshhift____ooo
dim-1)
13 for in range (se.ft size)
14
loss = loss + F.nll_loss probs, shift labelst, i1)
loss = loss / selt.patch_size
B SPEED MEASUREMENT
The practical speed-up achieved through patch-level training falls short of the theoretical x improvement, primarily due to overheads of data loading and gradient synchronization. Table gives the actual running speed of patch-level training in comparison with the token-level baseline setting (patch size=1, block size=2048, per device train batch size=4, accumulation steps=8), measured on NVIDIA 100 GPUs. Increasing the patch size to and the block length to 8192 results in a speed-up of approximately 3.8x, albeit with gradient accumulation steps should be reduced to 2 to keep the total batch size constant, which brings the speed-up down to around 3.5x due more frequent gradient synchronizations. Consequently, the actual runtime is approximately 3 cost of 0.5x.
efficiency loss due to data loading. In practice, the
2 x 3.5 +
z 0.523x, which is slightly larger than the theoretical
Table 6: Speed comparison between token-level and patch-level training.
Settings
Tokens / see
Speed-up
Baseline
247. IK
1.0x
933.5K
3.78
batch_sizezze block size=8192, accumulation 2
864.2K
3.50x
C
MIXUP TRAINING
The core idea of this work is to aggregate multiple tokens into unit of higher information density, which we refer to 'patch'. Our current strategy compresses every consecutive K tokens into single patch. Here we explore an alternative approach inspired by mixup data augmentation (Zhang al. 2018). This method involves randomly selecting K training samples and mixing their tokens position-wise to form patch sequence. Similarly, we train the model with the next patch prediction objective.
We conduct experiments under the settings of 90B training tokens. 370M model parameters, batch size of 512K, and = 1/2. The results are shown in Figure Mixup training offers some improvements over the model trained from scratch; however, its performance significantly lags behind that patch-level training. The primary difference between these methods lies in the composition of the patches, suggesting that it is more beneficial to form patches from consecutive tokens. We hypothesize that by compressing consecutive tokens into patches, the model may more effectively recognize and capture long-range dependencies, thanks to the reduced distance between tokens.
17

---

Published as a conference paper at ICLR 2025
3.2
w/o initialization
Mixup K=4
Test NLL Loss
3.0
Patch Km4
2.8
2.6
2.4
10
20
30
40
Training Tokens (Billions)
Figure 9: Test losses of Transformer-37000 w.r.t the number of processed tokens. Models are initialized by mixup training patch-level training with patch size K.
from scratch
patch-level training
from patch-level
3.1
Test NLL Loss
3.0
2.8
50
100
150
200
250
300
Training Tokens (Billions)
Figure 10: Negative log-likelihood (NLL) loss on test set w.r.t the number of processed tokens during the training of 370M-parameter Transformers. The LLaMA3 tokenizer is used.
D EFFECT OF TOKENIZER
Our approach enhances training efficiency by increasing the information density text units and reducing their quantity. Another dimension that works similarly is the choice of tokenizer, tokenizer with larger vocabulary typically represents the same data with fewer tokens. This raises interesting question about how the efficacy of patch-level training is affected by the tokenizer. The LLaMA2 tokenizer, used in our initial experiments, has a vocabulary size of 32,000. Here, we extend our study include the LLaMA3 tokenizer (Dubey et 2024). whose vocabulary size 128,000, to observe whether our approach remains effective with tokenizers that have a higher compression rate.
We continue to use the Pile dataset for the training, which comprises approximately 300B tokens when encoded with the LLaMA3 tokenizer, corresponding to about 85% of the token count compared to the LLLMA2 tokenizer. Figure 10 depicts the loss curves for the models trained using patch-level training (K X 2/3) and from scratch the token-level. Table presents a performance comparison between the two approaches. Remarkably, patch-level training remains its effectiveness, achieving slightly better performance than token-level training while halving the training cost.
The choice of patch size, we hypothesize, is also swayed by the tokenizer's compression rate. When the token representations are less compact, larger patch size might be employed to increase information density, and vice versa. We validate this hypothesis by comparing the effects of patch sizes K = and K = 4 under the settings of 75B training tokens, 370M model parameters, a batch size of 512K, À 1/2, and using the L.LMM33 tokenizer. The results are illustrated in Figure 11
18

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparison of Transformers trained the Pile dataset. The LLaMA3 tokenizer is used.
Model Type
Cost
PPL
MMLU
HellaSwag
PIQA
WinoG
ARC-E
ARC-C
Average
Transformer-310 370M
1.0x
15.48
22.9
41.2
67.3
54.1
45.6
25.8
42.8
Patch (A 2/3)
0.5x
15.32
23.1
41.1
68.1
53.2
45.9
26.2
42.9
K-4
3.4
K=2
Test NLL Loss
3.2
2.8
0
5
10
15
20
25
30
35
Training Tokens (Billions)
Figure 11: Test losses of Transformer-370M w.r.t the number of processed tokens. Models are initialized by patch-level training with patch size K. The LLaMA3 tokenizer used.
It is observed that there is slight performance gap between patch sizes K and K = this setting, whereas, in comparison, the loss curves for these patch sizes using the LLaMA2 tokenizer are almost indistinguishable in Figure This confirms our hypothesis that the suitable patch size varies with the tokenizer. Nevertheless, we posit that patch size of K = 4 is still more advantageous under this setting, offering significant efficiency improvements at the cost of minor performance trade-off.
19


Reviews:
[{"review_id": "0bHQowCeTC", "reviewer": "Reviewer_QqZw", "summary": "This paper seeks to reduce the training cost of LLMs by using patch-level training in the initial stage. A patch consists of multiple consecutive tokens, and patch-level representation can reduce training costs due to shorter sequence representations. Experiments performed across four model sizes and six NLP tasks show that the proposed method can reduce training costs by up to 50% with little to no performance degradation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Reducing LLM training costs is an important research question. The problem formulation, motivation, and improvement objective of the paper are clear.\n\n2. The proposed method is described clearly, and there seem to be adequate details to reproduce the work. The cost reduction of the proposed training method is significant and robust across multiple LLM sizes and NLP tasks.\n\n3. The experiments cover a wide range of settings such as the fraction of patch-level training (lambda), patch size (K), model size, etc.", "weaknesses": "1. The optimal hyperparameters and intuitions derived from the work may be very specific to the settings associated with the paper. For example, the optimal values of lambda and patch size may be sensitive to the tokenizer. Therefore, the improvements of the proposed work may not generalize similarly to a new tokenizer.\n\n2. It is not obvious whether the patch-level pretraining is suitable for larger scales. There are issues with training stability and convergence that only occur when the model size is scaled to several billions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cEZ5C2za9o", "reviewer": "Reviewer_zf1q", "summary": "This work introduces a novel pretraining scheme aimed at reducing computational cost of pretraining. The core idea is to reduce the effective length of training sequences by concatenating and averaging the token representation of mutliple tokens into a patch representation. Then the model predicts tokens from the next patch using the output representation of the next patch. The simplicity of this idea is very intriguing as it does not require any extra parameters in the model. After such path-level pretraining with higher efficiency authors go back to tokenlevel training in order to ensure that the model will adjust to next token prediction representation.\n\nAuthors completed experiments showing that their method scales well with larger models and supports both streaming and multi-epoch training scenario. Extensive benchmarking showed that models trained with their approach are cheaper to train and are better or on the same level of performance as baseline models.", "questions": "* Following my last weakness point, did you measure the speed up coming from patch level training in practice? e.g. in tokens per second metric or so. This would be very interesting to know.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the method is easy to implement and brings original contribution, i see it being used a lot in the community and industry if it will pass the proof of concept in real world scenario.\n2. High quality experiments cover wide range of model sizes and data settings. In addition, wide range of benchmarks are covered to evaluate the final model performance compared with usual token level training.\n3. Assuming that this approach would scale with even larger models, this work provides a significant contribution in the pretraining research.", "weaknesses": "* Unclear conclusions about the effect of architecture. In the architecture ablation the observations are suggesting that there are might be some use cases where subsequent token level training does not work very well. IMO this might be important to highlight better in the text to avoid overselling the method.\n* No numbers showing the actual speed improvements. Since all token projections are computed in patch level training, the expensive softmax operations over vocabularies have to be done. So it is interesting what is the speed improvement factor as we change K (patch size) in practice on real hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VmBBo5N62Z", "reviewer": "Reviewer_o6fE", "summary": "The paper presents an interesting approach to accelerate the costly pre-training of language models. The core idea is to change the basic unit from a token to patch which comprises several tokens. This results in computational savings for the main transformer model (the input encoding and the model prediction still produces tokens but it does so based on a patch encoding).", "questions": "* Line 266 mentions that the context length in the patch level training has little effect. Using context length KT performed as well as just T. However, these conclusions were drawn based on a very small model with 370M parameters which may not be able to exploit the larger context as well as higher capacity models. Did you perform a comparison on a larger model?\n* Does the patch level training objective have any notion of which token position is currently being predicted within a patch (eq 3)? It appears to treat a patch as a bag of words with no order? This would be interesting to contrast with a setting where there is a notion of order.\n* Section 3.5 analyses the patch size K and deems K=2/4 to have a good speed/accuracy tradeoff in your setting which is a two-stage procedure: (1) train with some K>1 and then (2) use K=1 at the end of training. However, you could also consider a more general setting where there are many more stages and where where you start with very large K and then progressively decrease K until you end up with K=1. Have you considered this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents an interesting approach to accelerate language model training, specifically:\n* the approach appears to be novel\n* the paper is well written and clear in the presentation\n* the approach is thoroughly evaluated (different (small) model sizes, including an array of ablations, an alternative approach to computing patches etc.)", "weaknesses": "* The evaluation is on the smaller scale of language mode sizes. Evaluation for larger model sizes is very challenging but it would be a large plus to understand the scalability of the method.\n* Table 3/4 show PPL results and draw conclusions based on the difference between PPLs in the range of ~10 and later on in ~7. This assumes that PPL is a linear measure but it is in fact an exponential quantity based on cross entropy. Therefore, the actual entropy difference between PPL 11 and PPL 10 is far lower than it is between, say PPL 8 and PPL 7. I would encourage the authors to simply report entropy instead of PPL for this particular analysis to make this clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mw6c1Jhjui", "reviewer": "Reviewer_kTBD", "summary": "This paper proposes a strategy to reduce training cost of LLMs using patch level training, where 'patch' refers to a sequence of consecutive tokens. The key idea is to perform training at the patch level initially whereby prior patches are used to predict the next patch, and then proceed with regular token level training.  The paper applies the approach to a range of model sizes (370M-2.5B) and shows that it can reduce the training cost by 50% without any performance degradation on model perplexity or downstream task performance. The paper presents variation in performance wrt patch size, model size, amount of training data and multi-epoch scenarios.", "questions": "* L156: \"Instead, we maintain a single output head and make its prediction cover all tokens in the next patch\". How is this done? Does your output vocabulary have a size of K*|V| where |V| is the number of symbols ?\n* Equation 3 uses 'p' to refer to both probabilities and patches. Consider using different symbols for the two.\n* Sec 3.7: Have you run experiments adding more transformer layers for mapping the token embeddings to patch embeddings and vice-versa at the output side? It is possible that a linear mapping is not powerful enough.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Presents a simple approach to reduce training cost using a patch level training for initialization the regular token-level training\n* Reports experiments on a range of model sizes showing that the approach can reduce training cost by 50% without any degradation in model performance", "weaknesses": "* There are some parts of the paper which are unclear. See questions below.\n* It would be good to see some additional analysis/ablations to understand why the approach can lead to performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)