Patch-Level Training for Large Language Models
Authors: name: Chenze Shao; name: Fandong Meng; name: Jie Zhou
Published: 2024-07-17 15:48:39

Abstract: As Large Language Models (LLMs) achieve remarkable progress in language
understanding and generation, their training efficiency has become a critical
concern. Traditionally, LLMs are trained to predict the next token in a
sequence. Despite the success of token-level training, it suffers from
considerable computational costs due to the need to process an extensive number
of tokens. To mitigate this issue, this paper introduces patch-level training
for LLMs, which reduces the sequence length by compressing multiple tokens into
a single patch. During patch-level training, we feed the language model shorter
sequences of patches and train it to predict the next patch, thereby processing
the majority of the training data at a significantly reduced computational
cost. Following this, the model continues token-level training on the remaining
training data to align with the inference mode. Experiments on a diverse range
of models (370M-2.7B parameters) demonstrate that patch-level training can
reduce overall computational costs to 0.5times, without compromising the
model performance compared to token-level training. Source code:
https://github.com/shaochenze/PatchTrain.