LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
Authors: name: Mostafa Elhoushi; name: Akshat Shrivastava; name: Diana Liskovich; name: Basil Hosmer; name: Bram Wasti; name: Liangzhen Lai; name: Anas Mahmoud; name: Bilge Acun; name: Saurabh Agarwal; name: Ahmed Roman; name: Ahmed A Aly; name: Beidi Chen; name: Carole-Jean Wu
Published: 2024-04-25 16:20:23

Abstract: We present LayerSkip, an end-to-end solution to speed-up inference of large
language models (LLMs). First, during training we apply layer dropout, with low
dropout rates for earlier layers and higher dropout rates for later layers, and
an early exit loss where all transformer layers share the same exit. Second,
during inference, we show that this training recipe increases the accuracy of
early exit at earlier layers, without adding any auxiliary layers or modules to
the model. Third, we present a novel self-speculative decoding solution where
we exit at early layers and verify and correct with remaining layers of the
model. Our proposed self-speculative decoding approach has less memory
footprint than other speculative decoding approaches and benefits from shared
compute and activations of the draft and verification stages. We run
experiments on different Llama model sizes on different types of training:
pretraining from scratch, continual pretraining, finetuning on specific data
domain, and finetuning on specific task. We implement our inference solution
and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
checkpoints at https://github.com/facebookresearch/LayerSkip.