Accelerating Transducers through Adjacent Token Merging
Authors: name: Yuang Li, name: Yu Wu, name: Jinyu Li, name: Shujie Liu

Recent end-to-end automatic speech recognition (ASR) systems often utilize a
Transformer-based acoustic encoder that generates embedding at a high frame
rate. However, this design is inefficient, particularly for long speech signals
due to the quadratic computation of self-attention. To address this, we propose
a new method, Adjacent Token Merging (A-ToMe), which gradually combines
adjacent tokens with high similarity scores between their key values. In this
way, the total time step could be reduced, and the inference of both the
encoder and joint network is accelerated. Experiments on LibriSpeech show that
our method can reduce 57% of tokens and improve the inference speed on GPU by
70% without any notable loss of accuracy. Additionally, we demonstrate that
A-ToMe is also an effective solution to reduce tokens in long-form ASR, where
the input speech consists of multiple utterances.