Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition
Journal: arXiv (Cornell University)
Year: 2023

Transformer-based models excel in speech recognition. Existing efforts to optimize Transformer inference, typically for long-context applications, center on simplifying attention score calculations. However, streaming speech recognition models usually process a limited number of tokens each time, making attention score calculation less of a bottleneck. Instead, the bottleneck lies in the linear projection layers of multi-head attention and feedforward networks, constituting a substantial portion