Attention or Convolution: Transformer Encoders in Audio Language Models
  for Inference Efficiency
Authors: name: Sungho Jeon; name: Ching-Feng Yeh; name: Hakan Inan; name: Wei-Ning Hsu; name: Rashi Rungta; name: Yashar Mehdad; name: Daniel Bikel
Published: 2023-11-05 21:30:10

Abstract: In this paper, we show that a simple self-supervised pre-trained audio model
can achieve comparable inference efficiency to more complicated pre-trained
models with speech transformer encoders. These speech transformers rely on
mixing convolutional modules with self-attention modules. They achieve
state-of-the-art performance on ASR with top efficiency. We first show that
employing these speech transformers as an encoder significantly improves the
efficiency of pre-trained audio models as well. However, our study shows that
we can achieve comparable efficiency with advanced self-attention solely. We
demonstrate that this simpler approach is particularly beneficial with a
low-bit weight quantization technique of a neural network to improve
efficiency. We hypothesize that it prevents propagating the errors between
different quantized modules compared to recent speech transformers mixing
quantized convolution and the quantized self-attention modules.