Low-rank Adaptation of Large Language Model Rescoring for
  Parameter-Efficient Speech Recognition
Authors: name: Yu Yu; name: Chao-Han Huck Yang; name: Jari Kolehmainen; name: Prashanth G. Shivakumar; name: Yile Gu; name: Sungho Ryu; name: Roger Ren; name: Qi Luo; name: Aditya Gourav; name: I-Fan Chen; name: Yi-Chieh Liu; name: Tuan Dinh; name: Ankur Gandhe; name: Denis Filimonov; name: Shalini Ghosh; name: Andreas Stolcke; name: Ariya Rastow; name: Ivan Bulyko
Published: 2023-09-26 19:41:34

Abstract: We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.