Low-rank Adaptation of Large Language Model Rescoring for
  Parameter-Efficient Speech Recognition
Authors: name: Yu Yu, name: Chao-Han Huck Yang, name: Jari Kolehmainen, name: Prashanth G. Shivakumar, name: Yile Gu, name: Sungho Ryu, name: Roger Ren, name: Qi Luo, name: Aditya Gourav, name: I-Fan Chen, name: Yi-Chieh Liu, name: Tuan Dinh, name: Ankur Gandhe, name: Denis Filimonov, name: Shalini Ghosh, name: Andreas Stolcke, name: Ariya Rastow, name: Ivan Bulyko

We propose a neural language modeling system based on low-rank adaptation
(LoRA) for speech recognition output rescoring. Although pretrained language
models (LMs) like BERT have shown superior performance in second-pass
rescoring, the high computational cost of scaling up the pretraining stage and
adapting the pretrained models to specific domains limit their practical use in
rescoring. Here we present a method based on low-rank decomposition to train a
rescoring BERT model and adapt it to new domains using only a fraction (0.08%)
of the pretrained parameters. These inserted matrices are optimized through a
discriminative training objective along with a correlation-based regularization
loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is
evaluated on LibriSpeech and internal datasets with decreased training times by
factors between 5.4 and 3.6.