Multi-Modal Retrieval For Large Language Model Based Speech Recognition (2024)

Abstract: Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multimodal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to 50% improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve stateof-the-art recognition results on the SpokenSquad question answering dataset.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 4435–4446 August 11-16, 2024 ©2024 Association for Computational Linguistics
1

heading: 1 Introduction
text: The wide adoption of large language models (LLMs) has driven new application areas levering this technology. One such direction is jointly modeling multi-modal inputs and outputs with a single generative LLM model. In the speech domain, models such as those proposed by Rubenstein et al. (2023), jointly model text and audio by tokenizing speech signals into discrete units. With an expanded vocabulary encompassing tokens of multiple modalities, this modeling approach has been used in both single task (Xue et al., 2023) and multi-task settings, with Maiti et al. (2023) arguing that the multi-task training of speech-LLMs improves overall generalization of the model through synergies across tasks and modalities. With an
1These two authors contributed equally to this work.
expanded vocabulary encompassing tokens of multiple modalities, this modeling approach has been used in both single task (Xue et al., 2023) and multitask settings, with Maiti et al. (2023) arguing that the multi-task training of speech-LLMs improves overall generalization of the model through synergies across tasks and modalities.
With generalization capabilities, multi-modal LLMs such as AudioPalm (Rubenstein et al., 2023) and Seamless (Barrault et al., 2023), have targeted many tasks, including Automatic Speech Recognition (ASR). These models rely on the decoder language model (LM) to generate the output text transcription when consuming tokenized speech as a prompt. Because the multi-modal LLM approach can leverage the use of a large text-only corpus for training and multi-tasking, the approach has an advantage compared to traditional ASR models such as recurrent neural network transducers (RNN-T) (Makino et al., 2019) or whisper-like architectures (Radford et al., 2023), which rely primarily on paired audio-transcription data. However, enterprise-grade ASR systems often include further advances, such as functionality to incorporate auxiliary information to assist decoding accuracy, which has yet to be fully addressed with these new multi-modal LLM-based approaches.
Perhaps the two most common approaches for incorporating auxiliary information for ASR have been shallow fusion with an external LM (Gourav et al., 2021; Zhao et al., 2019; Le et al., 2021) and contextual biasing (Sathyendra et al., 2022; Liu et al., 2021; Chang et al., 2021). Shallow fusion with an external LM is a modular way to bias the ASR model at inference time by interpolating the probability distribution of the ASR model with that of the external LM. Shallow fusion though, can suffer from a loss of generality since the external LM does not have direct access to acoustic information. Neural biasing, meanwhile, resolves this issue by ingesting the auxiliary information directly into the
4435
acoustic model training (Sathyendra et al., 2022). Yet, neural biasing is ultimately limited in the number of contextual documents it can ingest since attention over an ever larger number of documents renders the method less effective through dilution. We aim to address both of these limitations in this work through retrieval augmentation.
Retrieval augmentation is a well known approach to improve existing LMs for ingesting additional information (Mialon et al., 2023; Khandelwal et al., 2019; Guu et al., 2020; Borgeaud et al., 2022; Wang et al., 2023a; Zhong et al., 2022; Wu et al., 2020; Zhang et al., 2022a; Karpukhin et al., 2020; Yasunaga et al., 2022). The overarching idea of all retrieval augmented models is to use an external knowledge base, i.e. retrieval corpus, to improve LM performance. During inference, the retrieval corpus is queried for relevant context and information. The query usually consist of a key computed using an encoder model followed by a search step to find the closest neighbors to the key - typically in a cosine similarity or Euclidean distance sense. The retrieved neighbors are provided to the retrieval augmented LM as additional inputs, which can be used as prompts or cross-attended over.

heading: 1.1 Contributions
text: In this work, we show that using multi-modal retrieval can improve results significantly over canonical text based retrieval. Specifically, we demonstrate our method for speech recognition tasks in two settings 1) Ingesting dynamic multi-modal information; and 2) Domain adaptation of the multimodal LLM. We propose and detail two retrieval approaches to achieve this result: a kNN-LM and a cross-attention based neural model. Experimentally, we compare each retrieval approach using two model sizes: a small model with 300 million parameters (Zhang et al., 2022b); and a larger model with 7 billion parameters. We ultimately demonstrate that while both approaches are capable of significant reduction of word error rate (WER) for domain adaptation, only the cross-attention model improved consistently speech recognition performance for the dynamic information task. We also show that multi-modal LLMs can be used effectively as key encoders for nearest neighbour search, removing the need to use an external neural model as encoder. This result leads to a deployable, application-friendly simplification which has compelling savings of compute resources.

heading: 1.2 Related Work
text: One of the first retrieval augmented LM was kNNLM that used retrieved results to directly augment token softmax probabilities (Khandelwal et al., 2019). Since the kNN-LM did not use a neural network to ingest the dynamic information, the method is easy to apply on existing models, but limited the performance compared with more involved models.
Subsequent models such as RETRO (Borgeaud et al., 2022) or REALM (Guu et al., 2020) used a cross-attention based mechanism to incorporate the retrieved context into the causal and maskedLMs. RETRO devised a chunked cross-attention to retrieve text continuations which also allowed it to scale to a very large knowledge base, when compared with the kNN-LM.
For speech-recognition applications, Zhou et al. (2023) used a modified kNN-LM. In this work, the authors used a Connectionist Temporal Classification (CTC) decoder to create retrieval keys as opposed to an LM used in the standard kNNLM (Khandelwal et al., 2019). This change enabled the keys to have acoustic information, but limited the training data to consist only of transcribed speech compared with multi-modal LMs which can utilize both modalities independently. The output probabilities were computed in the same manner as in the standard kNN-LM. Further, retrieval methods have also been successfully applied in cold fusion (Yusuf et al., 2023). With a pre-trained LM as the key encoder, partial hypotheses from the decoder were used to search for text continuations, followed by contextual biasing for generating the transcription. However, the key encoder lacked phonetic context making the retrieved token accuracy low for the initial tokens and at entity start positions. In contrast, we will demonstrate that this limitation can be overcome by using a multi-modal LM instead of a pre-trained text-only LM, by incorporating the audio information into the retrieval context.
Chan et al. (2023) built key-value databases from semantic text and its corresponding text-to-speech (TTS) audio embeddings. The text and TTS embeddings were independently created using two different models. The retrieval database was used with approximate k-nearest neighbour search to bias the ASR model using attention. Meanwhile, Wang et al. (2023b) experimented with a multi-modal LM and Speech2Entity retriever. The retriever, however,
was not strictly multi-modal because keys were acoustic encoding of speech from a CTC model, later used to retrieve a set of textual candidate entities as values.

heading: 2 Proposed Approach
text: In the following sections, we describe the multimodal speech-LLM and the modelling approaches to incorporate the retrieved context.

heading: 2.1 Multi-Modal Language Models
text: Speech based multi-modal LMs model speech using quantized discrete audio tokens in addition to text tokens (Rubenstein et al., 2023). The discrete audio tokens are extracted from pre-trained HuBERT embeddings (Hsu et al., 2021) followed by k-means clustering. This is illustrated in Fig. 1.
For speech recognition, the multi-modal LM is decoded by concatenating the audio tokens x with a prompt p and generated text tokens y to form model inputs z:i = [p0, · · · , x0, · · · , y0, · · · , yi−1]. Next token probability is predicted by:
PSLM (yi) = softmax(Eo(f(z:i)), (1)
where PSLM (yi) is the posterior distribution of the next token yi; f(z:i) is the multi-modal LM’s last hidden state for token yi−1; and Eo is the output embedding matrix that projects the hidden state to the vocabulary dimension. The first text token - y0 - is a special start-of-sentence token and the
generation is continued until an end-of-sentence token is obtained or maximum sequence length is reached. In this work, the maximum sequence length is 2048 for all models.

heading: 2.2 Retrieval for Multi-Modal Language Models
text: We consider two retrieval augmented models in this study: a kNN-LM (Khandelwal et al., 2019), and a novel neural cross-attention based model. Both models aim to augment the posterior token distribution via dynamic information retrieved based on the prior tokens and follow the same retrieval search process. However, they differ in the way retrieved values are constructed and used.

heading: 2.2.1 kNN-LM
text: kNN-LM can be directly applied for speechrecognition with an exception of formatting the inputs as described in the prior section. The principal idea of kNN-LM is to directly modify the token softmax probabilities by interpolating them with a multi-modal distribution constructed from the retrieved neighbors:
PkNN−LM = αPSLM (yi) + (1− α)PkNN , (2)
where α is a scalar and PkNN is the probability distribution predicted from the retrieved neighbors given by:
PkNN ∝ ∑
j∈D(f(z:i)) 1yi=tje
−β∥kj−f(z:i)∥. (3)
Here tj is the retrieved next token; kj is its key embedding; D(f(z:i)) is a set containing retrieved indices; β is a constant used to normalize the Euclidian norm; and 1yi=tj is the indicator function. Retrieved neighbors are obtained by finding the closest keys (in Euclidian sense) kj to the models last hidden state f(z:i). In this work, we used β = 10−3 for all kNN-LM experiments and α is found by minimizing cross-entropy in a tuning data set.
For the prompt p and the audio tokens x we do not perform any retrieval, and retrieve the first neighbors starting with the first generated token y0 - a start-of-sentence token. After this token, retrieval is performed for all tokens until end-of-sentence token or maximum sequence length is reached.

heading: 2.2.2 Cross-Attention Based Retrieval
text: The cross-attention model consists of three submodules: 1) fr is a retrieval augmented decoder model; 2) g is a key encoder model; and 3) h is a value encoder model. The interplay between these models is illustrated in Fig. 2a. The retrieval augmented decoder model is used to decode the tran-
scription from the prior tokens (including a prompt and audio tokens). The key encoder model is used to encode all retrieval keys similar to kNN-LM while the value encoder model is used to encode the context documents.
For each token yi we encode the sequence z:i using the key encoder model g to obtain a key embedding ki. The encoder model is a multi-modal LM that uses the same tokens and audio tokenizer as the decoder model fr. The last hidden state of the multi-modal LM is used as the key embedding ki for retrieval lookup in the same way as in the kNN-LM model.
The retrieved values vi,j corresponding to the key ki are contiguous token sequences of fixed length extracted from a window around the key token from the corpus. The value tokens may include both speech and text tokens as opposed to individual tokens used in the kNN-LM. The retrieval database construction is illustrated in Fig. 2b. Each value sequence vi,j is encoded by the value encoder model h to obtain an encoded embedding ṽi,j = h(vi,j).
In this study, we employ a small BERT (Devlin et al., 2018) model with two transformer layers as the value encoder model (other hyper-parameters matched key encoder model g). Pooling the BERT embeddings is done by selecting the token embedding that follows directly after the token used for
computing the corresponding key. In this study, the context document length is fixed, hence this translates to selecting a token with a predefined index.
The encoded value vectors ṽi,j are stacked to form a context value matrix ṽi = [ṽi,0, · · · , ṽi,n]. The context value matrix is used as an input to the decoder model, fr, token level cross-attention layers as the key and value matrices as illustrated in Fig. 3. More formally, for token i we have:
P (yi) = softmax(Eo(fr(z:i, ṽ:i)) (4)
The retrieval augmented decoder fr is constructed from a multi-modal LM by adding a token level cross-attention block with a normalization layer to selected transformer layers before the selfattention as illustrated in Fig. 2a. In this work, we used the four topmost layers for the cross-attention blocks. During training, the original transformer weights are frozen and only the new parameters are updated. In this regard, the cross-attention layer can be considered as an adapter for new functionality.
We used pre-layer norm (Xiong et al., 2020) for the query inputs of the cross-attention block (see Fig. 2a) and applied an additional mask on the outputs that is constructed from the pooled context documents. If all context documents for a given token are omitted, we would zero out the output vector. This procedure in combination with the prelayer norm and the parameter freezing guarantees that when there is no context documents provided the model predictions for all tokens matches the underlying LM.

heading: 3 Experiments
text: In the following sub-sections, we describe datasets used for experiments, model adaptation specifics, and retrieval data construction.

heading: 3.1 Datasets
text: We investigate retrieval augmentation using Spoken-Squad (Lee et al., 2018) and Spoken Language Understanding Evaluation (SLUE) Voxpopuli (Shon et al., 2022). These datasets differ in utterance lengths and topics widely, allowing us to gauge the models in wide range of applications. For, Spoken-Squad, we used Amazon Polly TTS service to synthesize speech for questions and answers.
We applied a simple text normalization for all datasets: lower-casing and punctuation removal. In the case of Spoken-Squad, since the test partition was not available, we used validation partition for testing.

heading: 3.2 Models
text: We use two base multi-modal LMs in this work: a small model based on the public OPT model with 330 million parameters (Zhang et al., 2022b); and an internal larger model using the Llama architecture (Touvron et al., 2023) with 6.8 billion parameters. Model attributes are listed in Table 1.
Speech was encoded into continuous vectors with a pre-trained HuBERT model (Hsu et al., 2021) with ∼ 1 billion parameters and further discretized using 2000 k-means centroids. Text was tokenized using the corresponding sentence piece model for all models. We used greedy search for the small model and beam search with beam width of two for the large model.

heading: 3.3 Model Training and Fine-Tuning
text: The multi-modal LMs were first pre-trained with a large text corpus. Pre-training setup for the small model is identical to Zhang et al. (2022b). The large model was pre-trained using RedPajama (Computer, 2023) with an exception that the books subset was replaced by internal text corpus. Training hyper-parameters are obtained from (Touvron et al., 2023).
After text pre-training, LM vocabularies were extended with the speech tokens followed by multitask training. The small model was trained using multi-lingual Libri-Speech (Pratap et al., 2020) while the large model was trained using: multilingual Libri-Speech, Libri-Light (Kahn et al., 2020), People-Speech (Galvez et al., 2021), a largescale multilingual speech-to-text translation corpus
(CoVOST2) (Wang et al., 2020), Tedlium (Hernandez et al., 2018), and internal audio data. For all multi-tasking models, the training tasks included speech continuation, text continuation, speech recognition (ASR), and speech generation from text (TTS) with equal weights assigned. The training setup is similar to VoxtLM (Maiti et al., 2023). Task specific prompts are listed in Table 2.
For the cross-attention models, the underlying speech multi-modal LM parameters were frozen, with only the value encoder transformer layer and cross-attention adapter blocks trained. The value encoder embedding layer was initialized with weights from the corresponding multi-modal LM input embedding layer; for the small model we used the input projection layer to up-project the embedding to the hidden size whereas for the large model input embeddings were used directly.
The cross-attention model training was divided in two stages: 1) Starting with randomized weights, the model was trained with pre-training tasks and cross-entropy loss along with context extracted from a random document. For half of the tokens, one of the random context documents contained the correct next token while the rest were incorrect. This training approach helped the model distinguish between relevant and irrelevant context. 2) Next, the model was fine-tuned using retrieved context from SLUE Voxpopuli training partition and Spoken-Squad context paragraphs. For SLUE Voxpopuli, the training partition was split in two parts: one part used for the training samples and the other part to construct the retrieval corpus (along with all the context documents from Spoken-Squad).

heading: 3.4 Retrieval Data Construction
text: Retrieval data is constructed from the corresponding train partition of the datasets with the exception of Spoken-Squad where the context paragraphs are used. Audio-transcription pairs are encoded in ASR format shown in Table 2 for multi-modal memory. Given an audio-text pair, the keys for
each text token are the corresponding hidden states obtained after a forward pass with the entire speech as prompt. For text-only memory, transcriptions and contexts are encoded in text format and keys are obtained in a similar way without using speech as prompt. For both text-only and multi-modal retrieval memory, keys are encoded using the same model. Note that in both cases, the number of retrieval keys remains fixed because they correspond to the text tokens, but with the difference being whether audio was used as prompt or not. FAISS library (Johnson et al., 2019) is used for nearest neighbor search with a Voronoi based index.
For kNN-LM, the retrieved values for a key are the corresponding next tokens and their Euclidian distance from the retrieval query. For the crossattention model, the values included the tokens within a fixed widow, namely, seven tokens preceding the token used for creating the retrieval key and also the following eight tokens.

heading: 3.5 Optimization
text: The pre-training set-up is taken directly from the OPT and Llama papers. For the small OPT model, we used 1.0e−5 learning rate with exponential decay for training the cross-attention layers. Warmup steps were set to 500, with a total of 10,000 steps. For the OPT model, we used 4 nodes with 8 GPUs (A10G) for training, with micro batch-size of 1 and 2 steps of gradient accumulation resulting in 64 as global batch size. For the larger Llama model, we reduced the learning rate to 5e−6 and increased the number of GPU nodes to 8 (increasing the global batch size to 128). We used bfloat-16 precision for training both models.

heading: 4 Results and Discussion

heading: 4.1 Effect of Corpus Modality on Retrieval
text: We compare the effect of corpus modality using Spoken-Squad validation partition and quantify the recall of the transcription tokens in the retrieved values.
Retrieval recall statistics are shown in Table 3. The percentages show the fraction of the retrieved tokens matching the next token (relative to retrieval key) in the transcription. The next token is predicted with a high degree using a multi-modal corpus (69% and 85%) as opposed to the text-only corpus. We believe this is due to the corpus having both acoustic and semantic information. Higher number of retrieved documents increase the recall
slightly for the multi-modal corpus as opposed to the text-only corpus, suggesting that lesser number of retrieved documents (and consequently compute) can be effectively used in applications, when using multi-modal memory. The subsequent tokens are predicted with significantly lower accuracy and recall than the first token, which can be attributed to the fact that LMs are trained to predict the next token but not the subsequent ones.
This result has two implications: (1) the retrieved documents likely work well on a token level model predicting the next such as the kNN-LM; (2) models relying on chunks such as chunked crossattention used in RETRO (Borgeaud et al., 2022) are likely to have performance reduction when compared with token level models when the retrieved documents are obtained from a multi-modal autoregressive LM.
We demonstrate the impact of the retrieval modality on ASR with the large multi-modal kNNLM. For Spoken-Squad validation partition, using text-only memory for decoding results in a higher WER of 17.9% when compared to the multi-modal memory case, which achieves a WER of 16.5%. Corresponding 1-best retrieval accuracy is 34% and 85% respectively as shown in 3. Hence, acoustic information is principally important for retrieval key construction in speech recognition applications using speech-text LLMs.
Table 3 also shows the retrieval statistics with the bert-large-uncased model (Devlin et al., 2018), which has similar number of parameters (336 million) as the small speech-text LM. For text-only retrieval corpus, the speech-text LM has higher recall for the next token (Index 1) for 1-best and 8- best neighbors compared to the BERT model . For subsequent tokens, BERT show a less steep decline in recall and fares better in terms of absolute recall values. When both speech and text are used for the retrieval corpus, the speech-text LM has higher
1-best and 8-best recall for all considered tokens (indices one to four).

heading: 4.2 Speech Recognition
text: We investigate our retrieval approaches in speech recognition setting. For these experiments, we only consider a multi-modal corpus for retrieval as it produced superior recall compared to the text-only corpus.
Table 4 shows the WER evaluated on all datasets, which are grouped by their relationship to the multimodal LMs training. Adaptation refers to the stage of adding speech-modality to the pre-trained text LM by extending its vocabulary with that of the audio tokenizer and training with ASR, speech continuation and TTS tasks. Fine-tuning refers to the training (Cross-Attention) of the speechtext LM from the Adaptation phase with retrieval. The speech adaptation datasets were not used for retrieval fine-tuning. Training partition of the datasets was used to construct the retrieval corpus. For Libri-Speech, Tedlium and SLUE-Voxpopuli, this corresponds to audio and transcription pairs and for Spoken-Squad, it is the context paragraphs’ audio and text data over all Spoken-Squad titles. In the variant, Spoken-Squad (paragraph), we limit the retrieval corpus to each question/answer’s corresponding context paragraph.
The small kNN-LM model show consistent improvement over the baseline with an exception of the Libri-Speech other dataset. We observe similar trend also for the large model with an exception that both Libri-Speech datasets degrade slightly. This can be explained by the strong in-domain baselines - in particular for the large model. The WER Reduction (WERR) from kNN-LM ranges from single digits to ∼ 40% depending on the dataset and the baseline model.
For dynamic context in case of Spoken-Squad (paragraph), kNN-LM showed mixed results. For
the small model we observe relative degradation of 13%, as opposed to improvement of of 23% for the large model. This discrepancy can be attributed to underlying LMs retrieval recall (see Table 4) and the interpolation weight. In particular, re-tuning interpolation weight for the paragraph level SpokenSquad would guarantee that the model does not degrade the baseline.
The cross-attention (CA) model requires training and the performance is impacted by the finetuning data and the retrieval corpus. This aspect is magnified in the large model that has a large number of trainable parameters from the value encoder and cross-attention layers. In the case of fine-tuning data and retrieval corpus overlapping (Spoken-Squad, SLUE-Voxpopuli), the CA model performs better than the kNN-LM. For SpokenSquad, this is most prominent where the large CA model out performs all the other models by a wide margin.
The larger improvement from CA model compared with kNN-LM can be attributed to two main factors: 1) the model has more trainable parameters (about ∼ 40M parameters for the small model and ∼ 400M for the larger model) from the valueencoder and the additional CA blocks in the upper decoder layers; 2) kNN-LM is unable to discriminate incorrect context based on the context tokens and relies solely on the key distance while the crossattention and value encoder allow more complex interactions between the context documents and the input tokens. With this context, one can conclude that the cross-attention approach tends to be a better candidate than kNN-LM when the dataset that we are domain adapting to is covered to some extent in the pre-training data of the model.
For dynamic context (paragraph Spoken-Squad) we see consistent improvement over the baselines from both the small and large model. Interestingly, the improvement we see in the dynamic context
case is less than we observe for the whole corpus case. This finding could be attributed to two effects: 1) The CA model is fine-tuned using the whole corpus and hence might perform better using that corpus. 2) The paragraph based retrieval corpus may be missing some tokens that would be present in the whole corpus. Overall, the CA model seem a better choice for dynamic context than the kNNLM model.
Both models demonstrate also a high tolerance for unrelated keys in the retrieval corpus. Adding unrelated data to the retrieval corpus has minimal impact or can even improve the results (see SpokenSquad results for the whole retrieval corpus compared to paragraph level retrieval corpus) on retrieval precision or recall of the next token. This property allows domain adaptation of the kNN-LM or the cross-attention model to multiple domains by combining multiple retrieval corpus.
In Table 5 we compare our results against the large whisper v2 model (Radford et al., 2023) for entity heavy SLUE and Spoken-Squad datasets. As can be seen, we reach parity on SLUE Voxpopuli dataset (used in training data of whisper), and improve the WER of whisper by ∼ 40% for Spoken-Squad, achieving state-of-the-art results.
Finally, in Table 6 we provide qualitative example of how retrieval using multi-modal memory helps with ASR task. The correct transcription in this case is “some species of beroe have a pair of strips of adhesive cells”. We inspect the retrieved values when the phrase “some species of” has been
generated by the model and it’s about to decode the audio frames relevant for transcribing the word “beroe”. As can be seen in the table, recall of the word “beroe” and its derivatives is high in retrieved values in case of the multi-modal memory, thereby helping the model to correctly transcribe the audio. We note that, without retrieval, the model transcribed the audio as "some species of burrow have a pair of strips of adhesive cells". We also list the retrieved values using text-only memory at the same time step for comparison.

heading: 4.3 Inference Latency
text: We explored inference latency for the small model and a retrieval database consisting of 1 million keys. In this setting, we observed mean retrieval time of around 3-4ms for finding the 8 closest neighbors to a key using a single CPU core. We used an inverted index based search from FAISS library with 2000 Voronoi cells.
On the other hand, decoding ten tokens using the small OPT model from a 12 s audio took 260ms on a single Nvidia T4 tensor core. The overall decoding latency is almost a magnitude larger than the retrieval latency which is between 30-40ms for ten tokens. The above latency numbers depend on the hardware used, implementation details, and FAISS index used and may be different for different setups.

heading: 5 Conclusion
text: We investigate use of retrieval augmented multimodal LMs for ingesting dynamic context and domain adaptation in speech recognition. We showed that a multi-modal LM can be effectively used for contextualizing retrieval database with audio, leading to an improvement of 10-50% (absolute) in retrieved token accuracy and recall compared to the text-only counterpart. Furthermore, we compared
masked-LM (BERT) with a similar sized multimodal LM as key encoder for constructing a text based retrieval database. Overall, the multi-modal LM fared better for next token retrieval by 6-10% (absolute), whereas the BERT model had better recall for subsequent tokens by 3-4% (absolute).
We considered two different approaches for domain adaptation: a kNN-LM and a cross-attention based neural approach. Both approaches are studied using two different model sizes: a small model with ∼ 300M parameters and a larger model with ∼ 7B parameters. In all cases we used a multimodal LM to encode the keys used for retrieval. Domain adaptation using cross-attention outperformed the kNN-LM for both small and large models by 30-100% relative. The large multi-modal LM with cross-attention outperformed whisper model by 40% relative for the entity heavy SpokenSquad QA dataset, achieving state-of-the-art results.
Additionally, the multi-modal LM’s parameters can be shared between the key encoder and decoder. Activation from the early layers can be computed once, used for retrieval and re-used with the retrieved context in the modified top layers for retrieval augmented generation. This reduces memory footprint of the model, which can be an important consideration for embedded applications.
Future directions of this work could be integration of more modalities (e.g video, sound etc.) into the retrieval process for improving model performance in various applications.

heading: 6 Limitations
text: We believe that the limitations of our work mainly stem from the limitations of Retrieval Augmented Generation (RAG), namely hallucinations and size of the retrieval database. With regards to hallucinations, a mismatch between the retrieval database
and the task (in this case ASR) can lead to incorrect transcriptions. Homonyms and rare propernouns are especially prone to this. Additionally, size of the retrieval database is also a concern from two standpoints: 1. Having a very large database (billions of tokens) can negatively affect retrieval statistics like recall. Modeling better key encoder is one way to alleviate this. 2. Very large databases have higher memory requirements and can be costly to maintain.
In this work, we showed that for speech recognition using paired audio and text for creating retrieval database is better than using just text modality. This places a dependency on having paired audio-text data for improved performance, which might not always be readily available or accessible. In such cases, using a Text-to-Speech system can be relied upon for generating paired data but it comes with its own pitfalls viz noisy speech, inaccurate pronunciation of rare words etc. These can lead to hallucinations in the RAG process.

heading: Future Work
text: Future directions of this work could be integration of more modalities (e. g video, sound etc. ) into the retrieval process for improving model performance in various applications