Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries
Authors: Ashish Mittal
Sunita Sarawagi
Preethi Jyothi
George Saon
Gakuto Kurata
Venue: EMNLP 2023 Main (2023)
Keywords: ASR Adaptation
Inference-time adaptation
Contextual Biasing
TL;DR: We present a novel inference algorithm that improves the prediction of state-of-the-art ASR models on rare words (available as a list) using nearest neighbor-based matching.

Abstract: Despite the impressive performance of ASR models on mainstream benchmarks, their performance on rare words is unsatisfactory. In enterprise settings, often a focused list of entities (such as locations, names, etc) are available which can be used to adapt the model to the terminology of specific domains. In this paper, we present a novel inference algorithm that improves the prediction of state-of-the-art ASR models using nearest-neighbor-based matching on an inference-time word list. We consider both the Transducer architecture that is useful in the streaming setting, and state-of-the-art encoder-decoder models such as Whisper.

In our approach, a list of rare entities is indexed in a memory by synthesizing speech for each entry, and then storing the internal acoustic and language model states obtained from the best possible alignment on the ASR model. The memory is organized as a trie which we harness to perform a stateful lookup during inference. A key property of our extension is that we prevent spurious matches by restricting to only word-level matches. In our experiments on publicly available datasets and private benchmarks, we show that our method is effective in significantly improving rare word recognition.

Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries
Ashish Mittal Sunita Sarawagi', Preethi Jyothi', George Saon', Gakuto Kurata'
IBM Research
IIT Bombay² ar arakeshkēin. ibm. com
Abstract
high impact on the ASR output's value. E.g., a spoken language understanding (SLU) module that uses ASR outputs might carry out the desired action as long as named entities in the spoken utterance are correctly transcribed.
Despite the impressive performance of ASR models mainstream benchmarks, their per- formance rare words is unsatisfactory. In
enterprise settings, often focused list of enti- ties (such locations, names, etc) available which can be used to adapt the model to the ter- minology specific domains. we present novel inference algorithm that im- proves the prediction of state-of saeee-of-theeat ASR models using neighbor- based matching inference-time word list. We consider both the Transducer architecture that is useful in the streaming setting, and state-of-the-art incoder-decoder models such as Whisper.
Existing contextual biasing techniques typi- cally demand additional parameterization, either via attention layers jointly trained with the ASR model (Le et al., 2021a) or via dictionary mod- ules trained with the ASR model frozen (Sun et al., 2021). Inference-time adaptation to dictionaries has primarily been via the shallow fusion of an ex- ternal LM (Kannan al., 2018). However, such a late fusion overly biases the model to the external LM and incurs high decoding latency especially for streaming ASR. In our work, we address this gap and propose a new lightweight inference-time tech- nique PRISM (short for "AdaPtation of ASR at Inference-time using Speech-enriched Memory") to adapt to predefined dictionaries at test-time with no additional training. PRISM is employed during beam-scarch decoding and used to adapt two state-of-the- ASR architectures, encoder- decoder models (Chan et al., 2016) and Transduc- ers (Graves, 2012) models.
this paper,
our approach, a list of rare entities is indexed in memory by synthesizing speech for each entry, and then storing the internal acoustic and language model states obtained from the best possible alignment
the ASR model. The memory is organized as a trie which we har- ness perform a stateful lookup during infer- ence. A key property of our extension is that we prevent spurious matches by restricting to only word-level matches. In our experiments on pub- licly available datasets and private benchmarks, we show that our method is effective in signifi- cantly improving rare word recognition.
First, we use a text-to-speech (TTS) system synthesize speech for each dictionary entry. To the best our knowledge, we are the first to use synthetic speech for contextual biasing and find that it facilitates improved matches compared to the use of text alone. Utilizing both speech and text in each dictionary entry, we create memory mapping speech and text representations (keys) textual tokens (values). For encoder-decoder mod- els, the decoder states are natural choice for the memory keys. However, for Transducer models, implementing memory keys requires more delib- eration and careful handling of the alignment of acoustic and textual context (elaborated in $3.1).
1 Introduction
Rapid advancements in Automatic Speech Recog- nition (ASR) technologies have broadened their ap- plication in real-world settings. However, state-of- the-art ASR systems often fail to recognize named entities or critical rare words. For example, a chat- bot we deployed in the medical domain backed by the state-of-art Whisper ASR model, consis- tently misrecognized In patient to impatient. Such situations call for contextual biasing whereby the transcription from ASR model is biased by user- provided list of proper nouns (e.g., names from contact lists) technical terms (e.g., drug names from medical dictionaries). The accurate transcrip- tion of such tokens could have a disproportionately
During inference, we perform a k-ncarest- neighbor (KNN) search over the memory to retrieve

---

the & most similar dictionary entries. We define KNN probability distribution over the next token that assigns higher probabilities to nearby mem- ory entries and finally bias the model predictions via a linear interpolation with the kNN probabil- ity distribution. The memory-based matching is implemented using an efficient trie-based search. Notably different from prior work (Le et al., 2021a), we avoid triggering spurious matches in the mem- ory after only prefix match, by maintaining a backoff beam score that keeps track of the score assigned by the model to a suffix if it were not found in the trie. We show that such accounting is essential to avoid worsening baseline performance in the presence of a dictionary.
is usually more accurate than shallow fusion, but comes with the added overhead of retraining the ASR model for every new dictionary to be inte- grated.
Inference-time adaptation using dictionary. The last category of approaches make use of a dictio- nary during decoding (Williams et al., 2018; Huang et al., 2020; Sun et al., 2021). PRISM falls in the last category with two key differences from prior work. We also synthesize speech, using TTS, for every dictionary entry, unlike prior work that only uses text from biasing lists/dictionaries. This facili- tates improved matches to the dictionary terms at test time. We enforce word-level matches using a trie and additional beam state to avoid spurious pre- fix matches that could otherwise seriously distort baseline performance.
PRISM can be used with any existing pretrained encoder-decoder or Transducer ASR model without any retraining and incurs close no delays during inference. Using PRISM, we obtain significant performance improvements with both Transducer and
2.1 Background: ASR Models
Two prominent architectures for ASR are Trans- ducers (Graves, 2012; Graves al., 2013) popular when low latency and streaming (He et al.) are im- portant, and encoder-decoder used in state-of-the art ASR models such as Whisper (Radford et al., 2022). We present a brief overview cach.
ASR over diverse dictionaries, with up to 36% relative reductions in entity-specific eπror rates on the well-known Librispeech ASR benchmark.
2 Related Work and Background
Transducers. Transducers have three modules: (1) An audio encoder Mss (2) A text encoder ML, and (3) A joint network M,. Audio encoder Ms converts input specch I1, VT to the latent rep- resentations hı,... , hy using an RNN (Cho al., 2014) or Transformers (Vaswani al., 2017) or Conformer (Gulati et al., 2020). The text encoder My converts each text prefix yd = ;0-11 to the contextual latent representation Iu- The joint network M,, combines the latent represen- tations h4 and gu. followed by softmax layer that outputs the distribution Po h2) over vocabu- lary V plus the blank symbol Ø. Since explicit alignments between æ; and yı is provided, log- likelihood of the target sequence y is computed during training by marginalizing over all possi- ble alignments uf) using an efficient forward- backward algorithm (Graves et al., 2013). During inference, beam-search is used to find the best pos- sible (t, u) alignment (Graves, 2012; Saon et al., 2020).
Prior work on contextual biasing of ASRs can be organized into three categories:
Dictionary as external language model. In the first category, the dictionary is viewed as an exter- nal LM. This is integrated with the ASR's 's output either via shallow fusion where ASR scores are combined with scores from an external LM during beam-search (Ravi et al., 2020; Liu et al.. 2020; Le et al., 2021 la; Gourav et al., 2021) or deep fu- sion where the model has access to an external LM during training (Le et al., 2021b,a). While shallow fusion is a popular scheme that allows for contextual information be combined with any trained end-to-end ASR, it is also very sensitive to the weight assigned to the external LM and often helps ASR of contextual entities at the expense of overall ASR performance (Mittal et al., 2023).
Training-tine adaptation. The second category consists of approaches that modify the ASR model during training to incorporate contextual infor- mation, often relying on attention-based mecha- nisms (Jain et al., 2020; Chang et al., 2021; Huber et al., 2021; Sathyendra et al., 2022; Sun et al., 2023a; Munkhdalai et al., 2023; Chan et al., 2023). Such a direct integration of contextual information
Encoder-decoder. These models encode the audio in the same way as above but the decoder converts the text prefix Yu and audio using cross-attention on the audio states at each decoding step. Thus, the alignment of text and audio is implicit in the cross

---

A PRNNN Mau.llulli..... (i.... (1 X) P( as Bi....
Libertyvlle
My
Rentrew
Jasper
Alggimetts
TTS
RNN-T
Minnetonka
(h,g)
h
Londondarry
9 L
Auburn
Ms
Mr
Text Dictionary
Audio Dictionary
Memory
Zt
Yu-1
(a) Memory Creation from the text dictionary
(b) Inference with memory
Figure I: Overview of our approach for a Transducer ASR model.
attention. At decoding step u, cross attention from decoder to the encoder states hı...., hr pro- vides fixed length vector which is fused with self-attention layers on the decoder to get up- dated decoder state dy. The decoder state followed by softmax layer yields Po Poyy,,, h.. Inference happens via beam search.
by resorting to TTS to generate synthetic audio for each phrase. A second challenge is that for Trans- ducer models, the design and extraction of context to form the memory keys is significantly harder, since they do not follow tecture. Finally, flat memory in the unit of tokens causes several spurious matches with pieces of whole phrase during decoding. For example, dic- tionary entry for "Intuit" could spurious matches to the "In" or "it" tokens of this word. We therefore restrict to allow only whole phrase matches. To im- plement such whole phrase matches over multiple steps in existing token-level decoding algorithms. organize the memory trie and maintain state in the beam the prefix of the trie matched so far. Additionally, we maintain a backoff beam score that keeps track of the model score for retraction unless the match completes a full word. We first present how we create the memory and then discuss our method of interpolating default model token distribution Po with lookups from the memory.
decooer archi-
3 Contextual biasing in PRISM
We are given trained ASR model M and list of isolated word phrases Î {ğ] ,..'.....""" g²) infer- ence time. PRISM uses to first create a memory structure N. Subsequently, for each subsequent input speech æ, we perform memory augmented inference to bias its text to the words in D. As stated earlier, the challenge is that recognition of words not in D should not worsen in the process. In order to not adversely impact the streaming and la- tency performance of M, PRISM integrates with existing ASR decoding algorithms the token- level, by just modifying the token distribution at
each step of decoding.
3.1 Memory Creation
An efficient mechanism proposed in the NLP literature for sequence sequence models is k- nearest neighbor (KNN) language models (Khan- delwal al., 2019; Guu al., 2020; Yogatama al., 2021). ). A key insight of KNN LMs is that similar decoder contexts lead to similar next token distribution. This motivates the creation of memory in the form of a list of key-value pairs with context as key and next-token as value.
Since the provided list D is in text, we use TTS engine create audio for the correspond- ing phrases deseribed in Figure (a)) With this, we obtain target domain datasct Dr: {..., (0i, .'..... which is devoid of any context in which in these utterances are being used. We convert Dy into a trie-based memory of key-value pairs at the teken-level as follows.
However, applying this principle to ASR models raises several challenges. First, we are given as in- put only list of isolated word phrases, and context is missing both in terms the missing audio input, and an overall sentence context which the phrase lies. We partly offset for the lack of audio input
Memory keys for Transducers For Transduc- ers, the internal representations index are audio encoder Mg state and the text encoder Mr state Iu for each non-blank prediction Yu as they describe both the acoustic state and contextual lan- guage model state for the predicted token. For

---

nearest neighbors (K a hyperparameter) from the children of N as a set (N q:) {(k, ey)) ] de- noting key and token id of the matched neighbors. The distance function between Qi and k, model dependent. We use thresholds on the distances and it is possible for the neighbor set to be empty.
hs
hs
4:
Alignmet
(12g go) 31
(hs, gi) y5
9:
(hg.9t) Qz) >5
When the neighbor set NN(N, 9:) is non-empty define distribution over the next token from the memory lookup as:
Figure 2: For transducer models the best alignments from the decoding lattice are used to generate the mem- ory.
Pknn (3) qs N) x
Σ
exp(-dist(q;: (y)st)
(NTS))ENN(NN.)
each utterance ø' in Dr, we trace the lattice of the highest probability beam to find the best valid alignment A' that consists of a sequence of (h,, 9u- Du) corresponding to non- blanklan predictions as de- scribed in Figure 2. We augment the beam search algorithm to constrain the output vocabulary to the target sequence ÿ' and Ø. For instance, if the target sequence is "Minnetonka", and the predicted se- quence the beam "Minne", then for the subse- quent expansion of the beam, the output vocabulary is constrained to either 't' or Ø. To further nudge the model to predict the character 't', we incremen- tally add extra probability to the constrained token over Ø. We find that this algorithm provides better alignments opposed to finding the best path in the output lattice using a Viterbi algorithm.
(1)
The above distribution is interpolated with the to- ken distribution from the model as
P(yl-) (1 Poll) A:Pam(y)))) if N Pal(l))
otherwise
The KNN coefficient X3 can be considered as a hyper- prppe-parameter, but we find that deriving the value of the KNN Coefficient dynamically using a simple linear function on the minimum distance yields improved performance.
A 1
min (dist(gs, k,))
(3)
(ky JCNN
This function has the effect of increasing KNN weight when there is good match in the dictionary. Unlike in previous work, one crucial step we take to avoid spurious matches with the memory, is to only allow the memory scores accumulate for full word matches. Without it, we observe that when the actual word deviates from the dictionary word after a prefix match, the hypothesis still car- ries the extra probability and can lead to spurious recognition (e.g, twin becomes twen, if the dictio- nary has the word twente shown in Figure 3). In order to avoid such spurious matches and enforce word-level match only, we maintain the beam in addition the beam score s, default score sy that tracks the model score of the suffix if the current suffix zu of 3/, were not to match the trie. If the next token of y does not match a child of N, we reset the beam score s to 8g- If we reach the end of the trie at a decoding step, the complete word has matched and we update to be the beam score We will show in the ablation that such step essential to avoid penalizing recognition accuracy of words that do not match any full word in the dictionary. The overall algorithm appears Algo- rithm assuming a generic beam-search algorithm for encoder-decoder model. For the Transducer
Memory keys for Encoder-decoder Models We use teacher forcing constrain the predicted output the ground truth. For each inference step the memory is the output d after the last cross-attention block in the decoder. The value for the memory consists of the next token as per the ground truth.
3.2 Memory Augmented Decoding
We present the framework for memory augmented inference that applies both for Transducers and encoder-decoder models. Both models use some form of beam search during inference, and we mod- ify each decoding step of beam-search as follows: At each decoding step i, we have a current hypothesis Ye = V1:.... Mu- a decoder state d, a beam score 8, and a next token distribution Po h)) from the model. From the memory, that is organized form of the trie node N that has matched suf- fix of Yas call it 2n Va-e... yu where is the depth of N. The suffix could be empty, in which case is root of the trie. We define a query qi from the decoder state d and use that to find the K-
tric, maintain state in the

---

T-
WH-E-N-T+E
TW-E-N-THE
T--
E-NTTE
X
E-M-P+A
E-M-P
E-M-PAA
M
E-
model, while the overall framework is the same, the details are slightly different. We provide a de- tailed algorithm for Transducers in Appendix B. .W (0.81)
.WEE (0.77)
TWEE (0.63)
TWEE
(0.62)
TWIN.
(0.58)
Rank TWI (0.73)
Rank
TW
Rank 2
IN (0.59)
Rank
TWEN
No Rascoring
Rescoring
Figure 3: Step-by-step decoding using PRISM for the reference word twin. For this example, the numbers on the right of each step are the probability of that hypothesis. For each correct match, the probability is assumed to be 0.9. For match step 2, the probability before and after the interpolation are 0.95 and 0.6 respectively, With rescoring, the interpolation probabilities replaced by the model probability, which downgrades the hypothesis.
decoder incorporate PRISM. We used beam- width of 10 in all the Whisper experiments. The Transducer model (56M parameters) is trained on proprietary dataset of 56K hours of labeled speech. This model takes 140-dimensional log-Mel filter- bank audio features as its input and predicts prob- ability distributions over an output vocabulary of 45 English characters and blank character. We elaborate in Appendix A.
Algorithm PRISM Inference
1: Input: Encoder state h, beam width W max. decode
length I, Trie memory root N
2: Output: Most likely hypothesis in beam
3: Beam {(v o,d d initial state, N, 1, 80 1))
4: for I....
Memory. We use the FAISS library (Johnson et al., 2019) to index and efficiently search through embeddings. Recall that we index embeddings from both the audio and text encoder for the Trans- ducer model. We use a state-of-the-ar VITS (Kim et al., 2021) system from Coqui TTS (coqui.ai, 2023) to synthesize audio for dictionary entries. We set the thresholds for the audio match to 0.4 and text match to 0.05. Note that the same threshold is used across all datasets spanning 9 different dictio- nary lists. The text-based threshold strict, thereby only allowing for matches that the language model readily permits. However, the audio-based thresh- old more relaxed allowing for more variability in the audio matches to accommodate varying pronun- ciations, noise in the audio, etc. With the Whisper models, use an unnormalized embedding matching with distance threshold of 250 and 1000 for the base and small models respectively. (Nor- malized distances did perform as well.) We further apply a min-max normalization these un- normalized distances to use as a dynamic KNN coefficient for the interpolation (outlined in Sec- tion 3.2). For all the experiments, we use the four nearest neighbors while searching the index.
5:
A {}
for (y. 80) Ream do
Pa(y))), M(h,
8:
knn, A= Near nbrs(d, threhhoolds)
A-n KNN-probs(Knn, d) (Equ: 1)
10:
for E do
11:
soly + k) (k(k)
12:
s(y k) = (AP) +1 (k))
13:
k N.next then
14:
s(y + k) = sw + (forget scores match)
incomplete
15:
if N.end then
16:
(0 + k) <(y k (reset)
17:
A = A U {1y + k. N.next(B), s(y + k). (y ( + B})}
18:
-nr-PrunAAndReoombieefeee WF
19:
return Sorted(Beam) =0
4 Experiments
We present an extensive comparison of PRISM with existing methods on both state-of-the-art Transducer (Graves et al., 2013) and encoder- decoder (Chan et al., 2016) ASR models spanning three different dataset settings.
Models and Implementation Details. For the encoder-decodeer models, we use two publicly- available Whisper models (Radford et al., 2022): Whisper-base(74M parameters) and Whisper-small (244M parameters). Whisper was trained on over 600K hours of weakly labeled training data mined from the web. We do not do any finetuning of the Whisper model, and only alter the beam-search
Evaluation and Comparisons. Apart from com- puting overall word error rate (WER), also compute entity WER (E-WER) over the spans covered by dictionary entities in the test utterances. We compare against three existing techniques.

---

test-clean
test-other
test-clean
test-other
TCPGen (Sun et al.. 2023b)
TCP Gen (Sun et al., 2021)
Whisper Baseline
5.2/ 16.3
10.5 1/5/1144
RNN-T
5.5 / 18.7
15.3 42.9
Whisper TCPGen
4.8 12.5
10.1 / 26.5
RNN-T+TCPGGR
4.9 13.9
14.0/ 35.0
Relative Reduction
23.3
3.8 35.6
Relative Reduction
10.9
8.5 18.4
PRISM (Ours)
Deep Contextual RNN-T
(Le et al., 2021a)
Whisper Baseline
1.91
10.2 30.9
RNN-T
14.1
9.6 30.6
Whisper+PRRSMM
13.0
9.6 25.7
RNN-T+Deep Cont.
3.0/ 8.5
8.5 / 20.5
Relative Reduction
10.2 19.3
5.9 / 16.8
Relative Reduction
17.8/
11.5 / 33.0
PRISM (Ours)
Table Comparison of WER/E-WER on Whisper with TCPGen on base.en model. Whisper + TCPGen num- bers are reproduced from (Sun et al., 2023b), along with their Whisper baseline numbers. PRISM outperforms TCPGen on both the test sets from a superior baseline.
RNN-T
3.5 13.3
9.9/
RNN-T+PRISM Relative Reduction
3.1/ 9.3
9.0 20.6
1 11.4/30..
9.1/36.0
Table 2:
Comparison of WER/E- WER with TCP- Gen and Deep Contextual RNN-T for RNN-T models. PRISM obtains a 30% reduction in E-WER without needing an external language model used in Deep Con- textual RNN-T and outperforms TCPGen all metrics.
1. TCPGen (Sun et al., 2021) learns distribution over the output vocabulary that is limited by prefix tree constructed from the dictionaries. TCPGen has parameterized module, jointly trained with the frozen ASR model, estimate weights signifying how much contextual bias required to decode the current token. TCPGen has been used both with the Transducer model and Whisper (Sun et al., 2021, 2023b).
lends itself nicely to detailed ablations by en- abling fine-grained control on dictionary size, types of entities, etc. We will publicly release this benchmark to facilitate further research into dictionary- based ASR udaptation.
2. Deep Contextual RNN-T (Le et al., 202 con- sists of learnable trie-based biasing modules and is specific the Transducer model.
4.1 Evaluation on Librispeech
We evaluate on the standard test-clean and test- other splits of Librispecch (Panayotov et al., 2015). For dictionaries, borrow the experimental proto- col used for TCPGen Sun et al. (2021). Biasing lists comprise 200K rare words from the Librispeech training set. Dynamic dictionaries are constructed for each test utterance. Rare words in the test tran- script are augmented with randomly sampled rare words to create 1000-word dictionaries per test utterance. We reuse the same dictionaries used by Sun al. (2021) for fair comparison. Addi- tionally for PRISM, we create supporting audio for each dictionary entry using TTS.
Unlike PRISM, we note that both TCPGen and Deep Contextual RNN-T techniques require a joint training of biasing modules with the ASR models.
Overview of Experiments. We show ASR adap- tation experiments across multiple scenarios involy- ing three types of datasets:
1. Librispeech (publiclyavvailbble Dataset: We evaluate on the popular Librispeech ASR bench- mark (Panayotov et al., 2015) using dictionaries from prior work (Sun et al., 2021; Le et al., 2021a). These experiments are detailed in $4.1. 2. Enterprise Dataset: We show experiments on
Table shows how PRISM improves over Whis- per and how compares to TCPGen. Whis- per+TCPGen WERs are reproduced as it appears in Sun et al. (2023b), along with their reported Whisper baseline numbers. We observe that PRISM yields superior WER and E-WER reduc- tions on both test-other and test-clean. We achieve significant reductions compared to TCPGen, de- spite starting with a much stronger Whisper base- line; in fact, our baseline numbers are almost as good as the reported Whisper numbers.
a real consumer-facing chatbot application in the medical domain (in $4.2). We evaluate on real speech from users covering five different intents and dictionaries of varying sizes.
3. Entity-rich Dataset: Due the absence of aublicly-available entity-rich datasets, we cre- ate a new benchmark with sentences that are dense in entities such as locations, person names and medicine names (dctailed in $4.3). This is a synthetic dataset created using TTS that

---

Category
1
Category
Category 3
Category - 4
Category 55
Average
RNN-T Baseline
32.9/ 26.3
38.8 / 32.6
48.6 26.6
125.0/ 83.3
33.0/ 40.0
55.7 / 41.8
Whisper (small)
32.9/214
77.6 / 28.6
45.2 24.5
116.7 / 58.3
53,2 / 44.1
65.2 / 35.3
Peng et al. (2023)
19.2 13.1
63.3 / 26.5
57.1 / 40.0
58.3 / 33.3
58.7 30.9
51.3 / 28.8
PRISM
26.0 15.8
26.5 / 26.5
40.0 20.0
16.7 / 8.3
28.2 / 29.4
27.5 / 20.0
Table Comparison of WER/E- WER on the internal medical domain dataset using an RNN-T and Whisper(small) baseline. PRISM outperforms all the baselines most categories for this noisy short-form internal dataset halving the WER and E-WER.
Table 2 presents results on the Transducer ASR comparing PRISM with both TCPGen and Deep Contextual RNN-T baselines. PRISM outperforms both baselines on test-other in terms of relative WER and E- WER reductions and performs com- parably to Deep Contextual RNN-T (and better than TCPGen) on test-clean. For both Whisper and Transducer models, it is noteworthy that PRISM achieves comparable WER reductions with min- imal overhead, compared to TCPGen and Deep Contextual RNN-T that both require retraining.
et al. (2023) that is applicable only to the Whisper model. Whisper supports the of prompts with the decoder state to bias the predictions towards the dictionary entries during decoding.' Note that this method applicable in settings where the num- ber of contextual entities is small, as the decoder state is prompted on these entities. Hence, we com- pare this technique with PRISM for the enterprise dataset that contains small number of keywords in each category. We observe that PRISM signifi- cantly outperforms the technique from Peng et al. (2023) on our enterprise benchmark. The average WER/E- WER from Peng et al. (2023) 51.3/28. compared to 27.5/20.0 obtained using PRISM.
4.2 Evaluation on Enterprise Dataset
We evaluate PRISM using data from deployed chatbot in the medical domain, interacting with real users. We test utterances drawn from five intents/categories with keywords specific to each category. Category-1 contains utterances with domain-specific keywords (e.g., representative, au- thorization), Category-2 has 4 keywords (e.g., ph.rp macy), Category-3 has 4 keywords (e.g., acupunc- ture), Category-4 has 2 keywords (inpatient, outpa- tient) and Category-5 has 30 keywords (e.g., mam- mogram, chemotherapy).
4.3 Evaluation on Entity-rich Dataset
We create new entity-rich, synthetic benchmark spanning different categories of commonly occur- ing entities viz. locations, person names and med- ical drugs. We use VITS TTS from the Coqui toolkit (coqui.ai, 2023) to create both entities and sentences for this benchmark. We use different speakers for the dictionary entities and the test ut- terances to mimic real settings where these speak- ers are disjoint. This benchmark is released to encourage fair comparisons of dictionary based adaptation techniques. Prior work on contextual biasing either uses proprietary datasets or public lists (like from Librispeech), but the latter is nei- ther domain-specinn nor dense in entities. Table 4 provides a brief description of our dataset, along with entity types and sample sentences containing these entities. Table 5 shows results from both Transducer and Whisper models, with and without PRISM. We note the high baseline E- E-WER, espe- cially for Drugs and Location (big). indicating that the entities are difficult for the baseline models
Table 3 shows evaluation results for all five categories, using both Transducer and Whisper baseline models, along with PRISM applied to the Transducer model. For test utterance in Category-4 with the keywords inpatient and ouipa- tient, the baseline Transducer model predicts impa- tient which is more common word in the training set, while Whisper predicts patience. PRISM fixes such errors and dramatically reduces the base- line Transducer WER from 83.3 to 8.3 over 20 test utterances. Other categories also significantly ben- efit from using PRISM. The Whisper baseline is prone to hallucinate with irrelevant content, espe- cially when the input audio is short; Appendix C shows some examples.
Moore information is available at https: //gi thubthub.co http://////pee////.coemeo Discussiioncomment3722000 hhtns://gghuua.oomm///ssiiitttta
Additionally, in Table 3, we compare PRISM with a very recently proposed technique from Peng

---

Entity
#Entities
#Sentences
Sample Entities
Sample Sentences
Location
1000
567
Chicopee, Erlanger
live in Chicopee.
(small)
Minnetonka
Erlanger and Minnetonka are gorgeous.
Location
2818
1628
Albemarle, Saratoga
The route passes through Albemarle and Saratoga. Merthyr Tydfil is gorgeous.
(big)
Merthyr Tydfil
Person
4348
4220
Beaufort, Zebadiah
Zebadiah always makes to include others. Eberhard loves to skydive.
Names
Eberhard
Drugs
2874
2874
Aclovate, Primidone
Is Primidone safe to use during pregnancy?
Fluothane
What is dosage for Fluothane?
Table 4: Entity-rich dataset.
Location
Location (big)
Person
Drugs
Average
(small)
Names
Baseline
39.1 70.0
3
24.7 / 19.0
36.8 / 93.5
34.6 62.3
RNN-T
PRISM
27.1 /41.0
29.0/ 45.1
24.3/17.6
33.5 77.0
28.5 45.1
Baseline
24.4/455.1
23.5
25.1
12.8 15.3
23.3 / 82.2
21.0 46.9
Whisper (small)
PRISM
18.8 31.8
18.2
32.2
12.1 14.1
17.6 53.9
16.7 33.0
Table 5: Comparison of WER/E-WEL on different entities from Entity-rich dataset. PRISM consistently out- performs the baseline RNN-T and Whisper models yielding upto 20% and 30% reductions in WER and E-WER respectively.
Location (Big)
Drugs
PRISM
29.0 45.1
33.5 77.0
Dynamic Coefficient
36.4 / 63.5
35.3 85,4
Trie
53.9
36.5 80.3
Rescore at false Trie exit
32.9
36,8 79.3
WER
Audio-only Index
/43,
39.2 73.3
Text-only Index
/9..8
139.1 96.8
Table 6: WER/E-WER comparison after removing core components from PRISM for RNN-T. Using audio- only index for RNN-T yields better E-WER, overall WER degrades because of the unconstrained hallucinations due to lack of language model context.
1000
2000
3000
#Distractors
Figure 4: Comparison of WER vs. #distractors in the dictionary for Location (Big).
accurately transcribe. PRISM yields significant WER/E-WER reductions for entity types.
in the trie significantly degrades performance. This feature is absent from all prior trie-based biasing techniques (Sun et al., 2021; Le et al., 202 a; Sun et al., 2023b). Table 6 also shows the importance of indexing both audio and text embeddings. Overall WERs degrade with using an audio-only index (due to the lack of language modeling context), while both overall WERs and E--WERs drastically worsen with using text-only index.
4.4 Ablations and Analysis
We use the synthetic benchmark to present an abla- tion of various design choices that are critical to the success of PRISM. Table shows how PRISM fares on Location (Big) and Drug entity types if: I) we omit the dynamic KNN coefficient and use static coefficient instead, 2)
flat mem- ory rather than trie-based memory and, 3) remove the rescore feature in the trie (described in Section 3.2) that encourages full-word matches. We observe that each of these design choices were important. Note that disabling the rescore feature
Figure 4 shows how PRISM behaves with vary- ing dictionary sizes for Location (Big). For each test utterance, we identify the gold entities from the reference text and add a fixed number of randomly- sampled distractors to reach the desired dictionary

---

Datasct
No Entitics
Average Tokens
Memory
dictionaries of varying sizes for Location (Big).
Location (small)
1000
MB
Figure shows that PRISM scales well and incurs
Location (large)
2818
10.7
138 MB
Person Names
4348
7.0
126 MB
minor overheads with larger dictionaries.
Drugs
2874
11.3
132 MB
Conclusion
Table 7: Memory requirement of various dictionaries.
We present PRISM, an efficient inference-time dictionary-based ASR adaptation technique that synthesized speech for dictionary entries facilitate improved matches. PRISM includes an efficient trie-based memory lookup algorithm, with effective fallback scoring mechanisms to pre- vent spurious matches. PRISM can augment stan- dard beam-search decoders in both Transducer and Whisper cncoder-decoder ASR models. We demon- strate the effectiveness of PRISM with experi- ments across diverse dictionaries containing rare words and named entities. We obtain significant relative WER reductions of up 36% on entities, yielding an overall WER reduction close to 10%. on the challenging test-other split Librispeech benchmark. We also achieve superior performance on evaluation sets drawn from an enterprise setting, with overall and entity error rates almost reduced by factor of 2 (i.e., relative reductions of 40% to 55% over the baseline ASR systems).
size. We see that PRISM is steady in its perfor- mance and does not degrade the presence of large number of distractors.
4.5
Memory and Compute Overhead of PRISM
We compute the memory occupied by our dictio- nary for various entities in the entity-rich dataset in Table 7. It is worth noting that the sizes of the dictionaries are influenced by the number of tokens (characters BPE units) present in the words or phrases within the dictionary entries. While the Drugs entity dictionary contains fewer entries, the average token count per entry tends to be larger compared to the Person Names and Location entity dictionaries.
1.00
0.75
RTF
0.50
Limitations
0.25
We highlight two limitations of our current work: PRISM relies on TTS systems to create au- dío representations for dictionary keywords. For many rare words, TTS might not provide the correct pronunciation which can lead mismatches with the actual words during in- ference. ASR settings involving accented speech and low-resource languages, this dif- ference can be further exacerbated due to the lack of robust TTS engines.
0.00
1000
2000
3000
Figure 5: Comparison of RTF #distractors in the dictionary. Baseline RTF is 0.5
Next, we study the overhead of using PRISM during inference. Analytically, the complexity of the beam search algorithm for RNN-T models is OcBUUT), where is the cost of the forward pass, B the beam size, T' is the number of audio frames in the input and U is the maximum length of the output. PRISM method adds a lookup cost to mem- ory in each step of the beam expansion. the cost of lookup is p. then the overall complexity would become O((c + BUT). We use the state-of- the-art nearest neighbor index from Johnson et al. (2019) which can do fast look-up minimize the latency.
Memory and runtime requirements for PRISM dependent on the size of the mem- ory. This can be further optimized by quantiz- ing the representations.
References
David M Chan, Shalini Ghosh, Ariya Rastrow, and Bjöm Hoffmeister. 2023. Domain adaptation with external off off-co acoustic catalogs for scalable con- textual end-to-end automated speech recognition. In ICASSP 2023-2023 IEEE International Confer- ence on Acoustics. Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
To highlight the minimal computational over- head incurred by PRISM, we compute RTF (real- time factor) scores i.c., the time takes to decode second audio, using our Transducer model with

---

William Chan, Navdeep Jaitly, Quoc Le. and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recogmition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4960-4964. IEEE.
matic Speech Recognition and Understanding Work- shop (ASRU), pages 1-7. LEEE.
Mahaveer Jain, Gil Keren, Jay Mahadcokar, Geoffrey Zweig, Florian Metze, and Yatharth Saraf. 2020. Contextual mnnt for open domain ast. arXiv preprint arXiv2206.03411.
Feng-Ju Chang, Jing Liu, Martin Radfar, Athanasios Mouchtaris, Maurizio Omologo, Ariya Rastrow, and Siegfried Kunzmann. 2021. Context-aware trans- former transducer for speech recognition. In 2021 IEEE Automatic Speech Recognition and Understand- ing Workshop (ASRU), pages 503-510. IEBE.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. 7(3):335-547,
Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N. Sainath, ZhiJeng Chen, and Rohit Prabhavalkar. 2018. An analysis of incorporating an external language model into sequence-te to sequence model. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5828.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv: 1409.125999
coqui.ai. 2023. Coqui us.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 1. 1,00172.
Aditya Gourav, Linda Liu, Ankur Gandhe, Yile Gu, Guitang Lan, Xiangyang Huang, Shashank Kalmane, Gautam Tiwari, Denis Filimonov, Ariya Rastrow, et al. 2021. Personalization strategies for end-to-end speech recognition systems. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7348 7352. IEEE.
Jachyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversar- ial learning for end-to-end text-ta-speech. In Inter- national Conference on Machine Learning, pages 5530-5540. PMLR.
Alex Graves, 2012. Sequence Transduction with Recur- rent Neural Networks.
Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi. Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, et al. 2021a. Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion. arXiv preprint arXiv:2 104.0194.
Alex Graves, Abdelrahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recur- rent neural networks. In 2013 IEFF international conference on acoustics, speech and signal process- ing, pages 6645 6645-6649. leee.
Due Le, Gil Keren, Julian Chan, Jay Mahadeokar, Chris- tian Fuegen, and Michael L Seltzer. 2021b. Deep shallow fusion for mn-t personalization. In 2021 IEFE Spoken Language Technology Workshop (SLTT) pages 251-257. IEEE
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu. Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, al. 2020. Conformer: Convolution-augnmentee transformer for speech recognition. Proc. Interspeech 2020, pages 5036-5040.
Da-Rong Liu, Chunxi Liu, Frank Zhang, Gabriel Syn- naeve, Yatharth Saraf, and Geoffrey Zweig. 2020. Contextualizing asr lattice rescoring with hybrid pointer network language model. arXiv preprint arXiv:2005.07394.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 9929-3938 PMLR.
Nlya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05100
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recognition for mobile devices. In ICASSP 2019.
John S Mason and X Zhang. 1991. Velocity and accel- eration features in speaker recognition. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, pages 3673-3674, IEEE Computer Society.
Rongqing Huang, Ossama Abdel-Hamid, Xinwei Li, and Gunnar Evermann. 2020. Class Im and word mapping for contextual biasing in end-to-end asr. arXiv preprint arXiv:2007.05609.
Ashish Mittal, Sunita Sarawagi, and Preethi Jyothi. 2023. In-situ text-only adaptation of speech mod- els with low-overhead speech imputations. In The Eleventh International Conference on Learning Rep- resentations.
Christian Huber, Juan Hussain, Sebastian Stüker, and Alexander Waibel. 2021. Instant one-shot word- learning for context-specific neural sequence-to- to- sequence speech recognition. In 2021 IEEE Auto-

---

Tsendsuren Munkhdalai, Zelin Wu, Golan Pundak, Khe Chai Sim, Jiayang Li, Pat Rondon, and Tara N Sainath. 2023. Nam+: Towards scalable end-to-end contextual biasing for adaptive ast. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 190-196. IEEE.
Guangzhi Sun, Xianrui Zheng. Chao Zhang, and Philip Woodland. 2023b. Can contextual bias- ing remain effective with whisper and gpt-2? arXiv preprint wrXiv230660194.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015, Librispeech: an asr cor- based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE.
lan Williams, Anjuli Kamnan, Petar S Aleksic, David Ry- bach, and Tara N Sainath. 2018. Contextual speech recognition in end-to-end neural network systems us- ing beam search. In Interspeech, pages 2227-2231.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc V Lc. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: TTT1:/9004007799
Dami Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. 2021. Adaptive semiparametric lan- guage models. Transactions of the Association for Computational Linguistics, 9:362-373.
Puyuan Peng, Brian Yan, Shinji Watanabe, and David Harwath. 2023. Prompting the hidden talent of web- scale speech models for zero-shot task generalization. arXiv preprint arXiv:2305.1100
Joseph W Picone. 1993. Signal modeling techniques speech recognition. Proceedings of the IEEE, 121511225
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. arXiv preprint arXiv:2212.043356
Vijay Ravi, Yile Gu Ankur Gandhe, Ariya Rastrow, Linda Liu. Denis Filimonov, Scott Novotney, and Ivan Bulyko. 2020. Improving accuracy of rare words for mnn transduccc through unigram shallow fusion. arXiv preprint X31i::2012.00133.
George Saon, Zoltán Tüske, and Kartik Audhkhasi. 2020. Aligament-length synchronous decoding for mn transducer. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acousties, Speech and Signal Processing (ICASSP), pages 7804-7808. IEEE.
Kanthashree Mysore Sathyendra, Thejaswi Muniyappa, Feng-Ju Chang, Jing Liu, Jinru Su, Grant Strimel, Athanasios Mouchtaris, and Siegfried Kunzmann. 2022. Contextual adapters for personalized speech recognition in neural transducers. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 8537- 8541. IEEE.
Guangzhi Sun, Chao Zhang, and Phil Woodland. 2023a. Graph neural networks for contextual asr with tree-constrained pointer generator. arXiv preprint arXiv:2305.18824
Guangzhi Sun, Chao Zhang, and Philip Woodland. 2021. Tree-constrained pointer generator for end-to- end contextual speech recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 780-787. IEEE.

---

A Implementation details
The model consists of 6 bidirectional LSTM layers of hidden dimensionality 640 in the acoustic en- coder and single unidirectional -i8-dimensiooal LSTM layer in the text encoder. Both acoustic and text encoders are further projected down to 256 dimensions using separate projection layers, and a joint network combines these 256-dimensional embeddings via a simple element-wise product, followed by a tanh non-linearity. A final softmax projection layer is used to produce probability dis- tributions over an output vocabulary consisting of 45 non-blank English characters and blank char- acter. This model uses log-Mel filterbank audio features augmented with delta and double-delta spectral features (Mason and Zhang, 1991; Picone, 1993), resulting in 240-dimensional input features spanning 20ms audio segments. As is standard practice, audio augmentations such speed and tempo perturbations (Park al., 2019) are applied well. The Transducer models are trained for 20 epochs using AdamW (Loshchilov and Hutter, 2017) with a maximum lcarning rate of Sc-4.
Algorithm Pseudo-code for Memory Augmented
ALSD for RNN-T
1: Input: Encoder state h, beam width W, max. decode
length 1, Trie memory root
2: Output: Most likely hypothesis in beam
3: Beam = [(y = 0. initial state, N,* s = 1, 89
1)}
4: F =P=1] 0
5:
for i 1...T + do
6:
A= ()
7:
for 1u-1,,,,, E Beam do
8:
= [y]
9:
= ---++1
10:
i 11>7 thenn
11:
continue
12:
Po(ul-), = -Mih, y 9u-1)
B Detailed PRISM algorithm for RNN-T
13:
s(y) = (d))Po())
14:
*a(y) Pa(()
Here describe PRISM algorithm for the Trans-
15:
A == -AU {(g. 1. N, s(y), so(y))}
ducer model that follows the same structure as
16:
if t-== T 'hen
17:
F' -U {(y, <(y))
Algorithm defined earlier. We extend align-
18:
knn, À = Near nbrs h,, N. thresholds)
ment length synchronous decoding algorithm (Saon
19:
Pan KNNN-poobs(K (knn Our he) (Eqn:
20:
for k V do
et al., 2020) for transducers with PRISM. The key
21:
soly k) Po(k)
addition to the generic PRISM for the transducer
22:
s(y k) s(y)(AP((k)) + ()))((K))
models are the steps in lines 13-15. Lines 13-15
23:
if & N.next then
24:
s(y
suly k) (forget knn scores if
show the addition of the Ø hypothesis to move the audio state (h+) in the RNN-T lattice. The rest of the hypothesis expansion with non-blank characters remains the same as the generic PRISM algorithm.
incomplete trie match)
25:
if E N.end then
26:
Sy + k) = s k) (reset)
27:
A
((y
Dur N. nentx((), s(y
R), s6 + k))}
Note that PRISM can be adapted to other search algorithms such as the time-synchronous algorithm described in (Graves, 2012) using similar steps. For the transducer models PRISM uses separate indexes for audio and text embeddings. In the im- plementation, both indexes are queried separately with their respective thresholds. The common val- ues obtained from the indexes are combined to form a single output over which KNN probabilities (described in eqn 1) are computed.
28:
B Andneendd cwo..endnnceomieeiyyyyyysssssys (eeiltssss,,eeaassee beam_size)
29; return sorted
Examples
this section, we show anecdotal examples of how PRISM improves over the baseline models. Table

---

9 shows the examples from Whisper (base.en) base- line and PRISM adapted Whisper for librispeech dataset. Note that with PRISM many challenging dictionary words get corrected due to robust KNN- based matching. Next, in Table 10, we show anec- dotal examples from the Location dataset. Many challenging acoustic words such Kissimmee, Burnhamonsea, etc. are recognized correctly by PRISM. We also show some examples from the challenging Drugs dataset in Table 11, where hard- to-spell medical entities are recognized correctly.
In Table 12 and 13, we show the anecdotal ex- amples from the Location and Drugs benchmarks for transducer models, showing great effectiveness in improving the E-WER.

---

Reference
Representative.
Whisper (small)
For every single day.
Reference
Pharmacy.
Whisper (small)
Guys have great winter every one. Representative.
Reference
Whisper (small)
will finish up before we open all right Thank you O. K. just so they can remove of. Thank you very much remember before they married they were in the disable. family came home seen at the camp alarmed guruji we do not want to ruin the family in sight.
Table Hallucination examples on Internal Dataset.
the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races the curope they had come from lay out there beyond the irish sea curope of strange tongues and valid and woodbeagert and citadeld and entrenched and marshalled races the europe they had come from lay out there beyond the irish sea europe of strange tongues and valid and woodbegirt and citadelled and of entrenched and marshalled races
Reference
Bascline
PRISM
american school boys read with emotions of horror of the albigenses driven beaten and killed with papal legate directing the butchery and of the vaudois hunted and hounded like beasts the effect of royal decree and they yet shall read in the history of their own country of scenes terrible these in the exhibition of injustice and inhuman hate american schoolboys read with emotions of horror of the albiginses driven beaten and killed with papal legate directing the butchery and of the vaudoua hunted and hounded like beasts the effect of royal decree and they yet shall read in the history of their own country of scenes terrible these in the exhibition of injustice and inhuman hate american schoolboys read with emotions of horror of the albigenses driven beaten and killed with papal legate directing the butchery and of the vaudois hunted and hounded like beasts as the effect of royal decree and they yet shall read in the history of their own country of scenes terrible these in the exhibition of injustice and inhuman hate
Reference
Baseline
PRISM
Reference
Fauchelevent grumbled more to himself than to jean valjean fortunately von grumbled more to himself than to jean valjcan fauchelevent grumbled more to himself than to jean valjean and sicut terra sine aqua and sycote terrasin aqua and sicut terra sin aqua
Bascline
PRISM
Reference Baseline
PRISM
Table 9: Examples on Librispeech with Whisper (base.en) and PRISM. The text in the boldface are the entities present in the index.

---

Reference
live in Burnhamonsea
Baseline
I live in Bonamancee
PRISM
live in Burnhamonsea
Reference
Irvine, Salt Lake City, and Sandown are important cities
Baseline
I have unsolved Lake City and sundown my important cities Irvine, Salt Lake City, and Sandown are important cities
PRISM
Reference
Telephone was invented in Kissimmee
Baseline
Telephone was invented in kiss me
PRISM
Telephone was invented in Kissimmee
Reference
Distance between Massapequa Park and Beltsville is small
Baseline
The distance between Massacre Park and belzelis mall
PRISM
The distance between Massapequa Park and Beltsville is small
Reference
The distance between Failsworth and Atascocita is quite large
Baseline
The distance between Faresworth and Ataskasesia is quite large The distance between Failsworth and Atascocita quite large
PRISM
Reference
My house is near Aiken and Bloxwich
Baseline
My house is no Akon and Blockswich
PRISM My house is no Aiken and Bloxwich
Table 10: Examples on Location (entity-rich dataset) with Whisper (small) and PRISM. The text in the boldface are the entities present in the index.
Reference
What are the common brand names of Sudafed What are the common brand names of Sudest
Baseline
PRISM
What are the common brand names of Sudafed What is the recommended dosage for Flavoxate
Reference
Baseline
What is the recommended dosage for Flavix 8
PRISM
What is the recommended dosage for Flavoxate
Reference
What should I do if experience side effects from Epoprostenol Sodium
Baseline
What should I do if I experience side effects from Epipersonal Sodium
PRISM
What should I do if experience side effects from Epoprostenol Sodium
Reference
Can Diltiazem be used in combination with other medications
Baseline
Candleshays can be used in combination with other medications
PRISM
Can Diltiazem be used in combination with other medications
Table 11: Examples on Drugs (entity-rich dataset) with Whisper (small) and PRISM. The text in the boldface are the entities present the index.

---

Reference
Preston is near to the beach but far from Warrensburg
Baseline
Preston is near to the beach but far from war and spoke
PRISM
Preston is near to the beach but far from Warrensburg
Reference
People from Murfreesboro are generous
Baseline
People from murphy sproy are generous
PRISM
People from Murfreesboro are generous
Reference
Cypress and Alliance are very clean
Baseline
Cyprus and Alliance are very clean Cypress and Alliance are very clean live between Canterbury and Wichita live between Canterbay and which attack live between Canterbury and Wichita
PRISM
Reference
Baseline
PRISM
Reference
The branch of the bank is located in Los Gatos
Baseline
The branch of the bank is located in Los Angeles
PRISM
The branch of the bank is located in Los Gatos
Table 12: Examples on Location (entity-rich dataset) with RNN-T Baseline and PRISM. The text in the boldface are the entities present in the index.
Reference
What are the possible withdrawal symptoms if Fabrazyme is abruptly discontinued What are the possible withdrawal symptoms if february time is abruptly discontinued What are the possible withdrawal symptoms if Fabrazyme is abruptly discontinued there generie version of Dexmedetomidine Hydrochloride
Bascline
PRISM
Reference
Baseline
there generic version of dexperate attempting hydrochloride
PRISM
there generic version of Dexmedetomidine Hydrochloride
Reference
What is the mechanism action Colazal
Baseline
What is the mechanism of action of colousal
PRISM
What is the mechanism of action of Colazal
Reference
Are there any lifestyle modifications or precautions to consider while taking Raloxifene Are there any lifestyle modifications or precautions to consider while taking right so think Are there any lifestyle modifications or precautions to consider while taking Raloxifenc
Baseline
PRISM
Table 13: Examples Drugs (entity-rich dataset) with RNN-T Baseline and PRISM. The text in the boldface are the entities present in the index.


Reviews:
[{"review_id": "xvWetwCTnE", "reviewer": "Reviewer_wrWb", "summary": "", "questions": "Question A: What are the model complexities for models in Table 1&2, in terms of computation and memory?\n\nQuestion B: In Table 2, are the results of RNN-T+TCPGen and RNN-T+Deep Cont. taken from literature, or regenerated? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We address the reviewer’s questions below.\n\nQ: `What are the model complexities for models in Tables 1&2 and, in terms of computation and memory?`\n\nA: We first compute the memory occupied by our dictionary for various entities in the entity-rich dataset.\n\n| Dataset          | #No of entities | #Average Tokens | Memory |\n|------------------|-----------------|-----------------|--------|\n| Location (Small) |            1000 |            10.8 |  50 MB |\n| Location (Large) |            3000 |            10.7 | 138 MB |\n| Person Names     |            4347 |             7.0 | 126 MB |\n| Drugs            |            2874 |            11.3 | 132 MB |\n\nIt's worth noting that the sizes of the dictionaries are influenced by the number of tokens (characters or BPE units) present in the words or phrases within the dictionary entries. While the 'Drugs' entity dictionary contains fewer entries, the average token count per entry tends to be larger compared to the 'Person Names' and 'Location' entity dictionaries.\"\n \nHowever, as a fraction of the model size, the additional memory footprint due to the dictionary is modest because unused entries of the trie are not loaded in memory.  For example for the Location (Large) dictionary, we report the memory footprint of the baseline RNN-T model, and RNN-T with PRISM dictionary.   \n\nBaseline RNN-T: \t810.5 MB\t\n\nRNN-T with PRISM: \t850.5 MB\n\nAnalytically, the complexity of the beam search algorithm for RNN-T models is given as O(B * T * U * C), where B is the beam size, T is the number of audio frames in the input, U is the max length of the output, and C is the cost of the forward pass. PRISM method adds a lookup cost to memory in each step of the beam expansion. If the cost of lookup is P, then the overall complexity would become O(B * T * U * (C + P)). \nWe use the state-of-the-art nearest neighbor index from Johnson et al [1], which can do a fast look-up to minimize the latency. \n\n[1] Johnson, Jeff, Matthijs Douze, and Hervé Jégou. \"Billion-scale similarity search with GPUs.\" IEEE Transactions on Big Data 7.3 (2019): 535-547.\n\n\nQ: `In Table 2, are the results of RNN-T+TCPGen and RNN-T+Deep Cont. taken from literature, or regenerated?`\n\nA: Yes, these numbers are cited from the respective papers. The models from these papers are not open source, and Deep Contextual RNN-T is trained on proprietary data internal to the organization which makes reproducibility challenging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This proposed PRISM is applicable to different ASR model architectures, and it doesn't require to retrain ASR models. It also runs efficiently during inference. There are well-designed experimental validation and ablation studies presented.", "reasons_to_reject": "The experimental section could be made stronger with more details added (see questions in the section below).", "questions_to_authors": "Question A: What are the model complexities for models in Table 1&2, in terms of computation and memory?\n\nQuestion B: In Table 2, are the results of RNN-T+TCPGen and RNN-T+Deep Cont. taken from literature, or regenerated? \n", "paper_topic_and_main_contributions": "In this paper authors propose a new method named \"AdaPtation of ASR at Inference-time using Speech-enriched Memory\" (PRISM), which improves ASR results by adapting to predefined dictionaries during inference. PRISM leverages k-nearest neighbor (KNN) and text-to-speech (TTS), and experimental results show its effectiveness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bfDo4KZxdN", "reviewer": "Reviewer_KQzm", "summary": "", "questions": "Is it possible to do an ablation on TTS vs real audio for creating the memory?\n\nIs this method limited to word-based biasing lists? Can it extend to phrases?\n\nWhat is the added computational cost?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We address the reviewer’s questions below.\n\nQ: `Running TCPGen and Deep Contextual RNN-T on our benchmarks?`\n\nA: Since none of the above techniques are open-source, we weren’t able to run the models on TCPGen and Deep Contextual RNN-T. We tried the implementation of Contextual RNN-T (precursor to Deep Contextual RNN-T paper) and trained it on SWB 300H dataset, but we did not observe performance improvements with dictionary biasing. In the original paper, the model is trained on 8000 hours of proprietary data which indicates getting these models to work requires significant resources in terms of labelled data and computation.\n\nQ: `Is it possible to do an ablation on TTS vs real audio for creating the memory?`\n\nA: Collecting real audio for an entity-rich dataset was not feasible within the rebuttal timeline.  However, we perform a proxy experiment using the open-source Librispeech dataset. We use Librispeech’s training set to extract representations for dictionary items and evaluate using the same protocol described in the paper.  We show the WER/E-WER results on the test-clean split.\n\n|                       | WER / E-WER |\n|-----------------------|-------------|\n| Baseline              | 3.5 / 13.3  |\n| PRISM (w/ TTS Audio)  | 3.3 / 11.1  |\n| PRISM (w/ real audio) | 3.3 / 11.5  |\n\nFor this experiment, the training set of Librispeech did not have all the dictionary entries present which reduces the size of the dictionary. For the TTS audio experiment, we only use those entries that are present in the training set to ensure a fair comparison. As shown in the results, PRISM is robust to representations from real audio as well. We will add these results to the paper. \n\nQ: `Is this method limited to word-based biasing lists? Can it extend to phrases?`\n\nA: The biasing list entries can be phrases in our implementation. In fact, 3 of the 4 dictionaries for the entity-rich dataset have phrases with multiple words.\n\n| Dataset | %Entities with  multiple words |\n|---|---|\n| Location (small) | 32.9 |\n| Location (large) | 31.5 |\n| Drugs | 15.9 |\n\nWe plan to release the entity-rich dataset to promote further research in this area.\n\nQ: `What is the added computational cost?`\n\nA: We first compute the memory occupied by our dictionary for various entities in the entity-rich dataset.\n| Dataset          | #No of entities | #Average Tokens | Memory |\n|------------------|-----------------|-----------------|--------|\n| Location (Small) |            1000 |            10.8 |  50 MB |\n| Location (Large) |            3000 |            10.7 | 138 MB |\n| Person Names     |            4347 |             7.0 | 126 MB |\n| Drugs            |            2874 |            11.3 | 132 MB |\n\nIt's worth noting that the sizes of the dictionaries are influenced by the number of tokens (characters or BPE units) present in the words or phrases within the dictionary entries. While the 'Drugs' entity dictionary contains fewer entries, the average token count per entry tends to be larger compared to the 'Person Names' and 'Location' entity dictionaries.\"\n \nHowever, as a fraction of the model size, the additional memory footprint due to the dictionary is modest because unused entries of the trie are not loaded in memory.  For example for the Location (Large) dictionary, we report the memory footprint of the baseline RNN-T model, and RNN-T with PRISM dictionary.   \n\nBaseline RNN-T: \t810.5 MB\t\n\nRNN-T with PRISM: \t850.5 MB\n\nAnalytically, the complexity of the beam search algorithm for RNN-T models is given as O(B * T * U * C), where B is the beam size, T is the number of audio frames in the input, U is the max length of the output, and C is the cost of the forward pass. PRISM method adds a lookup cost to memory in each step of the beam expansion. If the cost of lookup is P, then the overall complexity would become O(B * T * U * (C + P)). \nWe use the state-of-the-art nearest neighbor index from Johnson et al [1], which can do a fast look-up to minimize the latency. \n\n[1] Johnson, Jeff, Matthijs Douze, and Hervé Jégou. \"Billion-scale similarity search with GPUs.\" IEEE Transactions on Big Data 7.3 (2019): 535-547.\n\nThanks for pointing out the missing reference, we will add the citation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is novel. Components are well-motivated and supported by ablations (i.e. dynamic knn weight, trie-based memory). Method is applied to both whisper encoder-decoder and rnn-t.", "reasons_to_reject": "Comparisons to other contextual biasing methods is limited. Results from two other \"inference-time adaptations using dictionary\" methods are shown for comparison, but the authors do not apply TCPGen or deep contextual biasing rnn-t to their own models/experimental setup. Perhaps I have misunderstood something, but it appears that we cannot directly compare these methods based on tables 1 and 2.", "questions_to_authors": "Is it possible to do an ablation on TTS vs real audio for creating the memory?\n\nIs this method limited to word-based biasing lists? Can it extend to phrases?\n\nWhat is the added computational cost?", "paper_topic_and_main_contributions": "This work applied memory-augmented decoding (kNN retrieval) to ASR systems (both transducers and enc-dec). This method requires a speech/text representation to map to target token sequences; the authors propose to use TTS to obtain speech inputs to the model. Results show consistent improvements over three datasets (librispeech, medical domain dataset, and an entity-rich dataset).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://arxiv.org/abs/2301.00066"}, "extra_text": {}}, {"review_id": "B037NoaKbM", "reviewer": "Reviewer_Kmjd", "summary": "", "questions": "For Table3: please say if PRISM is used with RNN-T or Whisper and please explain why E-WER is sometimes lower than WER. E-WER is supposed to contain rare words that are harder to decode.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We address the reviewer’s questions below.\n\nQ: `Please be more specific on how do you get the alignments.`\n\nA: For encoder-decoder based Whisper model, the alignments are obtained by teacher forcing the decoder output during beam search. We store the representations of the decoder state as the key in the memory, while the value is the token that is to be predicted for a given dictionary item.\n\nFor RNN-T, this is slightly more involved as a blank character is also part of the model’s output, which makes teacher forcing difficult. Our solution is to incrementally nudge the model’s probability towards the token in the sequence. For example, if the dictionary word is “As_ pi_ rin” and the current generation so far is “As_ pi_”, we increase the likelihood of token “rin” by a constant factor so that its prediction is favoured as opposed to other tokens. We find that this closely mimics the decoding process and the representations obtained are closely aligned with PRISM’s inference. \n\nWe will make this clearer in the paper. Thanks for pointing this out. \n\nQ: `Why E-WER is sometimes lower than WER? E-WER is supposed to contain rare words that are harder to decode.`\n\nA: In the internal datasets, we often encounter audio files that are short utterances with long pauses and silences in the beginning. This leads to hallucinations in both the RNN-T and Whisper models. In particular, Whisper significantly hallucinates as demonstrated in the two examples below.\n\n*Example #1*\n\nReference: representative\n\nWhisper output: I will finish up before we opena ll right. Thank you. O.K. Just so they can remove oh thank you very much. I remember before they married they were in the disabled family came home seen at the camp alarmed guruji we do not want to ruin the family in sight.          \n\n*Example #2*\n\nReference: \tpharmacy\n\nWhisper output: \tguys have a great winter everyone.\n\nThis results in an inflation in overall WERs. Entity WERs, on the other hand, are computed only over spans containing the desired entities in the alignment. This is the main reason why E-WERs are sometimes lower than the overall WERs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology is solid: large set of experiments, multiple datasets, comparison to the state of the art. Even if the improvement compare to Deep Contextual, the method is novel and bold (using TTS and k-NN was a risky solution as dealing with speech embedding is always harder than using text embeddings).", "reasons_to_reject": "The method is quite hard to follow. I am still not sure to really understand how everything works. Especially section 3.1 please be more specific on how do you get the alignments. ", "questions_to_authors": "For Table3: please say if PRISM is used with RNN-T or Whisper and please explain why E-WER is sometimes lower than WER. E-WER is supposed to contain rare words that are harder to decode.", "paper_topic_and_main_contributions": "The authors present an inference method for streaming and non-streaming ASR to decode rare words. The method is based on TTS and k-NN search to bias the decoding of rare entities towards a set of pre-defined words.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XKpj1OuiIx", "reviewer": "Reviewer_emcA", "summary": "", "questions": "1. Based on description in section 4.1 and table 1, I'm confused by line 455 \"despite starting with a much stronger Whisper baseline\", how can it be much stronger if both this paper and Sun et al 2023 are using base.en. \n2. In Table 2, the RNN-T baseline numbers for the three approaches are different, e.g. on test-clean, WERs are 5.5 for Sun et al. 3.7 for Le et al. and 3.5 for this work. How can that be the case, as the RNN-T model is the same, and contextual biasing modules are not added for baseline.\n3. For Whisper, there should be a very simple way to achieve contextual biasing. As indicated in figure 1 of the original whisper paper [1] and executed in the AVSR section of [2] (note that [2] is published after the submission of this paper), is to use a predefined dictionary in the prompt to the decoder can bias the model to output words from or related to the words in the predefined dictionary. Using prompt for contextual biasing seems particularly suitable for the experiment in section 4.2 where the evaluation is on Enterprise dataset, because the vocab. is small, and therefore the overhead to context length is small.\n\n[1] Radford, Alec, et al. \"Robust speech recognition via large-scale weak supervision.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Peng, Puyuan, et al. \"Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization.\" Interspeech, 2023.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We address the reviewer’s questions below.\n\nQ: `Latency compared with Neural methods.`\n\nA: We compare the latency of PRISM with our implementation of Contextual RNN-T paper by measuring the Real-Time Factor (RTF). RTF is the time it takes to decode 1 second of audio. \n\n| Dictionary Size | Contextual RNN-T | PRISM |\n|-----------------|------------------|-------|\n|             100 |             0.71 |  0.70 |\n|             250 |             0.76 |  0.71 |\n|             500 |             0.82 |  0.74 |\n|            1000 |             0.88 |  0.76 |\n|            2000 |             0.92 |  0.79 |\n\n\nWe show that the latency of PRISM is comparable to the latency of neural methods such as Contextual RNN-T [1] (precursor to Deep Contextual RNN-T). Computing attention for each of the context entities becomes expensive as the dictionary size increases, similar to PRISM.\n\n[1] Jain, Mahaveer, et al. \"Contextual RNN-T for open domain ASR.\" arXiv preprint arXiv:2006.03411 (2020).\n\nQ: `PRISM better than Le at al 2021a.`\n\nA: Since the model from Le at al 2021a is not publicly available, we could not evaluate it on the entity-rich datasets. We will fix our claim that PRISM is comparable to Le at al 2021a in terms of performance despite being an inference time adaptation to the dictionary and not requiring expensive model fine-tuning. \n\nQ: `Discrepancy on Whisper results on base.en from Sun et al and PRISM for Librispeech dataset.`\n\nA: While the evaluation on the Librispeech test set is performed using the same Whisper base.en model, some key hyperparameters such as beam size, patience, etc. are not described in Sun et al. For PRISM, we use a beam size of 10 and patience=1.0 for the evaluation. This would explain the discrepancy between results.\n\nQ: `Different WERs on Librispeech for RNN-T Models in Sun et al, Le et al 2021a, and PRISM?`\n\nA: While a uniform model for training RNN-T models is appealing, different papers have different model architectures for training the RNN-T models. \n\nSun et al: Has a 4-layer BiLSTM encoder with 1024 hidden dimensions. \n\nLe et al 2023a: Has a 20-layer streamable low-latency Emformer model for the acoustic network. (83M parameters)\n\nPRISM: Has a 6-layer BiLSTM encoder with 6024 hidden dimensions. (56M parameters)\n\nEach paper has a different model architecture with a different number of parameters used for training the baseline model. This would explain the discrepancy in the baseline results using RNN-T for the Librispeech test sets.\n\nQ: `Achieve contextual biasing in Whisper using AVSR biasing technique shown in Peng et al 2023.`\n\nA: We thank the reviewer for this exciting experiment. We show a comparison of Peng et al. 2023 with PRISM on the Enterprise dataset.\n\n|                              | Category - 1 | Category - 2 | Category - 3 | Category - 4 | Category - 5 | Average     |\n|------------------------------|--------------|--------------|--------------|--------------|--------------|-------------|\n| RNN-T Baseline               |  32.9 / 26.3 |  38.8 / 32.6 |  48.6 / 26.6 | 125.0 / 83.3 |  33.0 / 40.0 | 55.7 / 41.8 |\n| Whisper (small)              |  32.9 / 21.2 |  77.6 / 28.6 |  45.2 / 24.5 | 116.7 / 58.3 |  53.2 / 44.1 | 65.2 / 35.3 |\n| Whisper (small) + Peng et al |  19.2 / 13.1 |  63.3 / 26.5 |   57.1/ 40.0 |  58.3 / 33.3 |  58.7 / 30.9 | 51.3 / 28.8 |\n| PRISM                        |  26.0 / 15.8 |  26.5 / 26.5 |  40.0 / 20.0 |   16.7 / 8.3 |  28.2 / 29.4 | 27.5 / 20.0 |\n\nWe observe that PRISM significantly outperforms the Peng et al. method on our Enterprise benchmark. The average WER/E-WER from Peng et al. is 51.3/28.8 compared to 27.5/20.0 obtained using PRISM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of organizing memory as a trie and use a backoff beam to avoid spurious matches is elegant, and is shown to avoid worsening the performance on words not in the rare word dictionary\n2. the approaches works for both encoder-decoder models and Transducer models. While it's not as straightforward to apply their approach on Transducer models, the authors found tricks that make it work\n3. the evaluation is comprehensive - on publicly available dataset, proprietary data set in medical domain, and synthetic entity-rich dataset", "reasons_to_reject": "1. Latency of the proposed approach, and comparison with other approaches are not studied. I suspect that this approach is faster than neural nets based adaptation approaches that are compared in the paper, but this needs to be shown quantitatively.\n2. whether the proposed approach is superior to Le at al 2021a is not clear to me, because PRISM and Le et al. 2021a are only compared once in table 2, and PRISM performs worse than Le et al.2021a. However, the paper claims that \"PRISM achieves superior WER reductions\" in line 467.", "questions_to_authors": "1. Based on description in section 4.1 and table 1, I'm confused by line 455 \"despite starting with a much stronger Whisper baseline\", how can it be much stronger if both this paper and Sun et al 2023 are using base.en. \n2. In Table 2, the RNN-T baseline numbers for the three approaches are different, e.g. on test-clean, WERs are 5.5 for Sun et al. 3.7 for Le et al. and 3.5 for this work. How can that be the case, as the RNN-T model is the same, and contextual biasing modules are not added for baseline.\n3. For Whisper, there should be a very simple way to achieve contextual biasing. As indicated in figure 1 of the original whisper paper [1] and executed in the AVSR section of [2] (note that [2] is published after the submission of this paper), is to use a predefined dictionary in the prompt to the decoder can bias the model to output words from or related to the words in the predefined dictionary. Using prompt for contextual biasing seems particularly suitable for the experiment in section 4.2 where the evaluation is on Enterprise dataset, because the vocab. is small, and therefore the overhead to context length is small.\n\n[1] Radford, Alec, et al. \"Robust speech recognition via large-scale weak supervision.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Peng, Puyuan, et al. \"Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization.\" Interspeech, 2023.", "paper_topic_and_main_contributions": "Topic: contextual biasing for ASR systems. \n\nThe authors proposed an inference time adaptation algorithm inspired by KNN LM to bias ASR model decoding towards a predefined rare word dictionary, while do not worsen the performance on words not in that rare word dictionary by using a trie for exact word match and using a backup beam. The proposed method leverages speech synthesis systems, and is training-free, which are the main novelty compared to other contextual biasing approaches.\n\nExperiments show that the proposed algorithm works on both encoder-decoder architecture (the Whisper model) and Neural Transducers (the RNN-T model)\n\nIn addition, the release of the entity-rich dataset could be a valuable contribution to the community as well.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 35: state-of-art -> state-of-the-art\n\nLine 195: … perform a memory augmented inference to… -> … perform memory augmented inference to…"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel method that allows adapting ASR systems (transducers and enc-dec) to dictionaries containing rare words and phrases without retraining the original ASR model. The method uses kNN retrieval and TTS to obtain speech for rare words. The idea is well-presented and achieves consistent gains on three datasets. The authors will also release their entity-rich speech dataset to facilitate further research into dictionary-based ASR adaptation.

The authors addressed all reviewers' comments, especially regarding computation cost and comparison with additional methods. All reviewers are excited about this paper and assigned high soundness scores.
Decision: Accept-Main