PersonaLM: Language Model Personalization via Domain-distributed Span Aggregated K-Nearest N-gram Retrieval Augmentation
Authors: Puneet Mathur
Zhe Liu
Ke Li
Yingyi Ma
Gil Keren
Zeeshan Ahmed
Dinesh Manocha
Xuedong Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: speech recognition
language modeling
personalization
domain adaptation
retrieval augmentation
TL;DR: A novel language model retrieval augmentation technique that leverages a contrastively trained neural retriever model to select the best matching external domains/users for LM augmentation for ASR.

Abstract: We introduce PersonaLM - Domain-distributed Span-Aggregated K-nearest N-gram retrieval augmentation to improve language modeling for Automatic Speech Recognition (ASR) personalization. PersonaLM leverages contextually similar n-gram word frequencies for recognizing rare word patterns associated with unseen domains. It aggregates the next-word probability distribution based on the relative importance of different domains to the input query. To achieve this, we propose a Span Aggregated Group-Contrastive Neural (SCAN) retriever that learns to rank external domains/users by utilizing a group-wise contrastive span loss that pulls together span representations belonging to the same group while pushing away spans from unrelated groups in the semantic space. We propose ASAP benchmark for ASR LM personalization that consists of three user-specific speech-to-text tasks for meetings, TED talks, and financial earnings calls. Extensive experiments show that PersonaLM significantly outperforms strong baselines with a 10-16% improvement in perplexity and a 5-8% reduction in Word Error Rates on popular Wikitext-103, UserLibri, and our ASAP dataset. We further demonstrate the usefulness of the SCAN retriever for improving user-personalized text generation and classification by retrieving relevant context for zero-shot prompting and few-shot fine-tuning of LLMs by 7-12% on the LAMP benchmark.

PersonaLM:
Language Model Personalization via Domain-distributed Span
Aggregated K-Nearest N-gram Retrieval Augmentation
Puneet Mathuri; Zhe Liul, Ke Li', Yingyi Mai, Gil Keren', Zeeshan Ahmed,,
Dinesh Manocha', Xuedong Zhang?
University of Maryland, College Park Meta
*(puneetm, dmanocha G@umd. edu
(Zzeliu, , yingyimannnnnn gineiiia,,gkkeeeee Ikeren, ahzee, Xuedong)emeta.comm
Abstract
TRAINING
Trans Duta
We introduce PersonaLM Domain-distribure span-Aggregated K-nearest N-gram retrieval aug- mentation to improve language modeling for Automatic Speech Recognition (ASR) person- alization. PersonaLM leverages contextually similar n-gram word frequencies for recogniz- ing rare word patterns associated with unseen domains. It aggregates the next-word proba- bility distribution based on the relative impor- of different domains the input query. To achieve this, we propose Span Aggregated Group-Contrasti Neural (SCAN) retriever that learns rank external domains/users by utilizing group-wise wise contrastive span loss that pulls together span representations belonging to the same group while pushing away spans from unrelated groups in the semantic space. We pro- pose ASAP benchmark for ASR LM personaliza- tion that consists of three user-specific speech- to-text tasks for meetings, TED talks, and fi- nancial earnings calls. Extensive experiments show that PersonaLM significantly outperforms strong baselines with 10- 16% improvement in perplexity and 5-8% reduction in Word Er- ror Rates on popular Wikitext- 103, Userl.ihy Libri, and our ASAP dataset. We further demonstrate the usefulness of the SCAN retriever for im- proving user-pore seer-persomalized text generation and classification by retrieving relevant context for zero-shot prompting and few-shot fine-tuning I.LMs by 7-12% on the LAMP benchmark.
Fimetaiing
Copua
Vide
Nod Pard Pacdiiion
speech
Thnnterrer
INFERENCE
Now Whld Prefition
= (Mh M..))
Pear
El (U000)
Langaag
then (5,00))
(0.0))
Figure 1: ASR Personalization: During training. the LM pre-trained generic corpus and optionally fine-tuned on the out-of-domain corpus (see dotted). For query g in- ference, LMM output PLM interpolated with the prohability distribution pert retrieved from domain- external cor- pus word prediction p(wxlg) and ASR re-scoring.
streaming ASR re-scoring, and on-device stream- ing ASR models for unseen users and domains due to the existence of user-preferred rare word patterns, facts, proper names, and other domain- specific tail words not seen frequently in the train- ing data (Schick and Schütze, 2019; Maynez et al., 2020; Serai et al., 2022). Retrieval augmentation (Lewis et al.. 2020) (see Fig. 1) can help person- alize LMs by explicitly exposing them to external world knowledge during inference (Borgeaud et al.. 2022). LMs leverage the retrieval mechanism select contextually relevant users/domains from an external corpus and then attend over that knowl- edge to inform their predictions (Liu et al., 2022).
1 Introduction
Language modeling a core task NLP with im- portant applications in automatic speech recogni- tion (ASR) (Mikolov et al., 2010; Chen et al., 2015; Xu et al., 2018). Pre-trained LMs (Iric et al., 2019 9a; Li et al., 2020a) memorize a surprising amount of knowledge from their training corpora in the un- derlying neural network Partroters(Petroni et al., 2019; Jang et al., 2022). However, this makes it dif- ficult to personalize them for text generation, non-
Prior research has explored KNN-LM memoriza- tion (Khandelwal et al., 2020), RETRO (Borgeaud et al., 2022), and attention-base caches (Grave al., 2017). However, these methods are suited for retrieving relevant domains/users prior con- text selection, rather were proposed primarily to en- hance the memorization capabilities of LM. Recent approaches, notably REALM (Guu et al., 2020)
Work done during internship at Meta

---

and RAG (Lewis et al., 2020), incorporate a non- parametric retrieval step during LM pre-training, thus being unable to adapt their context representa- tion for unseen domains.
Downstream Application: SCAN retriever improves context retrieval in personalized text generation and classification via zero-shot prompting and few-shot fine-tuning of LLMs on LaMP corpus by 7 - 12%.
We address the challenge of capturing rare word patterns associated with specific users/domains by exploiting n-gram word frequencies from under- lying domains. Further, we hypothesize that gram patterns are domain/user-speciffe and aug- menting LM predictions with n-gram probabilities from a subset of query-relevant users/domains may lead to better personalization. We anticipate the retrieval augmentation through n-gram frequencies to have additional advantages of very low com- putational overhead, efficient caching, and asyn- chronous updates for newer data without the need for re-computation from scratch.
2 Related Work
Language Modeling for Rare Words Prediction: Earliest works explored the use of LSTM with auxiliary pointer networks to predict rare words and long-term dependencies in language modeling (Merity et al., 2017). Neural cache augmentation (Grave et al., 2017) stored past hidden activations in cache memory to predict out-of-vocabulag words. Implicit cache memorization (Li et al., 2020b) used cache to store past word occurrences as an alter- native to the uttention-based pointer mechanism. For the ASR re-scoring task, cross-sentence neural LMs proposed to use word usage in preceding sen- tences re-rank n-best hypotheses of upcoming sentences (Sun al., 2021: Irie al., 2019b). Retrieval Augmentation for Language Model- ing: kNN-LM (Khandelwal et al., 2020) interpo- lated pre-trained LMs with contexts extracted from an external data store using the kNN algorithm. REALM (Guu et al., 2020) proposed neural re- triever to leverage external knowledge during LM pre-training. (Ram al., 2023) augmented GPT- with large episodic memory for zero-shot reduction in perplexity. Retrieval-Enhancee Trans- former (Retro) (Borgeaud et al., 2022) retrieved document chunks similar to preceding tokens using cross-attention module. Our work is the first to use
We propose PersonaLM Domain-distributed
Span-aggregated
k-Nearest N-gram Language Model that aggregates top-k nearest n-gram co-occurrence fre- quencies from each domain weighted according to the domain's 's relative importance the input query, which is augmented with the target word probabil- ity distribution for next word prediction and ASR second- ppss re-scoring. We utilize novel Span Aggregated Group-Contrastive Neural (SCAN) retriever that can learn highly discriminative semantic representations to distinguish between text spans from the same group as opposed to random spans using group-wise contrastive Joss. SCAN retriever assigns relevance score to each textual documeentrrecording from an external corpus based on its semantic similarity with the input query to weigh their contribution the final prediction. Our main contributions are:
n-gram representations over document spans retrieve rare word patterns from the most relevant external knowledge domains.
PersonaLM retrieval augmentation for ASR personalization that leverages group-wise con- trastive loss to train Span Aggregated Group- Contrastive Neural (SCAN) retriever for ranking query-relevant external domains/users k-nearest n- and augments improve LM predictions. gram frequencies to
PersonaLM Retriever Augmentation
Fig. describes our proposed PersonaLM retrieval augmentation approach that biases the predictions from a base LM with the next word probabili- ties based on the relevance of unseen topic/users to the input query. Given an input query = (W1, wt-1) at inference, autoregressive LMs estimate the probability distribution for target to- ken w, Pry To augment the LM out- put with domain-specific word occurrence infor- mation, we calculate the probability distribution of next word prediction over the vocabulary condi- tioned on the relevance the underlying domains (d,, d.. dx to the query (q) by marginalizing
ASAP novel benchmark for ASR LM personalization consisting of three user- specific ASR tasks in the domains of meet- ings, TED talks, and financial conference calls. PersonaLM significantly outperforms strong baselines on ASAP benchmark, UserLibri, and Wikitext- 103 corpus by ~ 10 16% per- plexity gain and ~ 5 - 8% WER reduction.

---

Bigrem Frequencies
Trigram Frequencies
4-gram Froquancies
Rettwal
Bigsu Data Store
Treaam Outa Stoe
1pxt Quar
Data Stvro
Med Word Prediciio Rescorgg
Dorsin!
SCAN
Produc
Det
00
Pawoo zar(M)
Dersan.
Retrelver
BM
the 12
11200
PiAIv)
on (0,005
Prar (t..g)
Inferencing SCAN Retriever
Domain-distribuled
N-gram Co-occurrence Matrices
Language Model
Figure 2: Mr: At inference, compute the relevance score P(d:1g) between the query and domains d, as the dot
product of their SCAN retriever representations. We construct data store for each n-gram frequency matrix. k-most similar n-gram contexts wrt to the input query are retrieved and their weighted summation based on the domain's 's relevance seore is computed get probability distribution targets P (we and interpolated with LM probabilities PLac( (wrlg).
Group Connastivv Loss
text while paragraph-level spans can capture more abstract properties of the text such as topic infor- mation. In this work, we explicitly sample a set of text spans at the phrase level, sentence level, and paragraph level. We extract T spans for each level of granularity to obtain total of 3T spans corre- sponding to each input document D. Text spans (I..'.,, Ree are sampled such that their start posi- tion is taken from uniform distribution U (11, (-,7-110 1) the span length
Domalt
60C
Tiu
Tom
Encoder
Semaniic Space
(C
Donen
Donanu
e.e Domaan Xi
Push
4,
Training SCAN Retriever
Charry Trepneorrttttn
Span Deeppoennarnon
+ 1 is determined by a beta distribution B(a, 6) x (I (1-s), s), where denotes in the
Input query followed by text spans (Ze, ..:..... ... ..)) from the positive domain (d't) and N negative domains (d separated by [SEP] are passed through the encoder followed by projector layer and an average pooling layer. SCAN retriever is trained via group group-wise contrastive loss to force the hidden representation of the query ÿ close to its own spans while far away from other groups.
Figure 3: SCAN Retriever:
the number of phrasse/sentenneespaaagggg document with a, b as hyperparameters.
Multi-domain Text Encoding: Formally, let there be query with an associated positive domain d+ and pool of N domains (d.). For each domain, negative concate- nate the input query with multiple spans, add special [CLS] token before the query text and a [SEP] token between the multiple spans to obtain the concatenated text sequence [CLS], 9. [SEP], - [SEP] ... ..,"," ... .. We encode the input text sequence using multi- layer Transformer encoder which maps each word to a low-dimensional dense representation ho, h1,".. h4 = Transformer(nn) #1. 2.), where E RH with H the size of hidden dimension. We then pass the encoded repre-
over the retrieved documents as:
Pb- PD-RAG(welg) E P(dilg) x P(wildi, q)
3.1 SCAN Retriever
Fig. 3 shows the architecture of our proposed Span Aggregated Group-Contrastive Neural (SCAN) re- triever which is a Transformer encoder pre-trained with Masked Language Modeling (MLM) well novel group-wise contrastive span loss to force the semantic representations of the input query close its ground truth domain and away from random spans from the different domains. During the training of the SCAN retriever, we first sam- ple spans of varying granularities from multiple domains and encode them using the Transformer encoder. Group-wise contrastive loss is then ap- plied to learn discriminative semantic presentations for enhanced retrieval performance.
sentation through a project layer which is a fully-connected layer followed by a non-lincar activation p; = Tanh(F (FNFFFFNNMM (N:)) to prevent representation collapse during contrastive learning. An average pooling operation is applied over the projected word representations to obtain the output representations as = Apg.
Document Span Sampling: Different granular- ities of spans capture different properties of the input text. For example, phrase-level spans can capture specific words or entities mentioned in the
Group-wise Contrastive Training: We use group- wise contrastive learning that incentives for rep- resentations of spans in a group sharing the same semantics to be similar while penalizing the rep-

---

resentations of groups expressing different seman- ties be distinguished from each other. It en- courages the SCAN retriever to discriminate and score related queny-span pairs (from the same do- main) higher than unrelated (from different do- mains) pairs. Given mini-batch with N domains, the group-wise contrastive loss function Lec is applied over M = N * (3T 1) spans as:
neighbor n-grams based on their BERT representa- tions. The top-k probability distributions obtained from the n-gram datastore are summed over the entire vocabulary to get ((ve- >w)--))wul The next word prediction for a selected domain is cal- culated as a weighted sum of k-nearest bigrams, trigrams, and 4-gram frequencies as P(wxlde, q) Σj= 2j [[we -w--))1" fime, se -.) where or E [0, 1] are hyperparameters.
N
exp(sim(2s RT)/T)
LGC 37
log
اイ= Lezs exp(sim((z; 25)/T)
3.4 LM Augmentation
REN+
We compute the PersonaLM retrieved next-word probability PD-RAG(wilp) by summing the nor- malized k-nearest n n-gram co-occurrence probabili- ties weighted by the relevance score of the selected domain over the target vocabulary as:
where sim(.) refers the dot product and is the temperature parameter.
3.2 Retrieving Relevant Domains
At inference, we encode the concatenated query text with the document spans through the SCAN retriever. The relevance score assigned by the retriever model to a particular domain d, based on the input query q is denoted by p(d:(g) is computed via the dot product operation between the [CLS] token embedding (z₂) and the average pooled embeddings of the document spans (za,) as P(dl4) = sim(=g: zd,).
PD-RAG((:19) Σ
sim(2y 2d,)
a (1i-) (0 -0m-)))) (mm
Finally, we interpolate the retrieved next- word probability distribution through PersonaLM (PD-RAG) with the base LM output (PLM) using hyperparameter À to produce the final nort-word probability distribution as:
3.3 Constructing k-Nearest N-gram Co-occurrence Matrix
We hypothesize that words that occur together in specific domain have high chance of trigger- ing during inference. N-gram frequency matrices store co-occurrence information in a matrix such that each cell (i, j) corresponds the normalized frequency of the observed word wi E V follow- ing a sequence of n words in a document. To exploit the word-level co-occurrence probabilities in text, we construct the n-gram frequencies ma- trices for each target domain d, for n E [2, 1] over the entire vocabulary set V as f((or-. -wr-1) w),1 However, n-gram frequencies tend get sparser with higher values of n. Moreover, restricting the query n- n-grams to only exact matches leads to a loss of information due to ignorance of semantically similar n-grams (Li et al., 2017). To overcome these drawbacks, we utilize k-nearest n-grams. We construct domain-specific n-gram data stores with keys as the pair of Bert vector representation of the n-gram (BERT(u:,, - wi-1) and next word (w,); values as the frequency of co-occurrence in the given domain fi ([w w),=.)- At infer- ence, M uses K-NN with Euclidean dis- tance metric to query the datastore for top-k nearest
P(wxlg) = XPD nsc(wela) +(1 -11-A) AMM
4
Experiments
4.1 Training SCAN retriever
We start with pre-trained BERT model as the en- coder which is further trained using group-wise contrastive learning objective on the following IR benchmarks: (1) MS MARCO Passage Rank- ing (MARCO Dev Passage), (2) MS MARCO Document Ranking (MARCO Dev Doc) Nguyen et al. (2016): (3) TREC 2019 Passage Ranking (TREC2019 Passage) and (4) TREC 2019 Docu- ment Ranking (TREC2019 Document) Craswell et al. (2020). While training the SCAN retriever, we train the Transformer model with permutations of span order. Hence, the spans from positive do- mains are not necessary to always precede the spans from negative domains. Similarly, the spans from an individual domain are not necessitated to be al- ways together in the sequence. This ensures the learned span representations are robust in terms of their relative order. More details are in Appendix. B.9.

---

4.2 Datasets
Balowrt
Tan
V31
Wncch Soo
Doman
Eamingo2
40.88
1.3k
14.2k
20K
Preenced Compuayy
100
AMI Meriing Corpu
12.08
277
588
IIK
Meeting Specker
125
We evaluate the PersonaLM method on our pro- posed ASAP dataset and the UserLibri corpus. We propose the ASR Language Model Person- alization (ASAP) benchmark which aims to evalu- ate the efficacy of LMs for personalized automatic speech recognition based on user/domain-specifee information for next word prediction and ASR n- best second- pass restoring.
Ta3A IIM vs
188.9
76.6
9.7%
40K
TRUD spokkr
2151
Whlten 193 Use
25C
303K
10X
250K
Willyedia Page
30%
6.7M
703K
10K
10K
Books
107
Table I: Data stats of ASAP, UserLibri, and WikiText- 103,
I shows statistics on dataset size and distribution. UserLibri Dataset (Breiner el al., 2022a) refor- mulates the Librispeech corpus into user-specific trati-trrnncriptt pairs supplemented with personal- ized text-only data corresponding to each user with similar vocabulary, character names, and writing styles as the recordings.
(1) Personalized Meeting ASR of user's spoken utterances in professional meetings. This task as- sesses the model's ability capture user-preferred dialogue patterns and linguistic characteristics. We leverage the AMI Meeting corpus (Kraaij et al.. 2005) by splitting the user-specific recordings to obtain personalized utterance-text pairs.
4.3 Experiments for ASR Personalization
Language Model Architecture: We experiment with both LSTM and Transformer LMs. LSTM model has 2 layers with a 300-d embedding layer and a hidden dimension of 1500. Transformer LM consists of 4 layers of encoder-decoder with 12 heads, 128-d hidden representations, and feed- forward layer of 3072-d. For generating ASR n- best hypotheses, we use a pre-trained RNN-T ASR Model with Emformer encoder (Shi al., 2021), LSTM predictor, and a joiner with 80M parameters. More hyperparameter details in Appendix Sec.B.9 Pre-training LMs: LSTM and Transformer LMs are pre-trained on Librispeech (Panayotov et al., 2015) train set for 25 epochs with batch size of 256, Adam optimizer and cross-entropy loss for next word prediction. We select model checkpoints with least perplexity on the Librispeech validation. Adaptation to Unseen Domains: We evaluate the retrieval augmentation in two settings: (I) Without fine-tuning: LM pre-trained on generic corpus; (2) With fine-tuning: LM pre-trained on generic corpus and fine-tuned on the entire out-of-domain train corpus. In both cases, evaluation is performed on the out-of -f--ma-n test set.
(2) Personalized TED Talk ASR to convert recorded TED talks delivered by specific user into text transcript. This task evaluates the LM's capability to capture topics-aware word patterns in the speeches from the TED-LIUM v3 corpus (Her- nandez et al., 2018). We split the recorded TED talks temporally with historical utterance-text pairs forming the domain-speciffe train set.
(3) Personalized Financial Earning Calls ASR: Perform speech-to-text for financial earnings con- ference calls that include company-specine finan- cial information. The task aims to evaluate language model's capacity to extract company- specific named entities, abbreviations, facts, and long-tail word patterns. We adopt the conference call-transcript pairs from combined Earnings-21 (Rio et al., 2021) and Earnings-22 22 (Del Rio et al., 2022) datasets. Additionally, we use Wikitext- (Merity et al., 2017) to test the LM domain adapta- tion in topic-specific documents. PersonaL.M can be applied to a variety of different datasets with varying "domains" i.e. speakers/noiicssettegorriss For example, the recordings in the AMI Meet- ing corpus are split based on speaker IDs. Do- mains Earnings-21+22 data correspond to dif- ferent companies. specific domain in the TED- LIUM v3 dataset refers to a TED talk speaker. Do- mains in Wikitext-109 103 correspond to individual Wikipedia pages. Data Preprocessing: To study personalization in language modeling, refor- mulated all the listed datasets identify explicit users/domains. For each dataset, we combined the original train/val/test portions and splitted user- based data in the ratio of 70:10:20 such that each user/domain appears only in one of splits. Table
Baselines: (i) LSTM/Transformer:: Language model without any augmentation, (ii) Neural Cache Model (Grave et al., 2017) augments LM output with continuous a cache memory of previous hidden states. The stored keys are used to retrieve the next word through dot product-based based memory lookup with the query. (iii) KNN-LM (Khandel- wal al., 2020): Following (Das al., 2022), we adopt KNN-LM memorize context vectors from representations from out-of-domain train set an external data store. During inference, the k- k-nearest neighbors the decoder output representations are

---

interpolated with LM output. (iv) Unified N-gram Co-occurrenee: N-gram word frequency matrices built from the combined out-of-domain train set of each user/domain are augmented with LM at inference. (v) PersonaL.M w) other retrievers:
WikTTed-
Barning-21+22
Model
Perplesity ())
Perplesity (1)
L.TTM
1384.1
757.6
Neumi Cache
1325.3
RNN-LM
1191.8
65%
Wilhoot Pimes
United N gram Co occurencce Retrivaal
603.6
477.2
Penural LM APP
564.3
420.3
PensonaLM Komtneve
412.3
PersoraLM
527.9
405.6
Replacing SCAN retriever with DPR (Karpukhin et al., 2020) or Contriever (Izacard et al., 2021).
Personal M. who SCAN Reniever
542.8
415/4
PersonaLM wio Neares N graz
542.3
LSTM
101.9
66.2
Ablation Studies: (i) PersonaLM w\o SCAN re- triever: We use the dot product of query and do- main context encoded through pre-trained BERT to compute the weightage of each domain; (ii) Per- sonaLM w/o k-Nearest N-grams: Similar to kNN- LM, augment the NWP with k-nearest Bert con- texts vectors extracted from individual domains. We compute relevance scores from the SCAN re- triever to get the weighted sum of the probability distributions from each domain.
Namnd Cache
97.6
666.
INN-LM
91.8
65.7
With Tine
gram Co-occumence Retrieval
PersonaLM w WPP
84.2
63.0
PenunaL.M w Konnnevvr
5022
59.6
PersonaLM
77.8
57.8
PersonaLM who SCAN Reniever
82.7
61.5
Personal M who K-Neare Negrast
82.3
61.9
(a) LSTMLM
WikTestt-103
Earning-
Medel
Poipleity
Perplexity
Transfumner
1322.3
Vilhou Fine-tuiin
Noural Cache
1295.
802.4
ENN-LM
1150.4
717.8
Unified N-gram Co-accurenes Retrieval
585.3
454.8
Evaluation Metrics: We utilize word-level per- plexity scores to evaluate LM performance for next- word prediction. We also report Word Error Rate (WER) for ASR second-pas. re-scoring for ASAP corpus. For UserLibri, we evaluate WER per user for both streaming and non-streaming ASR settings. For each model, we report results with minimal per-
PersonaLM APR
578.2
Persotal M w (Comiiveer
560.3
446
Penunaal M
567.6*
440.4
wis SCAN
$72.9
Personal W-Nreee Negnes
Trunsfomner
55.2
With Inn-tuminng
Namm Cadie
858
$4.9
LNN-LM
79.3
$4.2
Unifeed gram Co co-orrnnence Retrivall
76.3
53.8
ADPR
PenonaLM Konntnever
72.5
09.6
PersonaLM
70.9
48.2*
plexity by iterating the interpolation parameter X between 0 to 1 in increments of 0.1.
Persoral who SCAN Resiever
74.8
$1.5
PensonaLM who Meares M grass
73.5
503
(h) Transformer I.M
ASR Model Architecture for UserLibri: We uti- lize separate architectures for streaming and non- streaming ASR. The Conformer Hybrid Autore- gressive Transducer (HAT) from Breiner et al. (2022a) has 86M parameter and consists of 12 en- coder layers of 512-d, 4 attention heads, convolu- tion kernel size of 32, and a HAT decoder with a single RNN layer of 640-d. Each label is embed- ded with 128-d, and inputs are tokenized with a Ik Word-Piece Model trained on the LibriSpeech train set. The models are trained with Adam using group-norm (Wu and He, 2018). For streaming ASR evaluation, the Conformer HAT model uses causal convolution, local self-attention, and left- sided context stacking to ensure no look-ahead. The ste-streaming version has multi-headed atten- tion. COnformer models in both cases are trained on 960 hours of LibriSpeech audio training set. The LSTM decoder in streaming ASR is a 2-layer RNN of size 1340 with 25M parameters. It uses a similar Word Piece model as the Conformer.
Table 2: Results comparing the performance of PersonaLM Retrieval Augmentation for (a) LSTM and (h) Transformer LMs with baselines and ablations (in red) for the Next Word Prediction task on WikiText- 103 and Famiing-21+22 datasets. PersonaLM achieves the lowest perplexity scores across all settings. indicates that the result statistically siguificant (S runs) based on Wilcoxon's 's signed rank test < 0.001).
ing user-specific text classification and generation tasks: (1) citation identification, (2) news cate- gorization (3) product rating prediction, (4) news headline generation, (5) scholarly title generation. Each data sample contains an input sequence to the model, a target output, and several text samples that encapsulate the profiles that can be employed for LLM personalization.
Baselines: Inspired by (Salemi et al., 2023), we compare SCAN retriever with strong baseline re- trievers for user-specific context selection: 1) Ran- dom, (2) BM25, and (3) Contriever.
Evaluation LLM Personalization: We evalu- different retrievers for personalized prompt con- struction in following settings: (a) Zero-shot LLM prompting: Retrieve top-k k most relevant user items from external corpus to append in prompts for GPT-
4.4 SCAN Retriever Experiments on LaMP
LaMP (Salemi et al., 2023) is a benchmark cor- pus to evaluate LM personalization on the follow-

---

Model
ANI Perplaity
WSB (1)
Peplocts ())
LIUMs
WIIK
Mold
Al
Ter.Con
Audk Midd Ouly (islerrss
Audia Madk LSIM
160.4
31.15
32.54
42223
12:23
Contorer Thankarr dlads Nede Gwog)
Contomer thaesurre LM (SM)
BM
53
112
1-3
2.5
2.5
6.1
37
4.5
Usifil Rasied
S1
PeoceaLMd LUP
Tiiifal Neyoe Coocammess Bariocal
606.
490.5
51.22
11.55
352.5
1364
1.23
1.5
Personal
Personal bt Conniver
50
833
6.5
6.
500
164
188
4.1
4."
30
471.2
51.15
3150
7.16
Pessenl ALM
463.1
11.1
113.9
7.81
Hemoeer
4802
31.33
2013
Nigios
Table 4: Performance comparison of PersO Retrieval Augmentation with baselines and ablations for personalized (a) streaming ASR and (b) non-streaming ASR on the UserLibri dataset. PersonaLM reduces the WER by 7-12% across all settings. indicates that the result statistically significant (5 runs) based on Wilcoxon's 's signed rank test (p < 0.001).
Asdia Makd ouly (inforrrr
Aude Modi
1525
INNSUM
31.27
131.
Unistd Negres Co-coamezze Nanioal
56.5
31,31
138.3
Pacsen MM IPP
963
31.16
1367
1.5
36.1
3114
125.7
183
3429
1L.11
137.99
SEAN Honiere
263
120
Negous
(a) LSTM LM
vectors selected amongst billions of stored contexts from a gigantic data store. Similar to (Drozdov et al., 2022), our experiments show that Unified N-gram Co-occurrence shows slight improvement over KNN-LM as n-grams are better at capturing highly domain-specific rare word patterns. How- ever, it still suffers from sub-optimal n-gram re- trievals from a mixture of domains the target probabilittis get averaged out when computed over the entire external corpus. Our proposed method achieves SOTA performance and improves the LM perplexity by a significant margin on WikiText-103 (57.1 - 61.8% w/o fine-tuning, 20.0 - 25.1% with fine-tuning), Earnings21+22 (46.4 - 47.2% woo fine-tuning, 11.4 12.6% with fine-tuning), AMI Meeting Corpus (71.6 71.9% w/o fine-tuning, 7.9 12.8% with fine tin-ttuning), and TED LIUMv3 (26.6 30.2% w'o fine-tuning, 3.8 4.7% with fine-tuning) over base LMs, demonstrating that con- textually matching query with most relevant do- mains via SCAN retriever module boosts retrieval performance which reinforces the next word pre- diction task. Replacing the SCAN retriever in Per- sonaLM with other baseline retrievers like Dense Passage Retriever (DPR) Contriever leads to de- graded performance. However, the performance does not decrease below kNN-LM or Unified N- gram Co-occurrence Retrieval methods, signifying the marginal benefit of domain-speciffe retrieval to augment LM predictions.
Made
Poploiiy 12)
AMI Meding Carges
WBK
Paplecls (1)
TED LIUMia
WER ())
Audio Ouly (Beforee'
Mockl Transforee
21143
442.3
Carra +VWG br
1982.
32.01
425.3
1579.
11.05
168.5
1.57
Unsied Negner Coeeocureeee Matiinal
53311
31.37
352.3
1.22
631.9
11.31
3574
Pessnal.0 M Kianniee
600.4
3001
N-grurs
31,27
3153
Aedia Mold Tronftemer
261
31.35
1167
Neaad Oudie
203
31.24
116.2
12.7
-2N-LM
26-1
31.99
1156
135
Uritel goor nonoorcce Retiona
26.1
31.14
1160
121
2333
31.00
1130
uau SPAN Rarlowe
25.6
1057
M wie NoNarit Negnrs
263
31,55
1123
(h) Transformer I.M
Table 3: Results comparing the performance of PersonaLM Retrieval Augmentation for (a) LSTM and (b) Transformer LMS with baselines and ahlations (in red) for the Next Word Prediction and Second-Pas ASR Re-scoring tasks on AMI Meeting Corpus and TED LJUMv3 datasets. Personal. achieves minimum perplexity WER both datasets. * indi- cates that the result is statistically significant (5 runs) based Wilcoxon's s signed rank test (p < 0.001).
3.5L and FlanTS-XXL (Chung et al., 2022); (b) Few-shot LM Fine-tuning: Fine-tuning F5 base (Chung et al., 2022) using top-k retrieved items from the user profile. More details on fine- tuning FlanTS-base in Appendix Sec. B.9
5 Results and Analysis
Perplexity Evaluation: Tables 2 and 3 compare the perplexity scores of the proposed PersonaLM retrieval augmentation against other baselines. We observe that the Neural Cache model (Li et al., 2020b) slightly improves over naive LM baselines but struggles due to its inability to handle long- range dependencies through pointer mechanism. Consistent with observations of Wang et al. (2023), KNN-LM (Khandelwal et al., 2020) reduces per- plexity by 5-10% but is still challenged by the non- parametric fuzzy nature of k-nearest Bert context
ASR Rescoring Analysis on ASAP dataset: Table 3 shows results of second-pass ASR rescoring on AMI Meetings and TED LIUMv3 datasets where our proposed approach improves WER relatively by ~ 5%. Retrieval-augmennte LMs when com- bined with the n-best hypotheses produced by the audio model lead to statistically significant WER reduction with respect to both kNN-LM and Per- sonaLM with Contriever baselines. Combining audio model and PersonaL M allows wins on tail

---

Monc
11x13-XXL
Cintnerr
Reriovrr Nen-pesortiiooo
Coumter
GPT-35
SCAN Rarisn
Netegenomalissa
BurTS-bpacc (lap-tuaed)
Ontrevs
SCAN Kemever
LAMP-L Renenaisal
Clustio Menification
Arromayy
0,522
0.65
0
0,510
0.701
0.715
0.522
0.531
0.766
1aMP A0: Persestiond
0.81
0.5%
6.008
0,991
0770
8333
NNws Cateorizatio
0.463
0471
0.484
0,504
0/48
LAMP
0.282
Product Rating
RMSE
0.668
0.884
0.565
1.100
0.64
0.571
LaMP-41: Pencnuliaal
BOUGE-1
0.164
0.192
0.211
0.133
0.100
0.172
0.158
0.201
0.212
Now Headine Grmestioon
ROUGFE-
0.149
0.178
0.187
0.118
0.143
0.155
0114
0.185
0.192
LLMP-SE: Penenalional
ROUGGE11
0445
0.47
0.475
0.395
0.398
0,419
0.44
0.433
0.470
Scaduay
ROUGEL
0430
0.424
0,322
0.425
Table 5: Performance comparison of zero-shot FS XX,LL, GPT-3.5, and few-shot fine-tuned F5an'5--baa base for personalized text classification and generation results on the eval set of LaMP dataset. For all metrics the higher the better, except for RMSE and MAE, Prompting LLMs with user-specific context selected by the SCAN retriever consistently reports the best performance.
words while avoiding losses on common word occurrences. Evaluation on UserLibri dataset (Breiner et al., 2022b) shows that PersonaLM im- proves WER for both streaming and non-streaming ASR. Compared fine-tuning the LM on the en- tire external personalized corpus (p13n LM), Per- sonaLM can selectively learn user-specific discrim- inative patterns in speech text and weigh it appro- priately for biasing the LM predictions.
Wistless AMI Cerpa
Wirdoou-SS - AMI Coepes
300
82
(a) LSTM
(b) Transformer
Figure 4: Plot of À (interpolation parameter) perplexity of PersonaLM with fine-tuned (a) LSTM, (b) Transformer LMs on WikiText- 103, Earnings-21+22
A Ablation Analysis: Tables 2-4 highlights in red show the ablation study for PersonaLM. We ob- serve that SCAN retriever is critical in all settings due its enhanced ability to learn enhanced dis- criminative document representations that help as- sign appropriate weights to external domains. Re- moving the k-nearest N-grams severely deteriorates the performance as the LM is no longer able to exploit the personalized n n-gram probability distri- bution from different domains. The severe per- formance drop in WER for speech datasets in the absence of either of the components underscores their significance for personalized ASR tasks.
AMI Corpus, and TED LJUMv3 datasets. Curves show convex characteristies with the optimal value of À varying with diffcrent settings.
Impact of Interpolation Parameter: Figure 4 shows the optimal value of interpolation parame- ter À varies in different settings. À vs perplexity curve shows convex characteristics for all variants of PersonaLM. Perplexity scores improve with in- creasing À as explicit memorization of rare word patterns mined from matching domains benefits the next word prediction task but starts to drop mono- tonically after reaching an inflection point.
Adaptation to Unseen Domains: Retrieval aug- mentation with fine-tuned LMs shows sustained relative gain of 5-18% across all settings despite having seen the same data during the fine-tuning stage. This observation validates our hypothesis that despite the benefits of transfer learning for out- of-domain generalization, explicit memorization is needed to effectively learn user-specific linguistic patterns not retained during fine-tuning.
Downstream Application of SCAN Retriever: Table 5 shows the application of different retriev- ers for improving user-personalized text genera- tion and classification on the LaMP dataset. We aim retrieve the most relevant user profiles that can be augmented with query context for zero-shot prompting or few-shot fine-tuning of LLMs. SCAN retriever outperforms both BM-25 and Contriever baselines and shows significant gains across dif- ferent metrics compared to pon-onesonll LMs across all subtasks. As opposed to earlier advances where retrieval augmentation ing LM training (Lewis et al., 2020; Guu et al., 2020), the core merits of our proposed SCAN re- triever are that it is extensible to any LM (LSTM, Transformer, GPT, etc), can scamlessly adapt to new users, enable in-context retrieval augmenta-
Qualitative Examples: Qualitative examples from ASAP, Wikitext, and UserLibri datasets in Table 6 along with model predictions. Personal.M M is able to able to correctly predict proper nouns, abbrevia- tions, and homonyms mistaken by fine-tuned LM and KNN-LM baselines, while also fixing the prob- lem of over-prediction of domain-specific frequent words commonly observed in Unified N-gram Co- occurrence Retrieval baseline.
performed dur-

---

Groaed Lish
Ese sanc La0
Uniice Si pran Netova
Tige
Mir
Seg
dreas ze singed
King
King
Engeliur dee
Roper
Wis
Musted
seacich Minsed
Proper
bediof
ofgge
loddol bell
bagiske
baae Whz
meem bedl
Proper
Wir
Toaks nny
Books le
Fack is
Teakss nny
hecnngee
Wir
Norts Trnay Milane
Norts Neceicas Treety Hands oter HSVV the aveatee
Narh Aoamious Trear Organiatioo uke coppect dex tevtaace
Narta Nuresion Towey Allanne
Norh Areeicas Treary Allane stmry conserncy for tntztee
Merccation Trin
200
contakmn
PREmY nou mniisaann
Aftroratio Tern
Table Qualitative examples: Ground truth, baseline predictions, PersonaLM predictions for a few samples from UserLibri and ASAP eval set. PersonaL. able to able correctly predict proper nouns, abbreviations, and homonyms mistaken by fine-tuned LM and kNN-LM baselines, while also fixing the problem of over-prediction comain--pecific frequent words commonly observed in Unified N-gram Co-occurrence Retrieval baseline.
tion without any LM-sppcific fine-tuning, and re- quires very small memory footprint with negligible computational overhead.
grounding mechanisms to cite what information was used for next-word prediction. Another major limitation of this method is that it ignores accented speech and speech that causes cold start problems for noisy speech samples wherein the contextual query is corrupted from the start. As a result, ASR rescoring unable take advantage of previously predicted utterances. This leads to exacerbating the ASR performance that was already suffering due to noise in the audio domain. For instance, the Conformer model personalized using PersonaLM for TED talks recorded in auditorium does not generalize well to political speeches conducted in open settings with background crowd noises.
6 Conclusion
We introduce PersonaLM retrieval augmentation for ASR personalization using SCAN retriever trained via group-contrastive learning to rank tex- tual documents from an external knowledge corpus based on their semantic similarity with the input query. We aggregate the probability distribution of the next word prediction by utilizing domain- specific n-gram word frequency representations weighted by the relative importance of the exter- nal domains to the input query. Experiments on our proposed ASAP benchmark and the UserLibri dataset show that our method achieves SOTA per- plexity and WER. We show that the SCAN retriever is also useful for in-context LLM augmentation for zero-shot prompting and few-shot fine-tuning.
Ethics Statement
We utilize the publicly available Wikitext, AMI Meetings corpus, TED-LIUM v3, Earnings-21, Earnings-22, UserLibri, and LaMP datasets for ASR and language modeling. Our curated bench- marks are composed of open-source and publicly available datasets repurposed for ASR and LM. Our proposed ASAP benchmark does not provide any new annotations but rather reformulates the data splits for our experiments. Our study on user domain personalization does not target any known individual, race, gender, topie, or ethnicity. Person- alization is limited to a user's linguistic character- istics. We do not utilize any PII at any step in our experiments.
7 Limitations
Through careful analysis of error cases, we found that there are two main types of prediction errors from the proposed model. First, our methodology is challenged by the phenomenon of contradictory information. As retrieval augmentation relies on external data that may be from varying sources, fact verification and rare word confounders in cer- tain cases can be confusing for our models. Our approach does not reconcile facts, names or out-of- domain patterns from varying information sources, but rather takes a frequentist approach due to the reliance on n-grams. For example, consider this query "Nigerian authorities report fresh aggres- sion on June 15, 2018, that led to the death of To complete the query, several external new article sources are ranked highly by our SCAN retriever. However, only few of them are relevant the date the query was executed, while some of them were later debunked as misinformation. Tackling this type of error requires fact-checking and document
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katic Milli- can, George Bm Van Den Driessche, Jean- -aptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
Theresa Breiner, Swaroop Ramaswamy, Ehsan Variani, Shefali Garg, Rajiv Mathews, Khe Chai Sim, Kilol Gupta, Mingqing Chen, and Lara McConnaughey.

---

2022a. Userlibri: A dataset for ast personalization using only text. arXiv preprint arXiv:2207.0011 000060
François Hernandez. Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. 2018. Ted- lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer:
Theresa Breiner. Swaroop Indra Ramaswany. Ehsan Variani, Shefali Garg, Rajiv Mathews, Khe Chai Sim, Kilol Gupta, Mingqing Chen, and Lara Me- Connaughey. 2022b. Userlibri: A dataset for asr
20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20. Springer.
personalization using only text. In Interspeech.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019a. Language modeling with deep trans- formers. In Interspeech.
Xie Chen, Xunying Liu, Mark JF Gales, and Philipp Woodland. 2015. Recurrent neural network language model training with noise contrastive estimation for speech recognition. In 2015 IEFE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 5411-5415, IEEE.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019b. Training language models for long-span cross-sentence evaluation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 419-426.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scal- ing cnrruction-finttune language models. ArXiv, abs/2210. 1416.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave 2021. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn. Rex., 2022.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the tree 2019 deep learning track. arXiv preprint arXiv:2003.07820.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retricval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing.
Nilaksh Das, Duen Horng Chau, Monica Sunkara, Sra- van Bodapati, Dhanush Bekal, and Katrin Kirchhoff. 2022. Listen, know and spell: khowledge-infused subword modeling for improving asr performance of oov named entities. In /CASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEF.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. 2022. Farnings-22: A practical benchmark for accents in the wild. arXiv preprint 2rX3v:2003.15533
Wessel Kraaij. Thomas Hain, Mike Lincoln, and Wil- fried Post. 2005. The ami meeting corpus.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel. et al. 2020. Retrieval-augmente for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9477
Andrew Drozdov, Shufan Wang, Razieh Rahimi, An- drew McCallum, Hamed Zamani, and Mohit lyyer. 2022. You can't pick your neighbors, or can you? when and how to rely on retrieval in the knn-lm. In Conference on Empirical Methods in Natural Lan- guage Processing.
generation
Bofang Li, Tao Liu, Zhe Zhao, Puwei Wang. and Xiaoy- ong Du. 2017. Neural ngg-ams. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 31.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017. Improving neural language models with continuous cache. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Pro- ceedings, ab./161..
Kc Li, Zhe Liu, Tianxing He, Hongzhao Huang, Fuchun Peng, Daniel Povey, and Sanjeev Khudanpur. 2020a. An empirical study of transformer- baased neural lan- guage model adaptation. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7934-7938. IEEE
Kelvin Guu, Kenton Lee, 7ora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.

---

Ke Li, Daniel Povey, and Sanjeev Khudampur. 2020b. Neural language modeling with implicit cache point- ers. In Interspeech.
Timo Schick and Hinrich Schütze. 2019. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In AAAI Con- ference on Artificial Intelligence.
Qi Liu, Dani Yogatama, and Phil Blunsom. 2022. Rela- tional memory-augmenter language models. Trans- actions of the Association for Computational Linguis- tics, 10:555-572.
Prashant Serai, Vishal Sunder, and Eric Lusser. 2022. Hallucination of speech recognition errors with sequence to sequence learning. IEEF/ACM Transactions on Audio, Speech, and Language Pro- cessing, 30:890-900.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-199 1919, On-
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching- Feng Ych, Julian Chan, Frank Zhang. Duc Le, and Mike Seltzer. 2021. Emformer: Efficient memory transformer hased acoustic model for low latency streaming speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Process- ing. ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021. IBEE.
line. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
G. Sun, C. Zhang, and P. C. Woodland. 2021. Trans- former language models with Istm-based cross- utterance information representation. ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7363- 7367.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cer- nocký, and Sanjeev Khudanpur: 2010. Recurrent neu- ral network based language model. In Interspeech, volume 2. pages 1045-1048. Makuhari.
Shufan Wang. Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha, and Mohit lyyer. 2023. Knn-Im does not improve open-ended text generation. ArXiv, abs/2335.11422
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: human generated machine read- ing comprehension dataset. choice, 2640:660.
Yuxin Wu and Kaiming He. 2018. Group normalization. International Journal of Computer Vision, 128:742- 755.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Hainan Xu, Ke Li, Yiming Wang, Jian Wang, Shiyin Kang, Xie Chen, Daniel Povey, and Sanjeev Khu- danpur. 2018. Neural network language modeling with letter-based features and importance sampling. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6109- 61 13. IEEE
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language mod- els as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 2463-2473. Association for Computational Linguistics.
Ori Ram, Yoav Levine, Itay Dalmedigos. Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown. and Yoav Shoham. 2023. In-context retrieval-augmennted lan- guage models. arXiv preprint arXiv:2302. 2000000000000
Miguel Del Rio, Natalie Delworth, Ryan Wester- man, Michelle Huang, Nishchal Bhandari, Joseph Palakapilly, Quinten McNamara, Joshua Dong, Piotr Zelasko, and Miguel Jette. 2021. Earnings-21. 21: A practical benchmark for asr the wild.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. Lamp: When large lan- guage models meet personalization. arXiv preprint arXiv:2304.1 1406.

---

A Appendix
B Reproducibility Checklist
B.1 Summary of paper's main claims
A.1 Limitations
The novel contributions of our work include:
Through careful analysis of enror cases, we found
PersonaLM retrieval augmentation for ASR personalization that leverages group-wise con- trastive loss to train Span Aggregated Group- Contrastive Neural (SCAN) retriever for ranking query-relvvant external domains/users and augments domain-distribuute k-nearest n- gram frequencies to improve LM predictions. ASAP a novel benchmark for ASR LM personalization consisting of three user- specific ASR tasks in the domains of meet- ings, TED talks, and financial conference calls, PersonaLM significantly outperforms strong baselines on ASAP benchmark, UserLibri, and Wikitext- 103 corpus by 10 16% per- plexity gain and
that there are two main types of prediction errors from the proposed model. First, our methodology is challenged by the phenomenon of contradictory information. As retrieval augmentation relies on external data that may be from varying sources, fact verification and rare word confounders in cer- tain cases can be confusing for our models. Our approach does not reconcile facts, names or out-of- domain patterns from varying information sources, but rather takes a frequentist approach due to the reliance on n-grams. For example, consider this query "Nigerian authorities report fresh aggres- sion on June 15, 2018, that led to the death of To complete the query, several external new article sources are ranked highly by our SCAN retriever. However, only few of them relevant the date the query was executed, while some of them were later debunked as misinformation. Tackling this type of error requires fact-checking and document grounding mechanisms to cite what information was used for next-word prediction. Another major limitation of this method is that it ignores accented speech and speech that causes cold start problems for noisy speech samples wherein the contextual query corrupted from the start. As result, ASR rescoring is unable to take advantage of previously predicted utterances. This leads to exacerbating the ASR performance that was already suffering due to noise in the audio domain. For instance, the Conformer model personalized using PersonaL.M for TED talks recorded in an auditorium does not generalize well political speeches conducted in open settings with background crowd noises.
8% WER reduction.
Downstream Application: SCAN retriever improves context retrieval in personalized text generation and classification via zero-shot prompting and few-shot fine-tuning of LLMs on LaMP corpus by 7 12%
B.2 Citation to creators of artifacts
We use four publicly available datasets for evalu- ation WikiText-103 (Merity et al., 2017), com- bined Earnings-21 Earnings-22 (Rio et al., 2021; Del Rio et al., 2022), AMT Meeting (Kraaij et al., 2005), and TED-LIUM v3 (Hernandez al., 2018). We use Librispeech (Panayotov et al., 2015) text corpus for LM pre-training. We prefer to combine both versions of Earnings Call datasets to make the corpus more domain-diverse. We use the UserLibri corpus to evaluate retrieval augmentation for per- sonalizing streaming and non-streaming ASR. We use the LaMP (Salemi et al., 2023) dataset for per- sonalized text generation and classification through retrieval augmentation with LLMs.
Potential Risks: Our models exploratory and academic nature and should not be used for real- world financil/politicaalhattttaa purposes with- out extensive investigations into their shortcom- ings//anddomnesshiases.. Unhandled Cases: The current work is limited to the English language and would need suitable tools in other languages to pro- cess audio, LM tokenization, vocabulary, and pre- trained multilingual language models. Moreover, our method has been tested on limited domains of Wikipedia text, professional meetings, recorded speeches, and financial earning calls. Applications to life-critical scenarios such as hcalthcare, public safety, and law will need further investigation.
WikiTexx-1003 2
Earnings-21 Datatset
Farnings-22 D
AMI Meeting Corpus
hhupp://boogsslesforreeeee callesfoeeairescarch.oomm www.:hhhhh.cheeeeeeeeee.eeeeeee
term hawprnddeeyy-laguugg-mmmmdllllle
latasetstree//main//annnngg www//revooocooo//peccc...
"hupss//githu..com/// www./ewdotcoom///ggeeooo sppechh
aatastt/tree/massse//ummnggg
https:://grups.. inf. ed uk/ami/sorpuss

---

TED-LIUM v3 Dataset
B.8 Total computational budget and computing infrastructure:
Librispeech Dataset?
We performed training and inference of the models on industrial strength CPU and clusters of multi- ple 32GB V100 GPUs. The model takes between 12-36 hours to train on either of the four datasets. Inference/ Fine-tuning time varies from 3-7 hours per dataset.
userLibri Dataset
LaMP Dataset
B.3
License and terms for use of data artifacts
All the datasets are publicly available and free to use for research purposes.
B.9 Experimental setup, hyperparameter search, and best-found values:
B.4 Intended use of data artifacts:
The intended use of text and speech datasets is to improve language modeling and ASR re-scoring. Enhancements in speech recognition and general NLP models can increase the accessibility of AT tools and open new pathways to deploy Al for so- cial good.
Hyper-parameters for experiments tuned on the respective out-of-distribution evaluation sets to find the best configurations for different datasets. We detail the training setup as follows:
Training Setup for PersonaLM Retriever: We experiment with range of hyperparameters in PersonaLM such as: size of hidden layers in Fully Connected Layer [64, 128, 256), dropout [le-- 2e 0.5.0, learning rate À cay E f1e-6, C 16, 128, 256, 512) and epochs (< 25). We use a pre-trained BERT model and further train it using contrastive learning loss.
B.5
Steps taken to protect/anonymize names, identities of individual people, or offensive content:
5, 5e (oee-5],, 5}, weight de- - 3}. batch size
We do not any identifiable user data for any experiments. All datasets already come with PII data redacted prior to their public release.
B.6
Coverage of domains, languages, linguistic phenomena, demographic groups represented in data:
Pre-training Language Models: For all experi- mental settings, we use an L.TTM model of lay- ers with a 300-dimension embedding layer and a hidden dimension 768. The Transformer model consists of 4 layers of encoder-decoder with 12 heads, 128 dimensions hidden representations, Layer Norm input dimensions of 768, attention dropout of 0.2, and ferw-forward layer of 3072 dimensions. We batch size of 256 and train till 25 epochs. We use learning rate of le 4, a fixed learning rate scheduler, force anneal 26, learning rate shrink of 0.8, and a Gelu acti- vation function. We use Adam optimizer with eps of le - 8, weight decay of le 6, and betas of (0.9, 0.999) to compute cross-entropy loss as word level. We select the model checkpoint with the least perplexity the Librispeech validation set. Fine-tuning Language Models: LSTM and Trans- former models were fine-tuned on the out-of- domain train set with a batch size of 256 for 25 epochs. We select the model checkpoint with the least perplexity the out-of-domain validation set.
Our work performs next word prediction and ASR second pass re-scoring on data from Wikipedia, financial earning calls, meeting recordings, and speeches in the English language. Adaptation to other languages may need appropriate processing.
B.7
Data statisties and Processing
The data statistics are given in Table 7. For each text sentence/utterance, all alphabets were lower- case, numbers were converted into their respective word forms, and punctuation was removed. To experiment with the out-of-domaal generalization ability of the proposed approach, we reformulated existing datasets to identify explicit domains. For each dataset, we combined the train/val/test por- tions and considered all sentences/utterances in each Wikipedia page/call/recordigg as part of par- ticular domain. We then split sentences within the same domain in the ratio of 70: 10:20 form new train/val/test splits.
"htps://www.openssr..ogg
ASR Audio Models: For generating ASR n-best hypotheses, we use a pre-trained RNN-T model with the Emformer encoder (Shi et al., 2021),
"hups://www..k wwwpw/wwwwwwwww.cc.p...........p wwt.ggll..o//mttttteeeeee comm/lannnetssoooglleeeeiiiii

---

Dataset
Train
Val
Test
Vocab Size
Domain
# Domains
Bamnings-11+22
49.6K
7.1K
14.2K
20K
Earning Call
169
AMI Meeting Corpus
17.1K
2.7K
5.8K
11K
Meeting Recording
135
TED-LIUM v3
188.9K
26.6K
9.3K
46K
TED Talk
2351
Wikitext-103
2M
300K
10K
200K
Wikipedia Page
30k
UserLibri
6.3M
700K
10K
10K
Books
107
Table 7: Data stats of ASAP, UserLibri, and WikiText-103.
LSTM predictor, and joiner with 80M param- eters.
ment: We utilize FlanT55-bas model for few- shot fine- fine-tuning experiments with a learning rate experimented between {le 2, 5,
PersonaLM Retrieval Augmentation: Probabil- ity distribution obtained from bi-gram frequencies for each vocabulary word in the co-occurrence ma- trices was normalized. Trainable parameters in the Neural Retrieval module were frozen during inference and used encode query and domain contexts. For each model, we report results with minimal perplexity by iterating the interpolation paramcter À between
5, 4e 5, 5e - 5), weight decay w E {le
le 4, le 3), batch size b E {4, 8, 16, 32, 64, 128, 256, 512} and epochs (S 25) and Adam optimizer. We utilize a linear warmup scheduler for 5% of the total training steps and train for a maximum of 20 epochs. The maximum input prompt length is 512 tokens.
6,
increments 0.1. Loss Functions: PersonaLM retricval augmenta- tion uses both group-wise contrastive loss and MLM loss to train the neural retriever. For pre- training and fine-tuning the LMs, we use cross entropy loss over target words in the scquence.
B.10 Implementation Software and Packages
We implemented our solution in Python 3.6 using the PyTorch framework. We used the following libraries and modules:
Huggingface's implementation for BERT transformers. 10
Model Parameters: The audio LSTM model has around 80M parameters. Transformer LM has ap- prox 247M parameters while the LSTM LM has around 6.5M parameters. The conformer model for streaming and non-streaming ASR in UserLibri dataset has 86M parameters.
FAISS Library !! for fast kNN retrieval.
Scipy 12 for scientific computations
NLTK 13 for sentence processing
Evaluation Metrics: We utilize word-level per- plexity scores to evaluate language model perfor- mance for next-word prediction. We also report Word Error Rate (WER) for ASR sccond- -aass-pssstte re- scoring in speech datasets.
OpenAl 14 for GPT-3.5 turbo
B.11 Model Decisions
We choose optimal k in the kNN algorithm to be 400. We experiment with bigrams, trigrams, and 4 4-grams. The optimal weightage of the n-grams is [00.6 0.7), (0.10 0.20), (0.05 0.10)]. We choose the maximum value of T 20 from each document for training the SCAN retriever.
Data Preprocessing: For each text sen- tence/utterance in the ASAP benchmark, all alphabets were lower-cased, numbers were converted into their respective word forms, and punctuation was removed.
Pre-training SCAN Retreiver: The maximum query and span lengths for the ranking datasets are set to 32 and 128, respectively. We experi- ment the learning rate between {le - -5, 2e
B.12 LLM Prompts
We utilize the prompts given by (Salemi al., 2023) in the paper Appendix for retrieval augmen- tation experiments. This was done to ensure a straightforward comparison with baseline BM25
5, 3e - 5, 4e 5, 5e - 5), weight decay w E {le - 6, 5, le 4, le - 3}. batch size
b (1,8, 16, 32, 64, 128, 256, 512) and epochs (< 25). and Adam optimizer with a linear warm-up for the first 10% steps.
"htts://hugggingface.co coo/
"btts:://gittuu.com// hww.:///ww....ooooooooooeeeeeaaa
hhuups://scppyorr/
19 "htps://www.lllk.orgg
Fine-tuning FlanTS-base for LaMP Experi-

---

and Contriever. The retrieved user profiles were added the prompt context while ensuring the maximum context length is respected.
B.13 Instructions given to participants annotators
We do not annotate any new data.
B.14
Annotator recruitment and payment
We do not annotate any new data.
B.15 Consent for data annotation
We do not annotate any new data.
B.16
Protocol approval by ethics review board
We do not annotate any data.
B.17
Demographic and geographic characteristies of the annotator
population
We do annotate any new data.


Reviews:
[{"review_id": "6dvHYhyqJS", "reviewer": "Reviewer_n2GV", "summary": "", "questions": "It's not clear from the paper what is defined as a domain for each of the sets in table 1. Is it simply different speakers? The use of speaker/domain throughout the paper is a bit confusing.", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and constructive suggestions. \n\n***Domains for each set in Table-1***: Our proposed method - DANK-LM is applicable to a variety of different datasets with varying “domains” meaning different speakers/topics/categories. For example, the recordings in the AMI Meeting corpus are split based on speaker IDs. Domains in Earnings-21+22 data correspond to different companies. A specific domain in the TED-LIUM v3 dataset refers to a TED talk speaker. Domains in Wikitext-103 correspond to individual Wikipedia pages. We will clarify these details in Table 1 and add them in Section 4.2 to make it easy for the readers. We will also release the data splits for enhanced reproducibility. \n\n***Improvements in Sec-4.2***: We will expand Section-4.2 to clarify the data split, domain selection, and any domain-specific preprocessing steps by utilizing the additional page provided to authors in the camera-ready version.\n\n***Missing Year in Citation Typo***: We thank the reviewer for pointing out the typo mistake on line 375 - \"Improving neural language models with a continuous cache.\" The correct citation is “Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In 5th International Conference on Learning 740 Representations, ICLR 2017, Toulon, France”. We will add the missing year in the citation in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method appears novel. The results are convincing on multiple datasets.", "reasons_to_reject": "Section 4.2 is not easy to understand.", "questions_to_authors": "It's not clear from the paper what is defined as a domain for each of the sets in table 1. Is it simply different speakers? The use of speaker/domain throughout the paper is a bit confusing.", "paper_topic_and_main_contributions": "This work proposed a contextual biasing method based on domain specific n-grams, which are interpolated with a neural LM. In this method, a query span is required to obtain domain importance weights. This method is applicable to asr rescoring.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"Improving neural language models with a continuous cache.\" citation is missing a year."}, "extra_text": {}}, {"review_id": "7CmGQFq9N3", "reviewer": "Reviewer_FqGg", "summary": "", "questions": "1. For SCAN retriever, it seems to be equivalent to extract embeddings of random texts for each domains and contrast them with the current query embedding. Would it be more straightforward to directly encode each sentence with a shared encoder and apply a contrastive loss similar in SimCSE?\n2. Could you explain again how you calculate the N-gram co-occurence matrix? Do you mean for each vocabulary word, you query top-k n-grams in a specific domain based on the vicinity of Bert embeddings within n-grams?\n3. What are the domains that are adopted for k-NN based retrieval in each dataset?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We clarify the questions as follows:\n\n***Q-1 SCAN Retriever Embedding***: The SCAN retriever is a significant improvement over the Dense Passage Retriever style shared encoder learned through SimCSE in terms of its training methodology. The core novelties that differentiate it from SimCSE-based contrastive retriever are as follows:\n\n1. SCAN retriever is trained to contrast between spans from different domains by encoding the spans in the Transformer input layer itself (see Sec-3.1 Line 205-210). As a result, it can take maximum advantage of the Transformer’s self-attention mechanism to discriminate between spans from unrelated domains.\n\n2. The group contrastive loss requires fewer iteration steps to converge as the model can encode multiple spans from both positive and negative domains (See Sec-3.1 Line 220-225) as opposed to SimCSE loss that relies on a single positive span from the batch of negative spans.\n\n3. The multi-granular encoding strategy helps SCAN to learn hierarchical concepts related to specific words, entities, phrases, and topics information (See Sec-3.1 Line 180-187). Shared encoders with SimCSE loss rely on averaging Bert vectors of the entire domain during training which weakens the representation power of the retriever.\n\nQuantitative results (See Tables 1-3) show how using DANK-LM with Dense Passage Retriever leads to suboptimal performance across different datasets. SCAN retriever is ~ 5% better than DPR across different settings.\n\n***Q-2 Construction of N-gram Frequency Matrix***: The N-gram frequency matric is a matrix where a cell (i,j) corresponds to the normalized frequency of observing word w_i ∈ V followed by a sequence of n-1 words {${w_j-n, w_j-n+1,..., w_j}$} in a specific domain. N-gram frequency matrices can be sparse and constructing large matrices is memory-inefficient (as mentioned in Sec-3.3 Line 252). Restricting the query n-grams to only exact matches leads to a loss of information due to ignoring semantically similar n-grams. Hence, we build domain-specific n-gram datastores that store key-value pairs of ((n-gram Bert vector, next word), co-occurrence frequency)). For each vocabulary word, we query top-k n-grams in a specific domain based on the vicinity of Bert embeddings of the n-gram within the datastore to augment with LM predictions. We will make this description clearer in Sec-3.3 of the camera-ready version to enhance the understanding of the readers.\n\n***Q-3 Domains in KNN-based Retrieval***: Our method is extensible to different types of domains (users/topics/categories) as highlighted in Table 1 and Sec-4.2. Domains in the AMI Meeting corpus are formed based on speaker IDs. Domains in Earnings-21+22 data correspond to different companies. A specific domain in the TED-LIUM v3 dataset refers to a particular TED talk speaker. Domains in Wikitext-103 correspond to individual Wikipedia pages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new group-contrastive neural retriever for ranking query-relevant external domains which are beneficial to improve LM prediction.\n2. The experiments are extensive to both show how the performance of the proposed framework is compared with other competitive baselines and how modified next-word probability distribution may affect the generation quality.", "reasons_to_reject": "1. Some experimental details should be further clarified to support the claim of the proposed method.\n2. The narration could be better improved where some sections are a little bit hard to follow such as Section 3.3.", "questions_to_authors": "1. For SCAN retriever, it seems to be equivalent to extract embeddings of random texts for each domains and contrast them with the current query embedding. Would it be more straightforward to directly encode each sentence with a shared encoder and apply a contrastive loss similar in SimCSE?\n2. Could you explain again how you calculate the N-gram co-occurence matrix? Do you mean for each vocabulary word, you query top-k n-grams in a specific domain based on the vicinity of Bert embeddings within n-grams?\n3. What are the domains that are adopted for k-NN based retrieval in each dataset?", "paper_topic_and_main_contributions": "The paper proposes a retrieval-based approach to recharacterize the next-word probability in LMs as the probability of which domain the query is in and the vocabulary probability conditioned on the relevance of the underlying domains. For the former, they design a neural retriever to calculate embeddings for the query and each span of domains, while the latter, they  design a n-gram co-occurrence matrix to learn the word frequency within domains. The experiments show that the perplexity and word error rate are both improved with the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kRBbgo2Wrl", "reviewer": "Reviewer_cwNA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his thoughtful comments and suggestions. \n\n***Q-1. Use of kNN-LM for LM personalization:*** kNN-LM can retrieve and augment relevant context with LM predictions, thus potentially useful for LM personalization. We utilized kNN-LM and Neural Cache (see Sec-4.3 Baselines) due to their capability to memorize context which can help lower perplexity for user/domain-specific text, thereby improving ASR word error rate. However, we agree that kNN-LM is not a direct application for user personalization tasks. We would incorporate the reviewer's suggestion to rephrase the motivation paragraph to enhance readability.\n\n***Q-2. Typo in line 164:*** K in the equation on line 164 refers to “K” domains. Yes, we are marginalizing retrieved documents. We will fix the typo in Line 163 - “domains ($d_1, d_2, · · ·, d_N $)” to “domains ($d_1, d_2, · · ·, d_K$)”.  We thank the author for suggestions to improve the mathematical intuition of LM conditional probability based on the marginalization of retrieved domains. We will refer to the RAG paper [1] to make the mathematical details more sound and intuitive in the camera-ready version of the paper.\n\n[1] Lewis, Patrick, et al. \"Retrieval-augmented generation for knowledge-intensive nlp tasks.\" Advances in Neural Information Processing Systems 33 (2020): 9459-9474.\n\n***Q-3.Definition of domain:***  Domains in our work refer to the categories that have a distribution different from the base data used to train the base model. It can mean different users/topics/call recordings based on the dataset. Domains in the AMI Meeting corpus are formed based on speaker IDs. Domains in Earnings-21+22 data correspond to different companies. A specific domain in the TED-LIUM v3 dataset refers to a particular user. Domains in Wikitext-103 correspond to individual Wikipedia pages. We will clarify all these details in Table 1 as well as Section 4.2 to make it easy for the readers. Domains are represented by a small letter “$d$” in our methodology. “$d^+$” and “$d^-$” refer to matching and contrasting domains with respect to the domain of the query. We will ensure that all mentions of domains have a consistent notation throughout the paper in the final version. Line 155 defines query as a sequence of words “$q = (w_1, · · ·, w_{t−1})$” for which the next word “$w_t$” is to be predicted.\n\n***Q-4. Position encoding in SCAN retriever:*** We utilize BERT to initialize the SCAN retriever as mentioned in Sec-4.1. Hence the SCAN architecture uses a learnable position embedding. We use [SEP] tokens to identify the representation of the spans in the input sequence. While training the SCAN retriever, we train the Transformer model with permutations of span order. Hence, the spans from positive domains are not necessary to always precede the spans from negative domains. Similarly, the spans from an individual domain are not necessitated to be always together in the sequence. This ensures the learned span representations are robust in terms of their relative order. We will clarify this more in Sec-4.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- One of the reasons would be the novelty of the idea and observed improvements.\n- The paper is well written in the latter half.", "reasons_to_reject": "I am merging reasons to reject and clarifications in one place:\n\n1) L061-064: \"However, these methods give subpar performance as they do not retrieve relevant domains/users prior to context selection from billions of candidates.\"-I am not sure if kNN-LM and follow-up works would even have that issue as they are not meant to solve the user personalization task. kNN-LM was proposed primarily to enhance the memorization capabilities of LM to generalize better i.e. lower perplexity. So I would suggest rephrasing this line to enhance readability. The problem is clear but the way it is motivated does not look very appropriate (by reading paragraph L058-068).\n\n2) What is K in the equation in line 164? Are you marginalizing retrieved documents? I would prefer authors refer to the RAG paper to make the mathematical details more sound and intuitive.\n\n3) Until after section 3, I am still unclear about the definition of domain, I think the notations d, D, Domain_1, are confusing, leading to a confusing methodology. I could not find the definition of the query as well, where do you obtain w1...w(t-1) from?\n\n4) Do you use positional encoding in SCAN retriever (encoder) input? If no---then in figure-3 Domain_1, Domain_2 positions would not matter much (which is good) however, how do you feed word position information? If yes---wouldn't the Domain_i position will impact its representation?\n\nI think the paper can be significantly improved in the explanation of methodology and motivation. Definitions of some basic yet crucial terms are missing. Hoping to get more clarification on the methodology in the author's response.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach to improve language model next-word prediction performance (lower perplexities) for better ASR personalization. The main hypothesis (idea) is that n-gram patterns are domain/user-specific, and augmenting LM predictions with n-gram probabilities from a subset of query-relevant users/domains may lead to better personalization. The paper shows a significant drop in LM perplexities and improvement in ASR performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Originality:**

This paper describes a novel method for domain / user personalization via language model adaptation using a domain encoder trained with a contrastive loss.

**Signifciance:**

Personalizing language models to domains/users, especially in order to accurately recognize proper nouns and other domain specific vocabulary is very important. This work describes a method that achieves impressive performance improvement by conditioning on cached n-gram statistics from detected close domains.

**Clarity:** 

The consensus appears to be that the paper is somewhat difficult to follow. In general the explanation of methodology and motivation could be improved, including definitions of some basic yet crucial terms which are missing. Furthermore, Section 4.2 difficult to understand given the diverse datasets and task construction described, and much of Section 3.3, which one reviewer found difficult to follow, might have benefited from description in algorithmic, or mathematical terms rather than a purely textual description.

**Pros**
   - Addresses a relevant and interesting problem
   - Novel model
   - Very extensive experiments
   - Thoroughly described training and decoding parameters and experimental setup
   - Impressive results on multiple domains

**Cons**
   - A very complicated experimental setup which may be difficult to reproduce, in spite of the fact that it is well described. 
   - For instance the ASAP task is well defined, but to reproduce results would require recreating the dataset setup from scratch as it is not released as a part of the paper,  nor is the code-base to reproduce results
   - The paper, as currently written, could significantly improve the narration, motivation and explanation of the data and experimental setup
Decision: Accept-Findings