Meta-Learning Online Adaptation of Language Models
Authors: Nathan Zixia Hu
Eric Mitchell
Christopher D Manning
Chelsea Finn
Venue: EMNLP 2023 Main (2023)
Keywords: meta-learning
question-answering
online learning
knowledge
adaptation
TL;DR: CaMeLS improves knowledge retention during online fine-tuning of language models by meta-learning how to reweight token losses.

Abstract: Large language models encode impressively broad world knowledge in their parameters. However, the knowledge in static language models falls out of date, limiting the model’s effective “shelf life.” While online fine-tuning can reduce this degradation, we find that naively fine-tuning on a stream of documents leads to a low level of information uptake. We hypothesize that online fine-tuning does not sufficiently attend to important information. That is, the gradient signal from important tokens representing factual information is drowned out by the gradient from inherently noisy tokens, suggesting that a dynamic, context-aware learning rate may be beneficial. We therefore propose learning which tokens to upweight. We meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of-date base question-answering model’s ability to answer questions about a document after a single weighted gradient step. We call this approach Context-aware Meta-learned Loss Scaling (CaMeLS). Across three different distributions of documents, our experiments find that CaMeLS provides substantially improved information uptake on streams of thousands of documents compared with standard fine-tuning and baseline heuristics for reweighting token losses.

Meta-Learning Online Adaptation of Language Models
Nathan Hu*
Eric Mitchell*
Christopher D. Manning
Chelsea Finn
Stanford University {zixia314, em7 )mm))@tttfoodeed
Abstract
IA sedical device for the early detection and renitoring Enterprise Iretand's Student Intresceneur Awerds 2020. Neurobell, which was developed by University tollegge Cork diagnoses of
Unweighted per-token NILL gredient norms
Large language models encode impressively broad world knowledge in their parameters. However, the knowledge in static language models falls out date, limiting the model's effective "shelf life." While online fine-tuning can reduce this degradation, we find that naively fine-tuning on a stream of documents leads to low level of information uptake. We hypothesize that online fine-tuning does sufficiently attend important informa- tion. That is, the gradient signal from impor- lant tokens representing factual information is drowned out by the gradient from inher- ently noisy tokens, suggesting that a dynamic, context-awer aware learning rate may be beneficial. We therefore propose learning which tokens to upweight. We meta-train a small, autoregres- sive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of- date base question-answering model's ability to answer questions about document after a single weighted gradient step. We call this approach Context-awart Meta-learned Loss Scaling (CaMeL.S). Across three different dis- tributions of documents, our experiments find that CaMeLSS provides substantially improved information uptake on streams of thousands of documents compared with standard fine-tuning and baseline heuristics for reweighting token
Mark O'sutlivon;
atnornal activity faster and with greater accurary Mark won E10 8000 cash prize fund, as [...]
X
Brain injurtes resborns has the top prize Citterprise Iretand' s Student Dntresceneur Awards 2020. Neurobest,
Pen-toknn
CaMeLS weights
University Cellege Cork
studont D'5UE, aims holp mtagnoses
atnornal brain activity faster and withigreaten accuracy Mark won ELB cash prize fund, as [...]
K sedical desice for The BETTY detection and ronitoring or bratn Ancurses to NEVDOERE has ven the tog prize at Nourobill,
Pen-token CaMeLS
Awards was developed by bylierryesss College Istucont MarK OESULLAVMD, aims to holp the GLoonoses of Intnormal Brider OCCENITY faster and WITh accunary
gradient norms
Figure The proposed method CaMeLS learns to rescale the per-token online loss, sparsifying the fine-tuning gradients to emphasize informative timesteps. The middle row shows the weights output by CaMeLS, The top and bottom rows show raw and weighted per-token gradient norms, respectively.
mitigating the staleness issue, even very large lan- guage models often fail to correctly update their memorized predictions when presented with coun- terfactual retrieved information (Longpre et al.. 2021; Li al., 2022; Si et al., 2023). Moreover, purely parametric language models are uniquely suited for edge computing due their compact size (relative to a large retrieval index) and simplic- ity of inference (Gerganov, 2023). Recent work has thus considered variants of online fine-tuning on a stream of documents to efficiently perform direct updates to the knowledge inside of large language model (Lazaridou et al., 2021; Jang et al., 2022).
losses.
1 Introduction
Large language models learn impressively broad world knowledge through large-scale unsupervised pre-training, which they can leverage for a wide variety of downstream tasks (Brown et al., 2020; Chowdhery et al., 2022; Bubeck et al., 2023). How- ever, large language models are typically static ar- tifacts, and the world changes, the knowledge encoded in their parameters becomes stale. While retrieval aagmenel- models are one approach to Equal contribution.
Ideally, we could simply fine-tune a language model on an online stream of documents, and the information contained in those documents would be readily available for the model to use in variety of downstream tasks such answering questions about the information in the documents. Unfortu- nately, we find that in this online adaptation setting, Ene-tuning with a well-tuned learning rate leads to a nearly negligible improvement in a question-

---

Unsupervised
Unsupervised Online Adaptation
Downstream Task Evaluation
Pre-training
12 say 2813 President Volodengr zetersky at resetute susport and reapons shippents even lusstar [-]
Financial corkets are bett ra that Congress
Japanese compary
white house will strike
lapace inst contact
sceents
Who le negriiting with Congresa"
Who ettempted landing Wedhesday?
sefore touchdons os Ete
uncertalinty alone is baving comsequences [-]
Lhe visstor had
apperentiy fatlec L-]
What la beina promitert Ulcalee?
X2
X3
2016
2022
Update
Updato
8,
Update
0g
ON
Figure 2: We study the setting of language model being adapted unsupervised (without annotation of important tokens) online stream of documents and being later evaluated on queries (e.g.. questions) about those documents. Downstream inputs are not provided during the adaptation phase, requiring the model integrate as much information as possible about the documents.
answering model's ability to answer questions re- lating to the stream documents. We hypothesize that naive fine-tuning is not effective in the online adaptation setting because the negative log likeli- hood (NL.) loss does not accurately reflect the im- portance of token. That is, tokens containing im- portant factual information may receive relatively small NLL loss and therefore small fine-tuning gradient. For example, consider the NLL of the word Rishi and the word Reports in the phrase The UK Prime Minister is Rishi Sunak. Reports suggest for a slightly out-of-date language model. Be- cause Rishi Sunak was well-known politician be- fore becoming Prime Minister, model may place reasonably high probability mass on his name (even if other completions are higher probability). On the other hand, 'Reports' will invariably receive low probability, because the distribution over the first word in a sentence unavoidably high entropy.
ability answer questions about the contents of the surrounding document.
Using this notion a token's 's utility for online learning, we propose Context-aware Meta-learned Loss Scaling (CaMeLS), an approach to online adaptation that meta-trains an importance weight- ing model to identify such tokens in document. Given a dataset of documents and queries, we meta-leamiing loss train our weighting model: first, in an 'inner loop,' we update a base model (a proxy for the model we will update at test time) us- ing the gradient of NLL of the document, weighted by the outputs of the importance weighting model. Next, in the 'outer loop', the loss is computed by evaluating the updated base model's performance on the corresponding query. This outer loss is used to updated the parameters of the importance weight- ing model. During online fine-tuning on stream of documents, we simply re-weight the online loss using the amportance-weightiingg model's 's output.
This hypothesis suggests that we can improve upon online adaptation by only fine tuning on a subset of tokens which are most likely to lead to useful updates. One natural approach to identify such factual tokens is through salient spans (Guu et al., 2020). Another common technique used weight words it via TF-IDF scores (Salton and McGill, 1986). We find that fine-tunin fine-tuning while using these heuristics does improve information uptake. However, is unclear if such heuristic choices are optimal. As an alternative, we explore a method for learning per-token importance weights corre- sponding to the utility of fine-tuming on that token. However, such utility is difficult to define, and even with suitable definition, dense per-token annota- tions of utility are extremely time-consuming to collect. We thus select definition of utility that en- ables using distant supervision of the utility each token: a high utility token is one whose fine-tuning gradient improves a question question-answering model's
Although the process used to train CaMeLS uses a proxy model (i.e., a stand-in for the model we will update at test time), one might hope that the impor- tance of tokens would be independent of the model used for inner loop updates; to a significant degree, we intuit that the importance of a token should be an innate trait of underlying text. Indeed, we find that the meta-learned importance weights gen- eralize across models; for each dataset, we meta- train our importance weighting model once using DistilGPT-2 (Sanh et al., 2019) as the base model and successfully use these weighting model with- out modification to update GPT-J 6B (Wang and Komatsuzaki, 2021). Across three online adap- tation benchmarks based on streams of news and Wikipedia articles, CaMeLS substantially improves knowledge acquisition over naive fine-tuning as well as salient span and TF-IDF based baselines.

---

2
Related Work
work. Other methods for dynamically updating the knowledge in parametric language models develop specialized techniques, called model editors, de- signed to make targeted edits to individual facts (Sinitsin et al., 2020; Mitchell et al., 2021; Meng et al., 2022) or behaviors (Mitchell et al., 2022). However, model editors assume access to annota- tions of the tokens or facts that must be updated; in this work, study the problem of learning which tokens in an unlabeled sequence of documents are important.
Adapting to new data or task distributions typi- cally studied in the context of continual or lifelong learning (Thrun and Mitchell, 1995; Mitchell et al., 2018). Continual learning in deep networks in- volves the challenge of simultaneously avoiding catastrophic forgetting (McCloskey and Cohen, 1989), the process under which neural network's performance on old tasks or data dramatically degraded by the process of learning new informa- tion, while maintaining plasticity (Dohare et al., 2022), or the ability to adapt to the latest change in the data distribution, even after many changes have already been experienced. While most work in continual learning considers sequences of super- vised data (Kirkpatrick et al., 2016; Lopez-Paz and Ranzato, 2017; Shin al., 2017: Chaudhry al., 2019), some work also studies continual few-shot (Ren et al., 2021) or unsupervised learning (Rao et al., 2019; Madaan et al., 2022), which closer to the setting in this paper. However, these works typically focus on streams of visual data.
3 Meta-Learning Improved Online Adaptation of Large Language Models
Given an out-of-date language model and stream of recent documents, we aim to update the model such that it effectively answers typical queries about the documents in the stream. By focusing only on retaining knowledge relevant to the 'typi- cal' queries, we avoid the need to completely mem- orize the documents, making the problem tractable. We study cally, as the question- answer format makes assess- ing model's knowledge straightforward. In this section, we formalize this problem setting and then describe approach to this setting, Context-aware Meta-learned Loss Scaling.
(QA) models specifi-
Dynamic, or streaming, language models were first considered in the context of n- n-gram language models, combining a cache of recently-used words to update the predictive probabilities of a tri-gram model (Kuhn, 1988; Jelinek et al., 1991; Osborne et al., 2014). Later work describes online EM- based algorithms for efficiently updating n-gram models (Yogatama al., 2014). Other studies in- vestigate the evolution of decontextualized word embeddings over as a result of temporal shifts in the use of language (Kulkarni et al., 2015; Hamil- ton et al., 2016) or the use of vector memories to store recent information when training recur- rent neural networks online (Rei, 2015). More recently, several studies have explored methods for updating large neural language models, typ- ically through online fine-tuning on stream of documents (Lazaridou al., 2021) with architec- tural constraints (Jang et al., 2022) or explicit con- ditioning on time (Dhingra et al., 2022) used as strategies to reduce forgetting of old information. Clark et al. (2022) use meta- meta-learming to reduce the compute requirements of online fine-tuning, How- ever, recent work suggests that while increasing the size of language models may largely mitigate the problem of forgetting old information (Driess et al., 2023), improving the efficiency of acqui- sition of new knowledge is still a challenge, and this problem is therefore the focus of the present
3.1 Unsupervised Online Adaptation
We consider setting in which an out-of-date model freas is updated with an online stream? of recent documents Drest = (w;), ultimately produc- ing an updated model for The updated model fo then evaluated with a set of queries Qiest = (4:) with labels Yeett = {yi}. where the the ith query is drawn from a distribution of queries relating to ith document: 4s Wi ~ p(g. v.) For example, Ji may be a question about some information in document z; and y/ the answer to that question implied by the document. Crucially, when using Diest update fr we do not have access Quest Thus, our methodology for updating for.. must be broad rather than query specific. In order to make this problem tractable (i.e., not requiring complete memorization of the document stream), we assume that we have an additional corpus of documents Diran and corresponding query samples Quaan and labels Yuain generated by a similar gener- ative process to Quest, YusI. This training set enables
2 Deu is typically an online stream of documents, but could he an arbitrary ordering over a static collection of documents.

---

Japasese company Ispace spacecraft asworis fore kednesday, say'se 's. the ereation had rosarently
locality example The from Dioc. Dise is a dataset of unlabeled text representing the distribution over which we want the base model's behavior to re- main generally unchanged. For all experiments, we use the OpenWebText dataset (Gokaslan et al., 2019) as Dioc. Let Ohase denote the parameters of the proxy base model at the start of the episode. The update to the weighting model involves three steps: 1) computing the weights for the training document, 2) updating the small proxy base model on the weighted NLL on the training document, and 3) backpropagating the 'outer loop' loss of the updated proxy model on a query and label from the training document. These steps are shown in Figure 3. Let I.[fo, a) denote the weighted NLL of Se on document x using weights a. Steps 1 & 2 described by the inner loop update rule:
W
Estimate importance weights for document using weighting model
Adapt model weighted NLL
compony Espace
Lost contect with
0' buse V
sracecraft
busse
00
ted appar enlyy [.]
3. Update weighting model to improve adapted model's knowledge retention
4++1 Φ, + V log p woon
Mtre did ispaco 5ty Land dednesdayy
Figure 3: A single step of CaMeLS meta- training. In step 1. the weighting model (red) produces set of importance weights over the tokens in given document. In step 2, the base model (blue) is updated using single gradient step the weighted NLI, producing In step 3, the weighting model is updated to improve the adapted hase model's ability to answer questions about the document. During test-time adaptation, steps and are applied repeatedly for each document in the test document stream.
adapted model (pink).
8' Opase V Ousse
L(f I., (x)x)) (1)
learning the types of queries that may be of inter- est, informing how we should update our model to test queries while
The inner loop learning rate a can be fixed, sam- pled, or learned. For all of our experiments, we use fixed inner learning rate of a 5e - 4. After the updated proxy model is computed, we compute an outer loop loss measuring the effectiveness of the weighted adaptation procedure the document x::
maximize the performance minimizing disturbance its prior knowledge or behaviors. We next describe an algorithm for lever- aging this dataset to more efficiently update our base model on the test stream of documents Diest-
3.2
CaMeLS: Context-aware Meta-learned Loss Scaling
Louueer log Por (ylg) + Que Laae (Oosse, O', (.oe) (2)
The goal CaMeLS is to distill the information in the training documents, queries, and labels into a parameter vector ø. This vector summarizes the optimal way to update base model on a document stream to maximize retention of information likely be relevant to test queries. CaMeLS accom- plishes this goal by training a weighting model wes small autoregressive language model 9) that re- weights the online NLL loss used in typical online fine-tuning, focusing on the tokens whose NLL gra- dient is most useful for updating a small proxy base model's knowledge. In other words, the weighting model is trained to re-weight the NLL loss such that the proxy model is able to correctly answer questions about document after one gradient step on the modified NLL of the document. The weight- ing model is trained with an episodic bi-level opti- mization, which we explain next in detail (also see Figure 3).
In addition to the negative log likelihood of label given the query and updated base model parame- ters, the outer loss has locality term Lioe which prevents the updated base model parameters from excessively changing the base model's behavior. Cloc is set to 11 for all experiments. T is the sum of the KL divergences Lice between the base model before and after adaptation conditioned on each prefix xfoc of the locality input Flocs with
Line (Orose, O¹, xine) = KL (pon.. (- (-1arhe Ilpo (af)ee) (3)
Finally, perform single update to the weight- ing model's parameters by computing the gradient of the outer loop loss with respect to ó. We opti- mize & with the Adam optimizer, using learning rate of le-5. We accumulate outer loop gradients over 24 examples (document-query-le triples) split into 4 batches of 6 triples.
During each episode, training ducument-query- label triple (z, 4, v) is sampled from Durain and
Performing a single update to the proxy model is the inner loop and updating the weighting model according to the updated proxy model's 's loss on the query is considered the outer loop of a bi-level optimization used to train CaMcLS.
"Section 3.4 contains details on the weighting model's architecture.

---

3.3 Mitigating Train-Test Shift
Method
Time Per Doc
Total GPU Memory
The single-step training procedure described above optimizes for effective knowledge retention for a single document. However, in our online adapta- tion setting, we may update for hundreds or thou- sands of documents before we evaluate on our downstream queries. In order to mitigate this train- test shift, we modify CaMeLS with two strate- gies. First, we do use the same base model parameters during each episode of training. This done to prevent the weighting model from over- fitting to a single base model state. For most training episodes, the starting base model param- eters adapted in the inner loop are the final base model parameters in the previous episode. Every Greset = episodes of training, the starting hase model parameters are reset those of the origi- nal base model. Second, instead of performing an inner update on a single document, we sample an triples
Uniform
772.72
36.62. GB 48.18 18 GB
CaMel
782.46 ms
Table I: Compared to standard uniform fine-tuning, CaMeLS requires slightly more GPU memory to store the weight model and is slightly slower per document. All compute measure- ments were taken while adapting GPT-2 XL. to StreamingQA documents using 80GB NVIDIA A100.
Dataset
Avg. text length
Texts per stream
StreamingQA
~500 tokens
1665 articles
SQuAD
~55 tokens
1170 paragraphs
ArchivalQA
~80 tokens
3001 paragraphs
Table 2: Basic statisties of the data in our online document streams. The sample text streams used to evaluate online adaptation vary significantly in length. For the SQuAD and ArchivalQA datasets, the answer to each query is span in its corresponding document; for StreamingQA, this is not the case.
We next discuss the compute costs of using a trained CaMeLS weighting model for online adap- tation. The additional compute needed for CaMeLS is very small compared to uniform fine-tuning is single forward pass of a weight model for each document we update weight model overhead is small compared the time needed run a forward and backward pass of the base model. Compared to standard uniform fine-tuning, CaMeLS requires slightly more GPU memory to store the weight model and is slightly slower per document. Table shows compute mea- surements during online adaptation of GPT-2 XL on Streaming
inner batch of = boccument-qucry-leen (I1, 91, 11), ...,, (Ix: 9k: Vk) for each episode. A sequence of k inner loop updates is performed:
0: = 0 - a VoL : (xx))
(4)
For large models, the
where 8gg 9 base and 8 Og- The outer loss is computed before, but now averaging the query- label loss over the inner batch. By allowing in- ner loop updates to accumulate during adaptation, & learns an updating strategy that preserves the knowledge of prior updates and maintains the base model's ' ability to learn from subsequent updates.
3.4 Compute & Architecture of CaMeLS
Optimizing bi-level objectives like the one used by CaMeLS is rather memory and compute-intensive requiring memory and compute proportional to the depth of the inner loop (the batch size used for mul- tiple inner loop updates) and proportional to the size of our base/proxy model each inner loop step creates an updated copy of the base model parame- ters in the computation graph. However, CaMeLS only requires a lightweight base model; our exper- iments use DistilGPT-2 as the base model during meta-training, but we find strong transfer to much larger base models during evaluation. The weight- ing model itself is also small; all experiments use DistilGPT-2 the weighting model (a MLP with single hidden state of size 128 is used the head produce token weights). Using the base and weighting models described, we are able to train weighting models using 6 inner loop steps on a single NVIDIA A40 GPU.
Experiments
After outlining datasets and experimental details, we present several experiments aimed at under- standing CaMeLS's behavior in unsupervised on- line adaptation. Section 4.3 studies the extent to which CaMeLS's importance weights improve knowledge retention in online adaptation. Sec- tion 4.4 qualitatively and quantitatively explores the weights themselves, suggesting several abla- tions of CaMeLS that we explore in Section 4.5. Section 4.6 evaluates the cross dotssedttsset generaliza- tion of CaMeLSS weights, and finally we examine the forgetting and plasticity dynamics of CaMeLS within the document stream in Section 4.7.
4.1 Datasets
We apply CaMeLS to three question answering datasets with corresponding source articles. We

---

partition the datasets into 5 splits. Three of these splits (train, valid, test) are used for training, hy- perparameter tuning, and evaluating the CaMeLS weighting model. order to fine-tune the initial QA base models from generic language models, we reserve two more disjoint splits (in order to pre- vent reusing questions during initial QA tuning and online adaptation), labeled QA train and QA valid. Additional details dataset splits and samples in Appendix A. At evaluation time, stream of documents is sampled from the test split. The doc- uments length and text stream lengths are shown in Table 2. In the StreamingQA setting, models must adapt to an entire article as opposed to a selected paragraph, making it our most challenging setting. StreamingQA (Liška et al., 2022): The Stream- ingQA dataset contains combination of human- written and language model generated questions. Questions are generated from English WMT news articles published between 2007 and 2020.
StreamingOA
Undaam
Unitam GA Gh-buning TF-oF 5% Grnff (0un)
Relative 6.
GFT33
GPF- 1.30
GF-333 Base Model
GFT-NA 272
SQuAD
ArchivaiQA
Relative Improvement
0.
3.0
250
2.00
-22
Ralative
225.
OFSMe 1.30 CPI2 Base Model
CIT-3
CFTJ
Figure 4: CaMeLS's meta-learned weights improve knowl- edge uptake after online language model adaptation on a stream of data. The Fl score of the base model before and after adaptation with CaMeLS are computed on questions about the documents used for adaptation. The relative change in F1 is plotted. Top. lower left, and lower right show StreamingQA. SQuAD, and ArchivalQA datasets, respectively. Error hars are standard error over sampled streams of test data.
SQuAD (Rajpurkar et al., 2016): The Stanford Question Answering Dataset (SQuAD) contains human generated questions from Wikipedia articles. The answer to each question is a span contained in a paragraph from Wikipedia.
the questions corresponding to the documents the sampled stream. We compare CaMeLS with 4 baselines. First standard fine tuning or Uniform where tokens are equally weighted. In Uniform + QA-tune we additionally fine tune for question answering after adaptation. Next we consider com- weighting heuristics. Salient Spans corre- sponds to assigning uniform weight tokens in salient spans and no weight all other tokens. In TF-IDF 5% Cutoff, we first compute TF-IDF scores using the both the adaptation documents and additional in distribution documents. To ac- count for stopwords, we remove the 5% of words with lowest TF-IDF scores. The remaining TF-IDF scores are used to reweight the tokens. For each combination of base model and online adaptation strategy, the learning rate used at test time chosen via hyper parameter sweep on a stream of documents sampled from the validation set.?
ArchivalQA (Wang et al., 2022): The ArchivalQA dataset contains automatically generated questions. Questions are generated from articles in the New York Times Annotated Corpus (Sandhaus, Evan, 2008). The answer each question is span con- tained in an article.
4.2 Experimental protocol details
We conducted evaluations on two families of au- toregressive language models, the GPT-2 (Rad- ford et al., 2018) and GPT-Neo familics (Black et al., 2021), 1, as well as GPT-J (Wang and Komat- suzaki, 2021). We note that all models evaluated use the same text tokenization. For all datasets, we first fine-tune each pretrained model on question- answer pairs from that dataset. These tuned models represent the static language models we wish to update and will be referred to base models. For each dataset, a single weighting model is trained. The proxy language model used during weighting model training is DistilGPT-2 fine-tuned on the QA train split of the respective dataset.
4.3 CaMeLS improves knowledge retention
We first compare the knowledge retained by CaMeLS and baselines for three different data dis- tributions in Figure 4. CaMeLS outperforms other
compute constraints, adaptation of GPT-Neo 2.7B and GPT-J 6B.
Adafactor optimizer for
At evaluation time, the base model is updated on a stream of documents sampled from the test split 5. The final adapted base model is evaluated
"TF-IDF scores are computed using a word level tokeniza- tion. These scores are then mapped to the BPE tokenization
the adapted language models.
'All learning sweeps are conducted over the following values: Ie-4, 2.5e-5, 6.25c-6, 1.625e-6). The optimal leaming rates are in Appendix D.
'We use an Adam Optimizer most experimental runs. Due

---

Relative Improvement
Part of Speech
Frequency
0.4
0.0
saleed Spare
Binwsta CaMALA
Importance Welght
Proportion
-0.1
POT: Resumpps
GPT Base Model
bottom 10% of range
central 80% range
tap ofrnage
The importance weight distribution learned by CaMeLS is bimodal, with proper nouns and numbers being the parts of speech most likely to have high importance weights. The overall importance weight distribution (left) and the dis- tribution conditioned by part of speech (right) shown on the validation split of StreamingQA.
Figure S:
Figure Ablations of CaMeLS. Bimodal Ablation restricts the weighting model from outputting intermediate values while the POS ablations remove context dependence, con- ditioning only on part speech. While restricting CaMeLS to output only one two values only slightly reduces perfor- mance, conditioning only on part speech, rather than full context, drastically reduces knowledge retention.
online adaptation approaches actoss range of datasets and weighting models. Despite the dif- ference scale between the proxy model used dur- ing weight training and the evaluated base models, CaMeLS's learned importance weights generalize well to the largest base model we evaluate, GPT-J 6B, which is over 70 times the size of the proxy model (DistilGPP-2,, 82 million parameters) used during training. We find that standard online fine tuning (uniform weighting) with Adam performs very poorly on online adaptation. Even with tuned learning rate and further training for question answering post adaptation, uniform weighting fails to achieve a significant improvement for several models tested.
an outdated language model's performance most rapidly declines on proper nouns and numbers.
4.5 Ablations
In order to verify that context-aware weights are truly necessary to achieving improved knowledge retention,
now examine several ablations of CaMeLS. In the POS: Resample ablation, the weight of each token is generated by sampling from the distribution of importance weights on all tokens of the same part of speech. In the POS: Mean ablation, each token is weighted by the mean im- portance weight ussigned to tokens of that part of speech. We additionally consider a Bimodal ab- lation where outputs of the weighting model are rounded to either the largest smallest value in the distribution of importance weights.
4.4 Analysis of learned weights
One benefit of CaMeLS over other methods for meta-lcarning model updating strategics is that learned updating strategy, token weights, is inter- pretable. Figure 1 shows the per-token weights on sample text and how they combine with the un- weighted gradient norms to produce sparsified per- token gradient norms. In this section, provide additional analysis of CaMeLS's learned weights. We examine the distribution of weighting model outputs on articles in the validation set of Stream- ingQA in Figure 5. As our qualitative evaluations show, we confirm that the distribution of weights over the entire validation split of StreamingQA is indeed sparse and bimodal. We thus interpret the weighting model acting context-aware bi- nary classifier, determining if token is informative uninformative. When binning weights by part of speech, we find that numbers and proper nouns are most frequently assigned a high weight. This result aligns with Lazaridou al. (2021), who found that
Figure shows the results on the StreamingQA datasct for two different base models. We observe that ablating the weighting model to only output two values slightly reduces performance, while still achieving significant F1 improvement and outper- forming baseline approaches. The strong perfor- mance of the binary ablation suggests that binary decision of whether to train on a given token is an effective approach online adaptation, though the full version of CaMeLS that allows for variation the weight magnitude still performs best.
In contrast, neither part-of-speech ablation pro- duces effective knowledge retention, either per- forming worse than the uniform baseline or fail- ing to significantly increase Fl score. This result strongly suggests that although part of speech core- lates strongly with learned weights, part of speech alone is not sufficient to determine when a token contains important information. We conclude that context- awwreness is indeed helpful for identifying

---

6-256
Y- 23e5
CaMoLS mStreamingQ4)
0.290
0.089
0:233
Test Quary
Adaptation Approach
4025
(SQuAD)
6:1
10.00
CaMel55 (ArhivalQN)
0.040
0.268
202 KI 290
1f 6.266
6.2
Uniform Qi-tune
0.096
0.043
0.060
6:10-
Salent Spans
0.072
0.028
0:210
Cutoff
0.086
0.158
Document Updates
StreamingGA
Evaluation Dataset
SQUAD
ArchivalQA
-0.2
CaMeLS
Sallent Spans
Uniforn
Iditial
TF-0F- 5%% Cutoff
Figure 8: Base model performance during StreamingQA line adaptation of GPT-2 XL. Performance evaluated every 200 article updates on the downstream answering task (top) and unrelated validation questions used in QA pretraining (hottom). Results are plotted for two learning rates. 6, 2,20e (left) is the optimal learning rate for the TF-IDF baseline while 2.500e-5 (right) is the optimal learning rate for all other methods shown. Shaded regions are standard error over 4 All adaptation methods lead to gradual degradation in unrelated questions performance. CaMeLS results in gradual increases base model test performance. Using its optimal learning rate, uniform fine-tuning with post adaptation QA tuning are only realizes its performance increases after post- adaptation QA-tuning step.
Figure 7: CaMeLS weight models unseen data distribu- tions (oft-diagonals of top three rows) frequently outperforms baseline online adaptation approaches (bottom three rows). Each CaMel.S model trained on single datasct (shown in parenthesis) and used adapt GPT-2 XL streams data from various datasets.
important tokens in online adaptation.
4.6 Cross Dataset Transfer
Beyond generalizing to new base models, we now study CaMeLS's ability to generalize to new data distributions. We evaluate CaMeLS's performance for all nine possible combinations of train and test dataset, using StreamingQA, SQuAD, and ArchivalQA. Figure 7 shows the results. We find that CaMeLS trained on different dataser still typically outperforms the baseline methods, provid- ing stronger evidence that the weighting scheme learned by CaMeLS is general- purpose. The gener- alizability of CaMeLS's weighting model a key attribute increasing its practical utility.
uniform fine-tuning with learning rate of 2.500e- and additional QA-tuning results in gradual degradation in test performance with improvement only becoming realized after the post-adaptation QA-tuning step. Turning to performance on un- related queries, we see that all methods result in gradual degradation in performance on indepen- dent queries. At lcarning rate of 6.250c-6, all methods lead to comparable degradation in perfor- mance on unrelated queries. At learning rate of 2.5e-6 CaMeLS leads the lowest drop in unre- lated query performance. Taken together, these results suggest that the CaMeLS is able to more ef- fectively update the base model's knowledge, while still preserving the model's 's pre-existing knowledge and its representation of the task.
4.7 Forgetting and plasticity
So far, our evaluations have considered only the QA accuracy at the end of online adaptation. In this section, we investigate the evolution of learning during the online adaptation process. While adapt- ing GPT-2 XL to data from StreamingQA, we eval- uate the intermediate models produced by CaMeLS and baseline methods every 200 document updates. Results are plotted for two learning rates. 6.250e-6 is the optimal learning rate for the TF-IDF baseline while 2.500e-5 is the optimal learning rate for all other methods shown. Figure 8 shows the perfor- mance when intermediate models are evaluated on the entire set of evaluation queries and additionally evaluated on set of unrelated queries sampled from the QA validation spit. CaMeLS consistently improves performance on test queries during online adaptation, while the best performing baseline
Finally, in Figure 9, we aim to answer the ques- tions how long does the model remember the an- swer to a question after observing it? We show the average improvement in F1 score across test queries against the number of timesteps since the model observed the document containing the an- swer to the query. Each adaptation method is ap- plied using uniquely tuned learning rate. After the 200 document sequence containing the rele- vant document, all methods see clear average improvement in Fl score, signifying learning is happening. However, we also note that CaMeLS

---

produces both a higher initial improvement well a higher asymptotic improvement in Fl score. CaMeLS not only improves the immediate plas- ticity of the model, integrating knowledge more readily, but also reduces forgetting, preserving the newly-integrated knowledge for longer.
F1 Increase
5 Diseussion
de Document Updates since Adaptation
<0
-04
20
While large language models are powerful, keeping them up-to-date remains challenge. In this pa- per, we consider the unsupervised online language model adaptation setting, which a language model's knowledge must be updated using stream documents, without annotations of key facts or information. Finding that naive online tuning provides little retention of knowledge from the doc- ument stream, we propose Context-awar Meta- learned Loss Scaling (CaMeLS), meta- meta-learning al- gorithm that learns importance weighting model reweight the per-token loss of the online data stream. CaMeLS leverages side information of the form of paired documents and knowledge queries about those documents to identify which tokens in the documents are most likely to be informative for answering downstream queries. Empirically, we find that the importance weighting model learned by CaMeLS consistently improves knowledge re- tention across three datasets of documents and ques- tions. Crucially, we find that CaMeLS's importance weighting model generalizes across outdated lan- guage models and datasets, meaning that an impor- tance weighting model can be trained once on a small proxy language model (such as DistilGPT-22 and then be immediately used to improve online adaptation of much larger models, like GPT-J 6B. This transferrability of CaMcLS's s weighting model significantly increases its practical utility.
Document Adsgtatinn CaMelS
Salient Spans Uniferm
Uniform 1 -.-tuune
TF-DPP + 5%% Cutorr
While adapting GPT-2 XI. on StreamingQA, amine the average improvement Fl score of queries against the time since the model observed the corresponding docu- The shaded region represents the interval in which the source document was presented. CaMeLS leads larger initial improvement and asymptotic improvement in Fl score than other methods. Although this pailaadaptation evaluation does not QA-tuning. Uniform + QA-tune corresponds uniform fine tuning using a learning rate optimized down- stream performance given an additional QA-tuning step. Each adaptation method applied using a uniquely tuned learning rate.
Figure 9:
adaptation on steams of thousands of documents. However, in order to effectively update outdated language models in real-world scenarios, it is rea- sonable to expect significantly larger volume of documents. Beyond dataset scale, our experiments study adaptation of base models up to 6B parame- ters, but recent work suggests the continual learn- ing dynamics of language models changes drasti- cally at extreme scale (100B+ parameters); future work may increase the scale of the present study by considering adaptation on longer streams of docu- ments using larger base evaluation models. Finally, study only question-answering models and the question-anie -awsworinnnneninngggg task, as is the most direct form of knowledge retention assessment. Future work may examine knowledge retention in other types of models through alternative downstream tasks that leverage the knowledge in the document stream more indirectly, as well as studying the abil- ity to continually update general-purposs genera- tive models of language dialogue models.
Limitations & Future Work
While our experiments suggest that learned im- portance weights consistently improve knowledge retention after unsupervised online adaptation, our study has several limitations. CaMeLS assumes access to side information in the form of training document, query, and label triples. This require- ment may be oncrous in domains where labeling is expensive. Future work may apply CaMeLS to settings without access side information queries and labels, i.e., only a purely unlabeled stream of training documents, using the temporal structure of the data as the signal for learning. We study
Acknowledgements
The authors thank Huaxiu Yao for his input at mul- tiple stages of the project. CF and CDM are CI- FAR Fellows. EM gratefully acknowledges fund- ing from Anight-Henness, Graduate Fellowship. This research was supported in part by Juniper Net- works.

---

References
Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. 2023. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large
Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Georgi Gerganov. 2023, llama.cpp. https:/ h/gp:://iitn ght.. com/ Eggrrgnov/llame cpp.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Ste- fanie Tellex. 2019. Openwebtext corpus. http: //Skylion007, gi thub. Lo//peenWebTextoorpuss
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang. 2020. REALM: retrieval- augmented language model pre-training. CoRR. abs/2002.08909.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1489-1501, Berlin, Germany, Association for Com- putational Linguistics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz. Ece Kamar, Pe- ter Lec, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with GPT-4. ArXiv preprint arXiv:2303.1277 12712.
Joel Jang, Sconghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In Interna- rional Conference on Learning Representations.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with a-GEM. In International Con- ference on Learning Representations.
Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991. A dynamic language model for speech recognition. In Speech and Natural Language: Proceedings of Workshop Held Pacific Grove, California, Febru- ary 12, 1991.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 22Xv:22204.02311.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Barnskkkaaaiikkaa...a Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114.
Kevin Clark, Kelvin Guu, Ming- Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. 2022. Meta-learning fast weight language models. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 9751-9757, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Roland Kuhn. 1988. Speech recognition and the fre- queney of recently used words: A modified Markov model for natural language. In Coling Budapest 1988 Volume 1: International Conference on Computa- tional Linguistics.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-Aware Language Models as Temporal Knowledge Bases. Transac- tions of the Association for Computational Linguis- Linguis tics, 10:257-273.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detec- tion of linguistic change. In Proceedings of the 24th International Conference World Wide Web, WWW "15, page 625-635, Republic and Canton Geneva, CHE. International World Wide Web Conferences Steering Committee.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. 2022. Continual backprop: Stochastic gradient descent with persistent randomness.
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomás Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization
Danny Driess. Fei Xia, Mehdi S. M. Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent

---

in neural language models. In Advances in Neural Information Processing Systems.
Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 687-692. Baltimore, Maryland. Association for Computational Linguisties.
Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022. Faithfulness in natural language generation: A systematic survey of anal- ysis, evaluation and optimization methods. ArXiv, abs/2203.05227.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners, Ms.. OpenAI.
Adam Liška. Tomáš Kočiský, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener. Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young. Ellen Gilsenan-McMahon. Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. Streamingqa: benchmark for adaptation new knowledge over time in question answering models.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. 2019. Continual unsupervised representation learning. Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. arXiv preprint arXiv:210900505.
Marek Rei. 2015, Online representation learning in re- current neural language models. In Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pages 238-243, Lisbon. Portugal, Association for Computational Linguistics.
David Lopez-Paz and Marc' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning, Ad- vances in neural information processing systems, 30.
Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin
Liu, and Sung Ju Hwang. 2022. Representational continuity for unsupervised continual learning. In In- lernational Conference on Learning Representations.
Mengye Ren, Michael Louis Iuzzolino, Michael Curtis Mozer, and Richard Zemel. 2021. Wandering within a world: Online contextualized few-shot learning. In International Conference on Learning Representa- tions.
Michael McCloskey and Neal J. Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, vol- ume 24 of Psychology of Learning and Motivation, pages 109-165. Academic Press.
G. Salton and M. McGill. 1986. Introduction to Mod- ern Information Retrieval. McGraw-Hill, Inc., New York, NY, USA.
Sandhaus, Evan. 2008. The new york times annotated corpus.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associa- tions in GPT. ArX1v:2202.05262.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster. cheaper and lighter. In NewrlPS EMC² Workshop.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast model editing scale. CoRR.
Hanul Shin. Jung Kwon Lee, Jachong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc
Eric Mitchell, Charles Lin, Antoine Bosselut, Christo- pher D Manning, and Chelsea Finn. 2022. Memory- based model editing at scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15817-15831. PMLR.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2023. Prompting GPT-3 to be reli- able. In The Eleventh International Conference on Learning Representations.
T. Mitchell, W. Cohen, E. Hruschka, Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gard- ner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis. Mohamed, N. Nakashole, E. Platanios, A. Rit- ter, M. Samadi, B. Settles, R. Wang. D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and Welling. 2018. Never-ending learning. Commun. ACM, 11351153
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In ICLR.
Sebastian Thrun and Tom M. Mitchell. 1995. Lifelong robot learning. Robotics and Autonomous Systems, 15(1):25-46 The Biology and Technology of Intelli- gent Autonomous Agents.
Miles Osborne, Ashwin Lall, and Benjamin Van Durme. 2014. Exponential reservoir sampling for streaming language models. In Proceedings of the 52nd Annual

---

Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: 6 Billion Parameter Autoregressive Lan- guage Model, https://8 http:/gggiuuu.ooooooo.ee watbb/ooo//nigflll/zz mesh-transformer-jaax
Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022. Archivalqa: A large-scale benchmark dataset for open domain question answering over historical news collections.
Dani Yogatama, Chong Wang, Bryan Routledge. Noah A. Smith, and Eric P. Xing. 2014. Dynamic language models for streaming text. Transactions of the Association for Computational Linguistics, 2:181- 192.

---

SQA
SQuAD
ArchivalQA
B Larger Proxy Models
Split
Na/o
Nr
Ng
N₂
N₂
We conduct a preliminary investigation on the feet of using larger proxy model during CaMeLS meta-training. By default, we use a QA-tuned Dis- tIGPP2 (82M) as the proxy model. We addition- ally meta-train using a GPT-2 Small (117M) as the proxy model. Due to compute limitations we were not able to meta-train using any larger proxy mod- els. Results StreamingQA are shown in table 5. We see no significant difference in performance in this setting. Qualitatively, the two weighting models generate similar outputs. We hypothesize that CaMeLS learns a weighting which reflects the innate importance of tokens in the text to answer- ing the meta-training questions, rather than a proxy model specific token importance. We emphasize that this is hypothesis and believe more rigor- ous exploration of proxy model size is an exciting direction for future work.
Train
21k
8.6k
39.9k
12.8k
21.7k
Validation
1.7k
1.2k
5.6k
3.0k
5.3k
Test
5k
2.1k
10.6k
5.0k
8.7k
QA Train
40k
40k
12.4k
QA Valid.
4k
2.1k
3k
Table 3: Number of documents N₂ and questions N, for each dataset. Each document in StreamingQA (SQA) corresponds to single question, while SQuAD and ArchivalQA contain documents corresponding to multiple questions.
A Dataset Details
The sizes of dataset splits are shown in Table 3. Sample documents, questions, and answers are shown in Table 4. Only documents from 2018 and on-wards are used to the train, validation, and test splits of StreamingQA. For SQuAD, the entirety of the validation set of SQuAD is used at our test split. The topics in training set of SQuAD are re- partitioned form the other splits. We divide the validation set of the ArchivalQA dataset form our splits. These splits are done temporally, us- ing documents from 1987-1990 for QA Training, 1991-1999 for QA Validation, 1993-2001 for Train- ing, 2002-2003 for Validation, and 2004-2007 for Testing.
C Combining CaMeLS with other online Adaptation Methods
There are various other methods for online adapta- tion which leverage the adaptation documents. Two such methods are in-context learning and retrieval. This section shows preliminary experiments lever-
Dataset
Document
Question
Answer
StreamingQA
Colin Farrell goes missing
trailer March 2 (UPI)
What does Artemis Fowl embark on?
dangerous journey into the unknown
Colin Farrell joins the cast Artemis Fowl in the latest
trailer Disney's upcoming fantasy-adventurr film. Farrell featured in the clip, released on Monday, the missing father of Ferdia Shaw's Artemis Fowl who also goes by same name. Farrell's 's character is criminal mastermind who mysteriously disappeared. Artemis Fowl learns that father protected powerful secrets that have kept mankind safe and learns that his disappearance is connected to secret fairy world. Artemis Fowl. with help from his loyal protector Butler (Nonso Anozie), embarks on dangerous journey into the unknown in order to save his father...
SQuAD
Luther is honoured on 18 February with a commemoration in the Lutheran Calendar of Saints and in the Episcopal (United States) Calendar of Saints. In the Church of England's Calendar of Saints he is commemorated on 31 Octoher.
When is Luther 18 February commemorated in the Lutheran Calendar of Saints?
ArchivalQA
If it feels like the Heat Miser ("Oh, some like it hot, but like it really hot") has been lurking of late, may be due to NBC's coming remake of the animated 1974 television movie "The Year Without Santa Claus." The four-time Tony Award winner Harvey Fierstein (" ("Hairspray")) signed on this week to replace Chris Elliott in the role of the Heat Miser; Mr. Elliot had to bow out because of scheduling conflict. The new version will be seen later this year.
Who replaced Chris Elliott as the Heat Miser?
Harvey
Fier-
stein
Table 4: Example documents, questions, and answers from the test split of each dataset.

---

Proxy Model
GPT-Neo 1.3B
GPT-2
models and retrievers, Using CaMeLS to adapt document-conditioned question-answering models consistently improves adaptation performance over vanilla retrieval.
DistilGPT2 (82M)
0.190 + 0.017
0.308 + 0.018
GPT-2 Small (117M)
0.176 + 0.023
0.309 + 0.012
Table S: StreamingQA F1 Increase comparison for CaMeLS metu-trained using DistilGPT2 (82M) and GPT-2 Small (117M) proxy models. Online adaptation of GPT-Neo 1.3B and GPT-2 XL. is evaluated. In the tested setting, varying the proxy model size does not change CaMeLS performance.
These results use the CaMeLS weighting model trained using a QA-proxy model on ArchivalQA, We expect the performance of CaMeLS to increase if meta-trained using proxy model and outer loss more analogous the evaluation setting. For ample, increase performance in the retrieval setting, we could present the source document when com- puting the outer loss and using a document condi- tioned QA proxy model. We acknowledge that we do not evaluate any baseline methods and think that extensive comparisons of parametric updating in conjunction with these other methods would be an exciting direction for future work. As is, these sults do show that parametric online adaptation can be used complement document-storage based methods.
Method
GPT-2 XL
GPT-Neo 1.3B
5-shot ICL
0.1091
0.0533
ICL w/ CaMeLS 0.1594
0.1398
Table Adapting the models with CaMeLS consistently improves the FI scores simple in-context learning setting.
aging CaMeLS in conjunction with these methods on the ArchivalQA dataset. We show that CaMeLS is complementary to both in-context learning and retrieval; for both methods, the adaptation perfor- mance is improved by CaMeLS.
In our first set of experiments, do five-shot in-context learning. We assume can prompt the model with the oracle document containing the answer the question (i.e., the best-case scenario for in-context learning). The prompt is formatted as [ex. doc 1] [ex. q 1] [ex. ans 1] [ex. doc 5] [ex. 5] [ex. ans 5] [oracle test doc] [test question]. We use the base GPT-2 XL and GPT-Neo 1.3B models (QA-tuned models performed much worse with in-context learning). As shown in Table 6, find that adapting the base models with CaMeLS consistently improves the Fl scores of in-context learning.
D Optimal Online Adaptation Learning Rates
When evaluating loss reweighing methods in our experiments, the learning rate used adapt base models is found via a learning rate sweep. For each combination of dataset, base model, and adap- tation method, we test a range of learning rates to adapt the base model on a stream of documents from the validation split of the dataset. The best performing learning rates are used for later exper- iments on the test split of the dataset. We test the following learning rates: [lc-4, 2.5e-5, 6.25c-6, 1.625e-6). The optimal learning rates found via these sweeps are shown in Figure 10.
In a second set of experiments, we consider a simple retrieval setup. Results are shown in Ta- ble 7. We fine-tune GPT-2 XL and GPT-Neo 1.3B answer questions with the source document in the context. We retrieved documents using ran- dom, oracle, and BM25 document retrieval. We use CaMeLS to update the parameters of the document- conditioned question-answering models. Across
GPT-2 XL
GPT-Neo 1.3B
Method
Random
BM25
Oracle
Random
BM25
Oracle
Vanilla Retriever
0.0694
0.6812
0.7290
0.0624
0.6898
0.7401
Retriever w/ CaMeLS
0.1156
0.7106
0.7565
0.1045
0.7356
0.7832
Table 7: Using CaMeL S to adapt document-condiciieeeeo gusstion-answering models consistently improves adaptation perfor- mance over vanilla retrieval,

---

(a) StreamingQA
Method
Base Model
Learning Rafe
CaMeLS
GPT-2 Large
2.500c-5
GPT-Neo 1.38
6.250c-6
GPT-2 XL
2.500c-5
GPT-Neo 2.78
6,250c-6
GPT-J 6B
6,250e-6
TF-IDF 5% Cutoff
GPT-2 Large
1.625e-6
GPT-Nco 1.3B
1.625c-6
GPT-2 XL
6,250c-6
GPT-Nco 2.7B
1.625c-6
GPT-J 6B
1.625e-6
Salient Spans
GPT-2 Large
6,250e-6
GPT-Neo 1.3B
1.625e-6
GPT-2 XL
2,500e-5
GPT-Neo 2.7B
1.625c-6
GPT-J
6,250c-6
Uniform QA-tuning
GPT-2 Large
1.625e-6
GPT-Neo 1.3B
2.500e-5
GPT-2 XL
2.500e-5
GPT-Neo 2.7B
6,250c-6
GPT-J
2,500e-5
Uniform
GPT-2 Large
1.625e-6
GPT-Neo 1.3B
1.625e-6
GPT-2 XL
1.625e-6
GPT-Neo 2.7B
1,625c-6
GPT-J 6B
1.625c-6
(b) SQuAD
Method
Base Model
Learning Rate
CaMeLS
GPT-Neo 1.3B
6.250e-6
GPT-2 XL
6,250e-6
Salient Spans
GPT-Nco 1.3B
1.625c-6
GPT-2 XL
1.625e-6
TE-IDF + 5% Cutoff
GPT-Nco 1.3B
1.625c-6
GPT-2 XL
1,625e-6
Uniform
GPT-Nco 1.3B
1.625c-6
GPT-2 XL
1.625e-6
Uniform QA-tuning
GPT-Neo 1.3B
1.625e-6
GPT-2 XL
1.625e-6
(c) ArchivalQA
Method
Base Model
Learning Rate
CaMeL.S
GPT-Neo 1.3B
1.625e-6
GPT-2 XI.
6.250c-6
GPT-J
6.250e-6
Salient Spans
GPT-Neo 1.3B
1.625e-6
GPT-2 XL
1.625e-6
GPT-J 6B
6.250e-6
TF-IDF + 5% Cutoff
GPT-Neo 1.3B
1.625e-6
GPT-2 XI.
1.625c-6
GPT-J 6B
6,250c-6
Uniform
GPT-Nco 1.3B
2,500e-5
GPT-2 XL
6,250e-6
GPT-J 6B
6.250c-6
Uniform QA-tuning
GPT-Neo 1.3B
2.500e-5
GPT-2 XL
6,250e-6
GPT-J 6B
6.250e-6
Figure 10: Optimal adaptation learning rates used to evaluate each comhination of adaptation method and hase model.


Reviews:
[{"review_id": "ulOFZy0PH6", "reviewer": "Reviewer_dDnG", "summary": "", "questions": "a.  Could you please provide more details on how the additional labeled dataset was generated to train the weighing model? \n\nb. What is added computational overhead of CaMeLS when compared to the baseline? Please consider including this comparison in the paper", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the thoughtful feedback! We're glad you appreciated the novelty of CaMeLS and found our experiments insightful.\n\n >CaMeLS assumes an additional corpus of labeled data to train the weighting model. However, it is unclear how this dataset was generated. This assumption will be a problem when the labeling process is expensive.\n\nIt is true that CaMeLS requires a corpus of document query pairs to train the weighting model. Many question-answering datasets - including the three we study - are generated by first selecting documents and then creating corresponding questions for the documents (via LLMs or humans) [1,2,3]. Our experiments with CaMeLS use the StreamingQA, SQuAD, and ArchivalQA data sets off the shelf. The only additional step needed was matching the questions with their original source documents - that is, given a document ID, retrieve the original document from the source corpus. Our dependence on labeled data is further mitigated by the ability of CaMeLS to generalize to unseen data distributions (4.6). We are excited about learning CaMeLS weight models using only unlabeled data as a future direction of research.\n\n >The paper lacks comparisons on the compute overhead required for CaMeLS compared to the baselines\n\nWe discuss the compute costs of training CaMeLS in Appendix B. In the comparison below, we find that **the additional compute needed for CaMeLS is very small compared to uniform fine-tuning.** When using CaMeLS for online adaptation, the compute overhead of CaMeLS is a single forward pass of a weight model for each document we update on. For large models, the weight model overhead is small compared to the time needed to run a forward pass and backprop of the base model. Performances of the two methods when used to adapt GPT2-XL on StreamingQA documents are shown in the table below. Performances are logged on a NVIDIA A100 (80GB, SXM4). CaMeLS requires slightly more GPU memory to store the weight model, and is marginally slower per document. \n\n| Model         | Time per Document (± std) | GPU Memory Allocated |\n|---------------|-------------------------------|----------------------|\n| GPT2-XL, Uniform | 772.72 ± 25.95 ms         | 46.62 GB              |\n| GPT2-XL, CaMeLS  | 782.46 ± 26.86 ms         | 48.18 GB              |\n\n\n\n\n>Writing needs to be improved. Many sections are hard to follow\n\nApologies for the unclear sections. We will further edit and improve the paper. Feel free to let us know which sections were the hardest to follow. \n\n[1] StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models. Adam Liška, et al., 2022. arXiv.\\\n[2] ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections. Jiexin Wang, et al., 2021. arXiv.\\\n[3] SQuAD: 100,000+ questions for machine comprehension of text. Pranav Rajpurkar, et al., 2016. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a novel approach that improves knowledge update after online language model adaption on a data stream. CaMeLS improves F1 score and  outperforms baselines like naive fine-tuning across different datasets and models. \n- The paper has ablation experiments to understand the importance of context aware weights for knowledge retention. These insights might be valuable to the community\n- Experiments also demonstrate that the CaMeLS approach can be generalizable on unseen data distributions", "reasons_to_reject": "-  CaMeLS assumes an additional corpus of labeled data to train the weighting model. However, it is unclear how this dataset was generated. This assumption will be a problem when the labeling process is expensive. \n- The paper lacks comparisons on the compute overhead required for CaMeLS compared to the baselines \n- Writing needs to be improved. Many sections are hard to follow", "questions_to_authors": "a.  Could you please provide more details on how the additional labeled dataset was generated to train the weighing model? \n\nb. What is added computational overhead of CaMeLS when compared to the baseline? Please consider including this comparison in the paper", "paper_topic_and_main_contributions": "The paper focuses on improving unsupervised online language model adaptation by proposing a meta-learning algorithm (CaMeLS) that learns an importance weighing model to re-weight the per-token loss of the data-stream. The main hypothesis of the paper is that online adaptation by naive fine-tuning can be improved by only fine-tuning on a subset of tokens in the data-stream, which likely lead to useful updates.  CaMeLS makes use of a small proxy model like DistilGPT-2 to train the weighting model, which uses an episodic bi-level optimization for training. CaMeLS also uses an additional corpus of documents having query samples and corresponding labels to identify which tokens in the documents are most likely to be informative. The proposed approach was evaluated on 3 datasets:  StreamingQA, SQuAD and ArchivalQA. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5GNjmiOe7U", "reviewer": "Reviewer_kJwz", "summary": "", "questions": "* I am a little confused about how, for squad and ArchivalQA, the authors see decreasing performance (ie negative relative F1 improvement) with additional finetuning and uniform weights, especially after QA adaptation.\n* In Figure 9 - I think you need to show this effect with more than one document, so you can provide error bars around the line plots.\n* Figure 8 - I think it is a stretch to say that your technique best preserves knowledge in the base model, since it leads to worse performance on unrelated questions than uniform weighting.\n* Line 498-499: calling the technique \"general purpose\" by evaluating three datasets (in one task!) is too strong, I would reword this.\n* I understand that using Distil-GPT2 for the proxy model makes sense from an efficiency perspective, but are there performance improvements from increasing the size of proxy model?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are glad you find CaMeLS to be an interesting new approach!\n>I don't see mention of code release, which would be helpful for reproducibility.\n\nWe unfortunately forgot to mention this. There will be a public code release. Until then, we have also anonymized the code which can be found here https://anonymous.4open.science/r/CaMeLS-63F8/. \n\n>I think a simple baseline, using TF-IDF weighting (perhaps with some thresholding to, for example, remove stopwords and extremely common tokens), might be a much simpler and cheaper alternative to the proposed technique. It would be nice to see this comparison in the paper.\n\nThank you for the suggestion. We have preliminary results showing TF-IDFs performance adapting GPT2-XL and gpt-neo-1.3B on streamingQA. We additionally show the standard error of mean over 4 seeds. TF-IDF Cutoff additionally removes the 5% of words with lowest TFIDF. We find TF-IDF to be an effective baseline for online adaptation, performing slightly better than salient span masking. **However, CaMeLS still enables significantly better knowledge uptake than TF-IDF.** We will add an extensive comparison to the final paper. \n\n| Method        | GPT-Neo-1.3B F1 Increase | GPT2-XL F1 Increase  |\n|---------------|--------------------------|----------------------|\n| CaMeLS        |  0.190 ± 0.017           | 0.308 ± 0.018        |\n| TF-IDF        |  0.073 ± 0.006           | 0.034 ± 0.008        |\n| TF-IDF Cutoff |  0.074 ± 0.005           | 0.082 ± 0.020        |\n| Uniform       | -0.099 ± 0.014           | 0.038 ± 0.026        |\n| Uniform + QA  | -0.016 ± 0.035           | 0.178 ± 0.041        |\n| SSM           |  0.058 ± 0.018           | 0.036 ± 0.041        |\n\n\n>I am a little confused about how, for squad and ArchivalQA, the authors see decreasing performance (ie negative relative F1 improvement) with additional finetuning and uniform weights, especially after QA adaptation.\n\nWe agree that negative relative F1 improvements are unintuitive.  This decrease in performance is primarily observed in the GPT-Neo 1.3B model, which is among the weaker performers across all datasets and adaptation methods examined. In these particular settings where F1 scores decline, we still observe a decrease in the Negative Log-Likelihood (NLL) for answer tokens following the questions. This suggests that the model is still adapting to the new text to some extent. We hypothesize that while the models do gain new information about the documents during the adaptation phase, they become less adept at responding to question prompts in the desired format. (even after additional QA tuning). In continual learning literature, work has found that after training on many different data distributions, the ability to learn the next data distribution diminishes. aka, neural nets lose plasticity over time [1]. Even for models which show an increase in F1, we see that QA-tuning post adaptation does not fully recover initial model performance on unrelated question-answer pairs (Figure 8, right). \n>Figure 8 - I think it is a stretch to say that your technique best preserves knowledge in the base model, since it leads to worse performance on unrelated questions than uniform weighting.\n\n>Line 498-499: calling the technique \"general purpose\" by evaluating three datasets (in one task!) is too strong, I would reword this.\n\nThank you for the critiques of particular figures and claims. We will reword the specified points to be more accurate. Figure 9 currently shows document performance averaged over all documents presented over multiple online adaptation experiments, not performance for just a single fact. We apologize for any confusion and will update the figure to be more clear and to show error bars. \n\n>I understand that using Distil-GPT2 for the proxy model makes sense from an efficiency perspective, but are there performance improvements from increasing the size of proxy model?\n\nWe are also curious about the effect of using larger proxy models. We trained a CaMeLS weight model using GPT2-small (114M params) as a proxy model. Due to compute limitations we were not able to meta-train using any larger proxy models. Results comparing performance between the two proxy model sizes on StreamingQA are shown below, errors are 1 standard error of mean. \n| Proxy Model       | GPT-Neo-1.3B F1 Increase | GPT2-XL F1 Increase |\n|-------------------|-------------------------|---------------------|\n| DistilGPT2 (82M)  | 0.190 ± 0.017           | 0.308 ± 0.018       |\n| GPT2-Small (117M) | 0.176 ± 0.023           | 0.309 ± 0.012       |\n\nWe see no significant difference in performance in this setting. Qualitatively, the two weighting models generate similar outputs. We hypothesize that CaMeLS learns a weighting which reflects the innate importance of tokens in the text to answering the meta-training questions, rather than a proxy model specific token importance. If this were the case, the performance of CaMeLS would not always depend on the proxy model used. We emphasize that this is a hypothesis and believe a more rigorous exploration of proxy model size is an exciting direction for future work. \n\n[1] Continual Backprop: Stochastic Gradient Descent with Persistent Randomness. Shibhansh Dohare, et al., 2021. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, while the technique is scoped to QA (with the existence of document-query-label tuples), I think it was a good case study of the technique, and I learned from it. I found this technique to be interesting and new, and quite effective against reasonable baselines (uniform weighting, adaptation for QA, and \"salient spans\"). I thought the authors had careful ablations that showed their technique was not just putting higher weight on nouns/pronouns.\n\nWhile the authors note that binarizing the weights does not produce substantially different outcomes, I am not sure if that is necessarily a weakness; it might just indicate that the relative weighting of different tokens may not matter as much, which simplifies the approach slightly. \n\nThe authors test their approach up to 6B parameters, showing that their technique brings consistent improvements across model scales.\n\n", "reasons_to_reject": "* I don't see mention of code release, which would be helpful for reproducibility.\n* I think a simple baseline, using TF-IDF weighting (perhaps with some thresholding to, for example, remove stopwords and extremely common tokens), might be a much simpler and cheaper alternative to the proposed technique. It would be nice to see this comparison in the paper.", "questions_to_authors": "* I am a little confused about how, for squad and ArchivalQA, the authors see decreasing performance (ie negative relative F1 improvement) with additional finetuning and uniform weights, especially after QA adaptation.\n* In Figure 9 - I think you need to show this effect with more than one document, so you can provide error bars around the line plots.\n* Figure 8 - I think it is a stretch to say that your technique best preserves knowledge in the base model, since it leads to worse performance on unrelated questions than uniform weighting.\n* Line 498-499: calling the technique \"general purpose\" by evaluating three datasets (in one task!) is too strong, I would reword this.\n* I understand that using Distil-GPT2 for the proxy model makes sense from an efficiency perspective, but are there performance improvements from increasing the size of proxy model?\n", "paper_topic_and_main_contributions": "This paper proposes a new online method for adapting langauge models to streams of text.\n\nThe authors propose a meta-learning approach with three general steps:\n    1) a weighting model produces weights over the text\n    2) the model is updated with respect to those weights\n    3) the weighting model is updated with respect to the adapted model's ability to answer questions about the document\n\nThe paper provides ablations of their technique, and analyze its properties (such as which POS tags it upweights) as well as the effect of model on forgetting and answer retention.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I could find.", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "None"}, "extra_text": {}}, {"review_id": "fJ5iMkUCik", "reviewer": "Reviewer_jgxp", "summary": "", "questions": "Question A: Following my reasons to reject, I'd like to see if you have other concerns about other methods and datasets.\n\nQuestion B: Have you tried other proxy models? (bigger size) I understand there are efficiency problems, but I wonder if bigger proxy models can improve the performance.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We're glad you liked the generality of CaMeLS as knowledge updater for pre-trained LLMs.\n\n**Comparisons to other methods:**\nFew-shot prompting is not applicable to our problem setting, where the model must adapt in an *unsupervised* manner to a stream of many (1000+) documents, and is presented with questions about them only at the end of the stream (some supervised data is only assumed present at *meta-training* time). Recent work [1] shows that even models with context lengths long enough to condition on this many documents are not generally able to effectively leverage such long contexts. Additionally, [2] show that gradient-based/parametric model updates can more robustly reason over knowledge compared to in-context learning, which can be more susceptible to distractor facts.\n\nWhile retrieval is one potential approach to updating LLMs over time, we note that successful use of retrieval is very difficult; recent work [3] finds that LLM-powered search engines only provide a citation for 51.5% of the sentences in LLM-generated responses, and more than 25% of citations do not actually support the associated LLM-generated claim! Further, retrieval adds significant system complexity, making edge deployment far more difficult compared with simple quantized LLM inference that allows large models such as llama-65B to be deployed locally on a user's laptop [4]. For these reasons, we focus on parametric updates to the model's knowledge in this paper.\n\n**Analysis of generation ability:**\nOur current experiments involve unsupervised adaptation of question-answering models. Therefore evaluations of generation quality on other tasks may be uninformative. However, similar unsupervised adaptation of generic generative LMs is an exciting direction for future work!\n\n**Question B**: Please see the answer given to reviewer kJwz.\n\n\n\n[1] Lost in the Middle: How Language Models Use Long Contexts. Liu et al., 2023. arXiv.\\\n[2] RECKONING: Reasoning through Dynamic Knowledge Encoding. Zeming Chen et al., 2023. arXiv.\\\n[3] Evaluating Verifiability in Generative Search Engines. Liu et al., 2023. arXiv.\\\n[4] llama.cpp. Georgi Gerganov, 2023. GitHub. https://github.com/ggerganov/llama.cpp", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Unsupervised adaption is an important problem in modern LLMs and this paper's solution has the potential as a general dynamic adapter for deployed static LLMs;\n\n2. The CaMeLS shows strong performance in knowledge retention;", "reasons_to_reject": "1. The paper lacks comparisons with some baselines, for example, retrieval-augmented, few-shot, and in-context learning methods. When there are some labeled examples, these are other natural solutions;\n\n2. The paper lacks further analysis of the model's generation ability after online adaption. Authors have considered this in Equations (3), but some experiments on other irrelevant datasets (such as SST-2) may help.", "questions_to_authors": "Question A: Following my reasons to reject, I'd like to see if you have other concerns about other methods and datasets.\n\nQuestion B: Have you tried other proxy models? (bigger size) I understand there are efficiency problems, but I wonder if bigger proxy models can improve the performance.", "paper_topic_and_main_contributions": "The paper proposes a new method, Context-aware Meta-learned Loss Scaling (CaMeLS), to dynamically update factual information in online streams. Specifically,  a small auto-regressive meta-network is trained to re-weight the NLL loss for each token. The meta-network is trained by the labeled dataset to help the proxy model adapt factual retention and maintain model behavior at the same time. The experiments show that CaMeLS can learn new knowledge without losing previous abilities.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bgMls3ezPz", "reviewer": "Reviewer_2YUV", "summary": "", "questions": "Question A: Since a training set is required in your setting, what are the key differences between Online Adaptation and Transfer Learning? \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We're glad you appreciated the innovation and effectiveness of meta-learning for online LM adaptation.\n\nRegarding the question of unsupervised online adaptation, we would like to clarify that while the meta-training procedure utilizes training data with labels (in a relatively common format of document + question + answer), adaptation itself is indeed assumed to be unsupervised, that is, the model is fine-tuned only with unlabeled documents (unlike supervised transfer learning, in which we would typically fine-tune on labeled examples). However, even the meta-training process does not include explicit labels of the *important information in each document*; thus in this sense we might consider even the meta-learning phase as well to be only distantly supervised. Further, our experiments show that the result of this distantly-supervised stage can be re-used for many models, so this stage only needs to be performed once, and we are able to re-use the produced weighting model to update many different LMs in an unsupervised fashion. We will clarify this distinction and nomenclature in the revised version of the paper!\n\nRegarding the realism of the setting, we note that fine-tuning on new data without forgetting has long been considered a fundamental challenge for neural networks, e.g. since McCloskey and Cohen's pioneering work on catastrophic forgetting [1]. Numerous subsequent works have validated the difficulty of continual learning of neural networks without forgetting, e.g. [2], or [3] for a review. While increasing model scale has been shown to reduce catastrophic forgetting [4], even very large language models show non-trivial forgetting: Fig. 6 in [4] shows a reduction in language generation task performance of over 60% even for a 62B parameter model (similar in size to the largest open-source language models) after adapting to a new suite of embodied tasks. Even a 540B parameter model shows a roughly 4% degradation in performance after adaptation.\n\nFor these reasons, we do not find it surprising that fine-tuning on new documents leads to degradation of previous knowledge.\n\nRegarding in-context learning, we note that our setting involves unsupervised adaptation to a long sequence of documents (more than one thousand), which is much larger than GPT-4's context window. Even if the model had a longer context window, [5] shows that models with context lengths long enough to condition on this many documents are not generally able to effectively leverage the information in long contexts. Finally, [6] show that gradient-based \nadaptation can provide more robust uptake of knowledge compared with in-context learning, motivating our emphasis on parametric knowledge updating.\n\nFinally, thanks for the useful suggestions on improving the paper presentation.\n\n[1] Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Michael McCloskey, Neal J. Cohen, 1989. Psychology of Learning and Motivation.\\\n[2] Overcoming catastrophic forgetting in neural networks. James Kirkpatrick, et al., 2017. Proceedings of the National Academy of Sciences.\\\n[3] Measuring catastrophic forgetting in neural networks. Ronald Kemker et al., 2017. arXiv.\\\n[4] PaLM-E: An Embodied Multimodal Language Model. Danny Driess et al., 2023. arXiv.\\\n[5] Lost in the Middle: How Language Models Use Long Contexts. Liu et al., 2023. arXiv.\\\n[6] RECKONING: Reasoning through Dynamic Knowledge Encoding. Zeming Chen et al., 2023. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introducing meta learning for unsupervised online language model adaptation is innotative and reasonable.\n\n2. The proposed method only requires training a much smaller proxy model for token weight estimation. And the quantitative and qualitative results show that the proposed method improves the model ability on new quires given new documents.", "reasons_to_reject": "1. The setting of Unsupervised Online Adaptation is a little bit strange. As described in Sec 3.1, the model requires a training set, including documents, quires and labels. It seems that the adaptation process is NOT \"Unsupervised\" because the training set also requires annatations.\n\n2. The problem that this paper focuses on may be unrealistic. Figure 8 shows that adapting to test documents leads to performance degradation on unrelated queries. In practice, we expect to update the knowledge of Large Language Models without affecting the performance on general tasks. Besides, existing large scale QA systems, e.g. GPT-4, show strong In-Context Learning abilities. In other words, the model can reason about new documents and answer questions that have never been seen before.", "questions_to_authors": "Question A: Since a training set is required in your setting, what are the key differences between Online Adaptation and Transfer Learning? \n", "paper_topic_and_main_contributions": "This paper considers the unsupervised online language model adaptation setting and proposes a meta-learning algorithm called (CaMeLS) that learns an importance weighting model to reweight the per-token loss of the online data stream. The experiments show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "1. More examples such as Figure 1 can be provided.\n\n2. A summarization of the algorithm can be given."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes learning which tokens to upweight in the training of language models, in order to adapt the model to new sources of information. They propose Context-aware Meta-learned Loss Scaling (CaMeLS), which uses an auxiliary language model to learn how to reweight the language modeling loss for each token. 

Reviewers recognize the novelty of this work, and find the method well motivated. Some raise questions about the efficiency of the approach which the author response addresses. More interestingly, a suggested baseline by a reviewer is implemented by the authors and compared to their own results, which still highlights the effectiveness of CaMeLs. Some proposed baselines by the reviewers are also compared with in the author response. Overall the reviewers seem satisfied with the author response.

We hope the authors include the new results and suggested changes in the next iteration of the paper.
Decision: Accept-Main