Long-Range Language Modeling with Selective Cache
Authors: Xinting Huang
Nora Hollenstein
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Modeling
Long Dependency
Long-term Memory
TL;DR: We explore methods to select important key-value pairs that serve as context for language models.

Abstract: The computational cost of transformer-based language models grows quadratically with the sequence length. In this paper, we introduce the selective cache, which stores the selected key-value pairs from the previous context. By selecting important key-value pairs the model makes better use of the cache so that in limited cache size, a longer context history can be stored. We design three kinds of selection methods. The first is based on human language processing. The key-value pairs are selected if they correspond to tokens that are fixated longer, as recorded in eye-tracking-while-reading experiments. We also incorporate the cognitively-inspired selection process into the language model as a trainable process, resulting in two additional methods with improved performance. The selection task is converted into a pruning task so they can be trained with differentiable masks. We demonstrate that the proposed selective cache improves the language modeling performance across different datasets. With the same number of stored key-value pairs (cache size), our selective cache outperforms XL cache and compressive cache by considerable margins.

Long-Range Language Modeling with Selective Cache
Xinting Huang University of Copenhagen wbn969@alumni. ku. dk
Nora Hollenstein University of Copenhagen nora. honlentelltttnnnnehuu ku. dk
Abstract
attention is O(n²). Hence, the computational cost becomes prohibitively expensive or even makes the task infeasible when processing long sequences. Extensive research is performed to address this problem, introducing number of X-former models (Wang et al., 2020; Kitaev et al., 2020; Choroman- ski et al., 2022).
The computational cost of transformer-base language models grows quadratically with the sequence length. In this paper, we introduce the selective cache, which stores the selected key-value pairs from the previous context. By selecting important key-value pairs the model makes better use of the cache so that in lim- ited cache size, a longer context history can be stored, We design three kinds of selection meth- ods. The first is based on human language pro- cessing. The key-value pairs are selected if they correspond tokens that are fixated longer, as recorded in eye-tracking-whin while-reading exper- iments. We also incorporate the cognitively- inspired selection process the language model as trainable process, resulting in two additional methods with improved performance. The selection task is converted into pruning task so they can be trained with differentiable masks. We demonstrate that the proposed se- lective cache improves the language modeling performance across different datasets.' With the same number of stored key-value pairs (cache size), our selective cache outperforms XI. cache (Dai et al., 2019) and compressive cache (Rae et al., 2019) by considerable mar- gins.
While many proposed models successfully miti- gate the problem to some extent, there still open limitations such the storage cost of preserving the whole sequence in memory and the inability be directly inserted into existing pretrained models. In this paper, we propose a novel cache that stores past key-value pairs, namely the selective cache, for auto-regressive language modeling (LM). This method aims to make better use of the cache in- troduced in Transformer-XL (Dai et al.. 2019) by selecting more informative key-value pairs. Dif- ferent from prior works such as (Rae et al., 2019). distills the previous context by extracting or se- lecting, instead of averaging. In other words, some key-value pairs are kept in the cache as it is, while others are simply discarded. It is utilized as easily as the cache in Transformer-Xe XL, that is, the stored keys and values are simply prepended to the current keys and values.
1 Introduction
We explore three selection methods for the cache:
(1) using eye fixation duration as importance scores,
Transformers (Vaswani et al., 2017) have been used the backbone architecture of various models and
(2) using neural networks which learn select, and
(3) replace. The first one uses eye-movement data.
achieve ofae--of-the-ar results in a wide range of tasks (Devlin et al., 2019; Dosovitskiy et al., 2020; OpenAI, 2023; Kirillov et al., 2023). Compared to other architectures such as Recurrent Neural Net- works, its attention mechanism directly connects long-distance elements in the input sequence. Thus, it better captures long-range dependencies. How- ever, it also constitutes the bottleneck in Transform- ers. The time and space complexity of Transformer
We simply consider the total reading time of each word. Long-fixated words are important for lan- guage comprehension (Rayner and Duffy, 1986) and are usually difficult to infer from the context (Rayner et al., 2011). Furthermore, fixation du- ration correlates with the muudient-basee saliency in Transformer language models (Hollenstein and Beinborn, 2021). Thus, selecting key-value pairs associated with long-fixated tokens a cognitively plausible modeling decision. The second method is to train a selector neural network. By learn-
'Our cnde is available at httpps//gitthe: com/ huangxtt9//ellectiiecaceee

---

ing to prune key-value pairs, the selector learns select. It is based on a neural network pruning method (Louizos et al., 2017), which uses differ- entiable stochastic masks and Lo penalty. The se- lector learns to mask out unimportant key-value pairs so it can be used to select them. Finally, the third method uses the same technique but applies the mask and its opposite mask simultancously, in order train a replacer network that directly learns the trade-off. The motivation is to simulate the actual situation where adding new items to the cache results in discarding old ones. addition, we introduce a novel method for training the selec- tor and replacer network. By sampling previous snippet of context to train the network and stopping gradient during cache updating, we avoid backprop- agation through time (BPTT), while the network can still learn to model long-term dependency. Experimental results show that using these meth- ods improves the model's performance, especially the latter two. They bring about considerable per- formance gain against XL cache and compressive cache (we use these two terms to refer to the meth- ods used in Dai et al., 2019 and Rae et al., 2019, respectively) with the same total cache size. Moreover, thanks to the enhanced interpretabil- ity of selective cache, we can directly investigate which tokens are stored. We find that the trained se- lector and replacer tend to keep named entities and rare tokens, which is a common tendency found in many prior works (Sun al., 2021; Wu et al., 2022; Hutchins et al., 2022). We also show that the se- lective cache can preserve information from a very distant context, or even from an infinite distance, depending on the characteristies of the processed document.
sparsity patterns can be learned for each attention head (Sukhbaatar al., 2019; Correia et al., 2019). Some other works (Kitaev et al., 2020; Roy et al., 2021) use clustering techniques to partition ele- ments in the sequence and perform intra-cluster attention. Besides, hierarchical structure (Zhu and Soricut, 2021: Ren et al., 2021) is incorporated into the attention mechanism to reduce computa- tional complexity. Long documents split into segments and processed a recurrent manner. Reck-Reeurreett Transformer (Hutchins et al., 2022) use a set of hidden representations to store past information and use cross cross-attention to interact with them, while Memory Recurrent Transformer (Bulatov et al., 2022) utilizes dedi- cated memory tokens to do so.
also be
Transformer-XL (Dai et al., 2019) achieves this segment-level recurrence by the cache mechanism, which stores the past hidden states and uses them as an extended context when the model processes the next input sequence. Compressive Transformer (Rae et al., 2019) extends the Transformer-XI. with a secondary cache which compresses the old hid- den states, On the other hand, Memorizing Trans- former (Wu et al., 2022) expands the cache to an enormous size, and uses an approximate k-nearest- neighbor (kNN) search retrieve key-value pairs from the cache efficiently. co-former (Martins et al., 2022) utilizes a continuous-ppace attention mechanism and represents the input sequence as a continuous signal, so that the long-term memory can be represented with lower precision. Expire- Span (Sukhbaatar et al., 2021) computes a span for each hidden state that determines how long it should stay in memory.
In this paper, we make use of eye-tracking data, which has been incorporated into many NLP frame- works (Mathias et al., 2020). Eye-tracking data provides advantages to NLP models in terms of both their performance and interpretability. Re- search has demonstrated that incorporating eye- tracking features can enhance prediction accuracy in tasks like named entity recognition (Hollenstein and Zhang, 2019; Tokunaga et al., 2017), part-of- speech tagging (Barrett et al., 2018), sentiment analysis (Mishra et al., 2017) and general NLP benchmark tasks (Khurana et al., 2023). In the meantime, eye-tracking data is utilized to explore the correlation between human behavior and neural attention (Hahn and Keller, 2023; Sood et al., 2020; Brandl and Hollenstein, 2022).
2 Related Work
Transformers have difficulty processing long se- quences. There are lot of works trying address this limitation. Linformer (Wang et al., 2020) and Performer (Choromanski et al., 2022) introduce new self-attention mechanisms, which can approx- imate the original full attention with linear space and time complexity. Many works utilize sparsity to achieve efficient long-range attention, includ- ing strided attention (Child et al., 2019; Beltagy et al., 2020), global attention by a few tokens (Belt- agy et al., 2020; Zaheer et al., 2020; Ravula et al., 2020) and random attention to a limited number of tokens (Zaheer et al., 2020). Moreover, suitable

---

We also use pruning technique based on stochastic masks (Louizos et al., 2017), which is mainly used in neural network pruning (Louizos et al., 2017) and interpretation (Voita et al., 2019; De Cao et al., 2020). Different from these ap- proaches, we aim to improve performance by us- ing it to select important parts of previous context, Even though the technique is used for different pur- poses in different scenarios, it actually does the same thing, i.e., learns what is less important.
will be added to selective cache current step
selective cache
XL cache
imput sequeace
Figure 1: In the selective cache, extend the attention layer with cached representations from previous context.
3 Model
In this paper, we use decoder-only transformers (Vaswani et al., 2017) to perform auto-regressive LM tasks. Long documents split into segments of 512 tokens. The segments are not shuffled and fed into the model sequentially. In other words, the language model processes the document step by step, one segment at a time, as is done in (Dai et al., 2019; Wu et al., 2022).
and of a fixed size like XL cache. each step. the key-value pairs that satisfy a certain criterion are selected and saved into the selective cache. the meantime, the same number of old key-value pairs are discarded. Like XL cache, the selective cache is also non-differemtiable. In this paper, all the models that selective cache also use XL cache in the meantime, The selective cache serves as a secondary cache and selects tokens discarded by XL cache, like the compressive cache (Rac et al., 2019). So the model contains a detailed recent con- text, as well as a distilled context that covers a wider historical period. The cached keys and val- ues from the selective cache are concatenated with those from XL cache and those from the current input. See illustration in Figure 1.
At each step, Transformer-Xi XL. (Dai et al., 2019) caches and fixes (stops gradient) the hidden state sequence computed for the current segment. In the next step, it is reused to extend the context. Although in (Dai et al., 2019), the hidden states are cached, we follow the practice in (Wu et al., 2022) and save the key-value pairs into the cache for the purpose of efficiency. When doing attention, the keys and values are prepended to the current keys and values. When the XL cache size Cut is greater than the input segment length Cimp. the XL cache is first-in-fisttouut (FIFO) queue.
We also introduce set of trainable bias param- eters, the selective cache bias. They are used the same manner as T5 position embeddings. Re- call that T5 position embeddings are scalars added to the dot product of queries and keys. When the query attends to the keys in selective cache, the po- sition embedding that corresponds to the maximum distance is used. Mcanwhile, the selective cache bias (also a scalar) is added to the dot product as well. For each layer, there are m bias parameters, where m is the number of heads. So the same bias is used for all attention scores of one head.
When using cached representations, is nec- essary to use relative position embeddings. We use T5 relative position embeddings (Raffel et al., 2020), which adds different biases to different rela- tive offsets when doing attention.
When switching new document, the cache may contain some content from the old document, we apply document masks to solve this problem. Concretely, for the cached key-value pairs, we keep track of their document IDs. Each token can only attend to other tokens with the same document ID. These the common settings used for all experiments involving the cache, including base- lines and proposed models.
3.2 Using Eye-fixation Duration as Criterion
A crucial component of the selective cache is the selection criterion. One plausible choice is human fixation data. While eye-tracking data contains various information about human eye movements during reading, we simply utilize the total read- ing time (TRT) of each word, which is the sum of all fixation duration on a single word. If an input token is associated with a TRT longer than a cer-
3.1 Selective Cache
We aim make better use of the cache by select- ing those key-value pairs which are more beneficial than others. The selective cache is FIFO queue

---

tain threshold, its corresponding key-value pairs are selected into the cache in all attention layers. The threshold is a hyperparameter that needs to be tuned. Because human reading is optimized for ef- ficiency and accuracy, they only fixate words to the extent necessary for task success (Hahn and Keller, 2023). Long fixated words are usually important and are likely to contain new information which cannot be inferred from the context.
Binary Concrete
lggu =
25-
hart Danadd
38-
13-
3.-
65-
The probability density function of Binary Concrete distribution and Hard Concrete dis- tribution. In this example, log = 0, 3 = 0.5.
Figure 2: Left:
However, there is no available eye-tracking data for the LM datasets used in this paper. We choose use simple long short-term memory (LSTM) network to predict the fixation duration for the text in LM datasets. tracking corpora. After training it is able to predict the TRT within an acceptable error range (see Ap- pendix B). The fixation prediction model only pro- cesses the text once at the preprocessing stage. The predicted fixation duration is used repeatedly for many epochs when training the language model.
-011 = 1.1, and p(z = 0) = p(= = 1) = 0.23.
Right: Hard Concrete distribution with different log a when 3 = 0.4, = 0.1, 1.1.
trained on the available eye-
dicted masks.
||=110 Σ 1 [R 5=1
(2)
where ||=||o denotes Lo norm, 1(-) stands for in- dicator function. Lo norm penalizes the number of non-zero values. It imposes a constant penalty everywhere except for 2j = 0. It encourages the selector to completely switch off some key-value pairs.
3.3 Using Selector Network Criterion
We introduce an automatic selector, which gives free rein to the model and let it decide which to- kens should be cached. It is small neural network integrated into the self-attention layer. The selec- tion task is converted into a task similar to network pruning. The language model is provided with some previous key-value pairs when doing the LM task, the selector is encouraged to prune some of the key-value pairs. Its architecture described in Appendix E.
As can see, Lo norm is not a differentiable function and has zero derivative almost everywhere. Meanwhile, the outputs 2j are not produced from a continuous function either.
Stochastic Masks The solution to these two prob- lems is to use the stochastic masks drawn from some distributions controlled by the selector, and the Lo norm becomes the expected number of non- masks. More specifically, we use Hard Con- crete distribution (Louizos al., 2017), which is mixture of discrete and continuous distributions, ranging in the closed interval [0, 1). gives non- zero probability to the two endpoints 0, 1, while between 0 and 1 the probability density function is continuous, shown in Figure 2. The Hard Con- crete distribution contains few parameters, one of which is log It controls the probability mass skewing towards 0 or 1. See Appendix C for more information.
Binary Masks For a sequence of past hidden (the time index is omitted here for simplicity), the selector network takes the hidden states as input and outputs binary masks for each of them [21, 22, ... =n). The masks are used in self-attention follows:
states [h, Tyg... ... .RA]] ,
Sij = q k, + In
where s, k are the attention score before softmax, the query, and the key respectively. So when 2j is zero, the corresponding key-value pair is masked out, when 2j is one, the attention score is not af- fected. Note that here the position embeddings, document masks, and selective cache bias are all omitted for simplicity.
With the Hard Concrete distribution, the selec- tor is trained as follows: On the one hand, the network outputs the parameter log a, for each in- put hidden state hy. Then one sample zj is drawn from each distribution and is used to mask attention scores as Equation With the reparameterization trick (Kingma and Welling, 2013), the gradient can backpropagate through the sampling process
In the meantime, Lo norm is applied to the pre-

---

the selector network. The training objective of language modeling would encourage the selector to output larger log a, so that extra information can be obtained to help predict the next token. On the other hand, To norm is treated in expectation, which becomes differentiable function of log a. It encourages the selector to output smaller log a. With the influence of both aspects, important key-value pairs are likely be kept. When the selector is used as the selection criterion, the log aj is compared with threshold O. If log aj > 0, kj and ūj are put into the selective cache.
For this reason, it is possible that it selects very similar representations. To address this issue, we calculate the distance matrix of the selected values in each step. Values with a distance smaller than a certain threshold are considered in the same "clus- ter". Only one key-value pair in each "cluster" is finally selected. See more details in Appendix F. In this process, we only calculate the distance matrix once among the selected values. This process be regarded as a secondary selection. In the follow- ing part of the paper, we refer this technique as removal of similar instances (RSI).
When using selector, the training loss becomes:
3.5 Replacement- Selective Cache
L = LIM + ACo
(3)
When training the selector network, the Lo penalty implies the capacity limit of the selective cache. Because the cache size is fixed, storing some new key-value pairs means discarding some old ones. Inspired by this process, further propose a way to directly model this trade-off. Concretely, the kcy-value pairs are grouped two by two, neu- ral network compares the two key-value pairs and masks out either of them.
where L is the total loss, LLas is the language modeling loss. Lo represents the Lo norm. À is the coefficient. A key problem is determine We design an adaptive follows:
#sclected
X = X Tt = 1
*0.1 (4) #total
where Tt is a rumning average of the selection ratio, d is the coefficient used at step t. is a hyperpa- rameter. In this way, if the selection ratio grows too big, the "suppression force" also increases.
Suppose that the representations at position j and j' are compared. The approach calculates stochas- tic masks as follows.
log U, ( At,) frp Aty)
(5)
Simultaneously Selecting and Training The se- lective cache stores context of multiple previous steps, one way to train the selector is to backpropa- gate gradient through all relevant steps. We avoid this by separating the selecting and training process. We sample a random snippet of length Camp = 128 from previous context to train the selector, i.e., per- form the pruning task, while it also performs se- lection as non-differentiable process. In other words, during training, some additional keys and values are prepended to the concatenation of selec- tive cache, XL. cache, and current keys and values. We control how distant the snippet is through a geometric distribution parameterized by a hyperpa- rameter Z. It should be determined based on how long the dependency we intend to capture. See more details in Appendix D.
2j3' HardConcrete(log ogjy, 3, 7, (6)
where (-) denotes the neural network trained (we refer to it as the replacer network, see its ar- chitecture in Appendix E), Alj is the time interval between the current step and the previous step when processing position j. HardConcrete represents the Hard Concrete distribution, B, 7.5 S are constant pa- rameters. Then the sampled random mask Ejy is used as follows.
8ij T kj In 2jy" Siy = Th by + In(1 -25²)
(7)
(8)
Therefore, ideally, only of the two can be attended. Intuitively, if the key ky and value Ty provide more important information for the current step. the replacer network should output a higher "score" (hyj Atz) which results in high log Ogy- The cost, on the other hand, is masking out ky and Ty.
3.4 Removal of Similar Instances
In preliminary experiments of training the selector, we find that some tokens are repeatedly selected. Because the selector network makes decisions on leven- it does not have the overall information, e.g., which kind of information is already selected.
However, this one-to-one relationship deviates from the actual situation. Adding one into the cache does not necessarily result in removing the other,

---

the removed key-value pair could be any one in the cache. Nevertheless, we still opt use this relationship for its simplicity.
WikiText-2 The WikiText language modeling dataset (Merity et al., 2016) consists of articles from Wikipedia. It provides two sizes, WikiText-2 and WikiText- 103. They share the same validation and test set, while the training set of WikiText-2 is truncated version of that of WikiText-103.
When training the replacer, there is no need for Lo penalty. We again use a random snippet of previous context for training. A random set of key- value pairs in selective cache is chosen to do the one-to-one comparison with those in the snippet. The replacer network outputs masks, and each of the masks and its opposite mask are applied to one in the snippet and one in the selective cache respectively.
CMU-DoG CMU Document Grounded Con- versations Dataset (Zhou et al., 2018) provides conversation data about the contents of specified documents. Following (Martins et al., 2022), we also use it to test long-range modeling, The dataset contains conversations about movies, the Wikipedia article about the movie is also provided. In this paper, to test the models' ability modeling Long-dependency, we concatenate all conversations to their corresponding Wikipedia articles. The resulting document takes the following form: WWikiArricleeCCovvrsttaaii Converstation 1, 1,... Convverttttion n). Note that the Wikipedia article accounts for only a small portion (1.3%) of a concatenated document on average.
When updating, the replacement-based selec- tive cache is not a first-inffirst-ne queue anymore. The input sequence is compared with the selec- tive cache element-wise. The key-value pair with higher "score" (hy. Atz) is retained. Impor- tantly, when switching to a new document, the key-value pair from the new document is always preserved no matter what the replacer outputs.
4 Experiments
We evaluate the models described above on three datasets using two different model sizes, The ex- periments are all language modeling and the per- formance evaluated by perplexity per token. Be- sides, we also do various qualitative and quantita- tive analyses on the models. From these analyses, we provide strong evidence of the effectiveness of the proposed models.
4.2 Experimental Methods
Due to limited computational resources, we do not train the language models from scratch. We fine- tune a pretrained language model. We extract the decoder of T5v 15v1.1-LM. 1-1.111----aaaapp 2, and remove the encoder-decoder attention layers from it. It is a version of T5 (Raffel et al., 2020) adapted to LM objective during pretraining. We choose it because not only uses relative position embedding which is necessary for our purpose, but also is available in a wide range of sizes. Its Small and Base sizes suit our budget. Even though taking only part of seems crude, we find it works well on LM tasks in general, even before our fine-tuning.
4.1 Data
We introduce the language modeling datasets used in the experiments. The eye-tracking corpora are described in Appendix A
PG-19 PG-19 language modeling benchmark is a LM dataset introduced by Rae et al. (2019). It includes collection of English books published be- fore 1919 from the Project Gutenberg books library. Each document in the PG-19 a full-length book. However, due to limited computational resources, we only use 1% of the training set, which is 286 books, around 18M tokens. The documents are randomly selected and then fixed. So the same set of documents is used for all models. On the other hand, we use full validation and test set, which con- tain 3M and 7M tokens respectively. Even though only 1% is used, the training data still has a reason- able size and is enough for fine-tunie fine-nuning.
We fine-tune two sizes of (15v1.-LL--adapl namely Small and Base. We simply refer to them as T5 Small and T5 Base. The former has layers and heads, and the latter has 12 layers and 12 heads.
The input size is always Cipp 512 tokens. There are two groups of experiments with total cache size of 512 and 640 respectively. In the first group one baseline pure XI. cache of size O₂ = 512, in other words, same as Transformer- XL (Dai et al., 2019). The other baseline is Com- pressive Transformer (Rac et al., 2019). We imple-
rranffrmnchhhb/mmii/cccccad
http:///wwinnnccccccccccccccnnsa checkpoints.md

---

PG-19
WikiTex12
CMU-DoG
XL
2nd
Small
Base
Small
Base
Small
Base
Vanilla Transformer
30.85
25.24
22.30
17.56
20.77
17.22
XL
512
29.89
24.38
19.85
16.31
20.15
17.19
XL, CPR
256
256
29.39
24.18
19.95
16.17
20.09
17.02
XL, SLC (fix)
256
256
29.32
24.04
19.78
16.27
19.97
16.75
XL, SLC (fix,RSI)
2.66
256
29.46
23.99
19.77
16.19
19.83
16.63
XL. SLC (slc)
128
384
29.08
24.14
19.13
15.65
19.12
16.11
XL, SLC (sIc,RSI)
128
384
28.89
23.83
19.05
15.60
18.98
16.15
XL
640
29.62
24.43
19.77
16.27
20.23
17.41
XL, CPR
384
256
29.38
24.21
19.83
16.14
20.20
17.16
XL, R-SLC
128
512
28.50
23.63
18.57
15.20
17.95
14.82
Table Test perplexity on three datasets. Columns "XL" and "2nd" denotes the XL cache size and secondary cache size. "XL", " "SLC", "CPR" in rows refer to XL, selective and compressive cache respectively. "(fix)" means using fixation duration as selection criterion. "(slc)" means using selector network. "R-SLC" stands for replacement- based selective cache.
ment their compressive cache our experimental framework and use the best confi configuration reported in their paper. For each model, we choose the best cache size configuration between (Czt, C2nd) (256, 256) or (128, 384), where C2nd is the size of the secondary cache. In other words, given the total size limit, we treat the cache sizes hyperparam- eters and search for the best one. In the second group the total cache size 640 tokens. Because our replacement bassee selective cache needs to be greater or equal to the input size, i.e., >= 512, and meanwhile, XL cache plays an important role so it is necessary to have a minimum size of it. Thus use (Czt, C2ma) = (128, 512). For compres- sive cache, choose the best configuration be- tween (Car, O2nd) (128, 512) or (84), or (384, 256). See more details Appendix G.
cache size, the configuration cache size shown in the table is the optimal configuration. More- over, replacement-baate selective cache produces even larger improvements. In general, the most substantial performance gains are the concate- nated CMU-DoG dataset. While using XL cache and compressive cache only slightly reduces the perplexity, using selective cache reduces the per- plexity by substantial margins. Regarding RSI, it slightly improves the performance in general. It appears to be more effective when the selector is used. It is probably because the key-value pairs selected are more homogeneous in that case.
In addition, the compressive cache shows smaller improvements on PG-19 compared to (Rae et al., 201 9), we think is because of different experimen- tal settings. For example, models while they train from scratch; we keep the total cache size the same when comparing models; we use smaller models and datasets.
fine-tune pretrained
4.3 Results
Table shows the results on three datasets (see Appendix H for statistical significance). We can clearly see that using selective cache achieves the best results in all scenarios. Partly replacing XL cache with selective cache results in considerable gains across datasets and model sizes.
4.4 Analysis
Tokens selected by selector Figure 3 show the in- put tokens corresponding to the selected key-value pairs (See Appendix K. and K.3 for more exam- ples). It's obvious that the selector has strong tendency to select named entities. Other than capi- talized nouns and some rare tokens. it also selects some normal nouns that are keywords of the text, e.g., "exploration". Note that there is no explicit guidance for this behavior during training. It's in- teresting that this pattern is automatically learned with a simple LM objective and a Lo penalty. One
We can see that the fixation based selection sur- passes XL cache, especially on PG-19 dataset, which demonstrates the validity of using eye fixa- tion. It achieves similar performance as compres- sive cache. On the other hand, using the selec- tor network brings about larger improvement, be- cause allows the model to select what needs. Note that this is not because of larger selective

---

expedition and shafies cargerantinn duiing vubuequeent years have added greatly to our knowledge of land wonders Mayor Powellss Severa lee the some year essoyee to explore the Laadd of the colorado together the upper coryons that strem and the great conyans of the lower porton Greon Shan For this purpose employed four rowboats and made the descont from what now Groeeifios statial Brough the wi lo couse Barmals to the mouthol of the Ri Momiun a disann more than shosendd nike,, D halnsiio From Ka Bhilial Looking South the spe ri of 1570) again aer with tree boets and decranded the row to the Croe dn of the fath where med a pook tran and went our with party of men to explore waye down into the Coand
layer proiiuully been swen by Docto Bansiann 1688 Gabert max the grolegid
from the north and devnted the summer fell winter and following spring to this undersiing in the summer of 1871 retumed to the rowbbats and descended through Marble Canyon to the GrandGamyan oittton and then through the greater part of the Grendl Lanyon rtehl subsequent years were then given to exploratio the country adjecont to the Grand Canyoni On these tris Mr. Glbert the oclloged who had boon with Ucutonand Mithelll and Cap a Bumto were my giologia (comaasion On the second bard and duing all the subsequent years of Innnioatioon in this legiin Frot I Thempson was my gigiapiii (omaa vin asdited numbe ol loe grophical arginwer In 1860 M. Wileoll
evdelard the United Suat Seeningt Seasen went with inio the depilts of the Grand Caneeni We Plasenl the north tre softch buil down side descent mode foll tanland emall party Way
desorned from the summit of the
diection loward the mouth of the Uitle ioloraa (leen The ofthe goology/ of
with Mr Maacom ina
with his party.
the
was known when
The selected tokens in PG-19 19 validation set. The impur tokens of the selected key-value pairs are marked with blue. Deep blue means the key-value pairs pass the secondary selection and are stored in selective cache, while light blue means they are selected by selector but then removed by RSI. The model is T5 Small XL, SLC (sIc.RSI).
Figure 3:
possibility is that keeping the named entities in the cache largely facilitates the prediction when the target tokens are themselves. More importantly, it seems that the selector tends to select tokens starting from the second token of noun phrase. For example, "Doctor <Newberry>" (using" "<>" to represent the selection), "Grand <Canyon>> etc. This a reasonable behavior because when the first token of these noun phrases occurs as the input token, the target is very likely to be the sec- ond token. In other words, given that in real data distribution p(2nd token 1st token) is much larger than p(2nd token), successfully modeling this conditional probability can reduce perplexity sig- nificantly. But it is hard without seeing any prece- dents. Keeping these tokens in cache largely helps to model this kind of conditional probability. From these examples, we can see that the selector tries to make the best use of the cache.
tantly, this retrieval-based working pattern is in line with prior works (Sun et al., 2021; Wu et al., 2022; Hutchins et al., 2022). Therefore, this pattern is not because of the design of selective cache, but rather general tendency.
Replacement process As for replacement-based selective cache, we calculate the replacement ra- tio over time. As expect, we find that the placement ratio is very high at the beginning of a document, then it quickly decreases and finally sta- bilizes around a certain level. This is because the density of important key-value pairs in the cache increases over time. Interestingly, on the concate- nated CUM-DoG dataset, the replacement ratio stabilizes around zero, which means it preserves the beginning of the document for many steps. Im- portantly, we observe the same tendency as when using the selector. Most of the newly added to- kens in each step belong to named entities. See Appendix L.
Effect of RSI In the examples shown, RSI seems to work well, at least from the observation of input tokens. The selector repeatedly selects "Canyon". By RSI, only the last one is kept (bottom right cor- ner), and all other "Canyon" tokens are removed. To some extent, it avoids storing similar informa- tion in the cache repeatedly, thus saving space for other kinds of representations that carry more di- versified information.
Fixation- based selection The fixation duration is predicted by a separate LSTM network. We ex- amine the selected tokens and find that the fixation duration is reasonable in general (see Appendix M). It does not simply stick with a single kind of token such as named entities or rare words. It usually focuses on the core actions in the sentence and the subject and the object of the action, as well as some relevant adjectives and dates. Therefore, it largely reflects human reading behavior (Tokunaga et al., 2017).
Effect token-level loss We also examine the selective cache from the output side, i.e., the token- level cross-entropy loss. We find that the model makes worse predictions on named entities after masking out part of the selective cache. This, to some extent, explains where the overall perplexity improvements come from. See Appendix K.2.
Regarding the reason why fixation-based selec- tion is good but not good enough, we think it mainly lies in the discrepancy between the LM task and the task that humans are doing. On the one hand, humans are performing a task of language comprehension when reading. On the other hand, a lookup table for noun phrases could be more advan-
Combining the findings from both sides, we find the model mainly uses selective cache to store rare tokens or names of characters and places. Impor-

---

tageous for prediction when they appear multiple times.
ter. We thank the anonymous reviewers for their thoughtful comments on the paper.
5 Conclusion
References
In this work, we propose the selective cache that stores selected key-value pairs from previous con- text for language models, which significantly in- creases the length of the context that a language model can attend to. Moreover, we propose three different ways to select important key-value pairs, namely using fixation duration, the selector net- work and the replacer network. The experimen- tal results from different datasets and model sizes demonstrate the effectiveness of the proposed ap- proaches. Without increasing total cache size, the selective cache outperforms XL cache by consider- able margins. Moreover, further analyses reveal im- portant characteristics of the selective cache, such as the neural network-based selection tends to se- lect named entities and rare tokens, as shown by the increased interpretability of our models.
Abien Fred Agarap. 2018, Deep learning using rectified linear units (relu). arXiv preprint arXiv: 08375.
Maria Barrett, Ana Valeria González-Garduuio, Lea Fr- ermann, and Anders Søgaard. 2018. Unsupervised induction of linguistic categories with records of read- ing, speaking, and writing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 2028 2028-2038, New Orleans, Louisiana. Association for Computational Linguisties.
Beltagy, Matthew Peters, and Arman Cohan. 2020. Longformer: The donummecument transformer. arXiv preprint arXh::2004.05755
Stephanie Brandl and Nora Hollenstein. 2022. Every word counts: A multilingual analysis of individual human alignment with model attention. In Proceed- ings of the 2nd Conference of the Asia-Pacijie Chap- tr of the Association for Computational Linguistics and the 12th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 72-77, Online only. Association for Computa- tional Linguistics.
Limitations
Due to limited computational resources, eval- uate the proposed selective cache by fine-tuning pretrained models on small datasets. In contrast, most of other relevant works train models from scratch on very large language modeling datasets. It is possible that the selective cache performs bet- ter or worse in different settings. We cannot di- rectly compare our results with other related work because of the same reason. Moreover, among all the datasets commonly used to evaluate long-ranp rangrange models, we only use PG-19. Other datasets include character-level LM datasets text8 and enwik8 (Ma- honey, 2011 1), source code datasets GitHub (Wu et al., 2022). We have to train from scratch we evaluate models on those datasets. Finally, while processing long sequences is an ability needed in many applications, we only apply the selective cache language modeling tasks. The effect of selective cache on other modalities, such speech and time series is also worth investigating. We hope that future work will continue to evaluate our mod- els on a wider range datasets and experimental settings.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing $ystems,35:11077 35:11079- 11091.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1944.15009
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiuddin. Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022, Rethinking attention with per- formers.
Uschi Cop. Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting geco: An eyetracking cor- pus of monolingual and bilingual sentence reading. Behavior research methods, 49(2):602-615.
Gonçalo M Correia, Vlad Niculae, and André FT Mar- tins. 2019. Adaptively sparse transformers. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 2174-215 2184.
Acknowledgements
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond
We acknowledge the computing resources provided at the UCloud platform at SDU eScience Cen-

---

a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
transformers. Advances in Neural Information Pro- cessing Systems,
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categori- cal reparameterization with gumbel-softmax. arXiv preprint arXiv:161
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3243-3255.
Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The dundee corpus, In Proceedings of the 12th European conference on eye movement.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North Americun Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Varun Khurana, Yaman Kumar, Nora Hollenstein, Ra- jesh Kumar, and Balaji Krishnamurthy. 2023. Syn- thesizing human gaze feedback for improved nlp per- formance. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 1887-1900.
Diederik P Kingma and Max Welling, 2013. Auto- encoding variational bayes. arXiv preprint arXiv: 1312.6111
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tele Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything. arXiv:2304.0006 300443433..
Michael Hahn and Frank Keller. 2023. Modeling task effects in human reading with neural network-based attention. Cognition, 235.995289.
Nikita Kitaev, Lukasz Kaiser, and Auselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprimt arXiv:2001.044
Nora Hollenstein and Lisa Beinborn. 2021. Relative importance in sentence processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume Short Pupers), pages 141-150, Online. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018 SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Nora Hollenstein, Jonathan Rotsztejn, Marius Troen- dle, Andreas Pedroni. Ce Zhang, and Nicolas Langer. 2018. 7uco, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific data, 5(1):1-13.
Iyya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization, In International Confer- ence on Learning Representations.
Christos Louizos, Max Welling, and Diederik P Kingma. 2017. Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv: 1712.0132..
Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020. ZuCo 2.0: A dataset of physi- ological recordings during natural reading and anno- tation. In Proceedings of the Twelfth Language Re- sources and Evaluation Conference, pages 138-146, Marseille, France. European Language Resources Association.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relax- ation of discrete random variables. In International Conference on Learning Representations.
Matt Mahoney. 2011. Large text compression bench- mark.
Nora Hollenstein and Ce Zhang. 2019. Entity recog- nition first sight: Improving NER with eye move- ment information. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 1-10, Minneapolis, Minnesota. Associa- tion for Computational Linguistics.
Pedro Henrique Martins, Zita Marinho, and André FT Martins. 2022. -former: Infinite memory transformer- former: Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 5468-5466 5485.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent
Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharya. 2020. A survey on using gaze

---

behaviour for natural language processing. In Pro- ceedings of the Twenty-Ninn International Joint Con- ference on Artificial Intelligence, IJCAI-20, pages 4907-4913. International Joint Conferences on Arti- ficial Intelligence Organization. Survey track.
Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang. Jure Leskovec. Dale Schuurmans, and Bo Dai. 2021. Combiner: Full attention transformer with sparse computation cost. Advances in Neural Information Processing Systems, 34:22470-223 22482..
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53- 68.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations.
Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sareasm classification using convo- lutional neural network. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 377-387.
Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202
Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc Thang Vu. 2020. Interpreting attention models with human visual attention in ma- chine reading comprehension. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learning, pages 12-25. Sainbayar Sukhbaatar, Édouard Grave, Piotr Bo- janowski. and Armand Joulin. 2019. Adaptive at- tention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 331-335.
OpenAT. 2023. Gpt-4 technical report.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan. 2021. Not all memories are created equal: Learning to forget by expiring. In International Conference Machine Learning, pages 9902-9912 PMLR.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019. Com- pressive transformers for long-range sequence mod- elling. In International Conference on Learning Rep- resentations.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella- Micke, and Mohit lyyer. 2021. Do long-range lan- guage models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807- 822.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667.
Takenobu Tokunaga, Hitoshi Nishikawa, and Tomoya Iwakura. 2017. An eye-tracking study of named entity annotation. In RANLP, pages 758-764
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Anirudh Ravula, Chris Alberti, Joshua Ainslie, Li Yang, Philip Minh Pham, Qifan Wang, Santiago Ontanon, Sumit Kumar Sanghai, Vaclav Cvicek, and Zach Fisher. 2020. Ete: Encoding long and structured inputs in transformers. In 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP 2020).
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov, 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Keith Rayner and Susan A Duffy. 1986. Lexical com- plexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & cognition, 191-211
Sinong Wang. Belinda Z Li. Madian Khabsa, Han Fang. and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint a0Xiv:2006. 04766.
Keith Rayner, Timothy| J Slattery, Denis Drieghe, and Simon P Liversedge. 2011. Eye movements and word skipping during reading: effects of word length and predictability. Journal of Experimental Psychology: Human Perception and Performance, 37())5144
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,

---

Clara Ma. Yacine Jernite. Julien Plu, Canwen Xu. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
A Eye-tracking Corpora
To train fixation prediction model, we gather eye-tracking data from four eye-tracking Corpora. Namely, Dundee (Kennedy et al., 2003), GECO (Cop et al., 2017), ZuCol (Hollenstein al., 2018), and ZuCo2 (Hollenstein et al., 2020). We only take the eye-tracking data during natural reading En- glish text. The collected data comprises multiple domains such as news, novels, movie reviews, and Wikipedia articles. All data is recorded by profes- sional researchers and equipment, with minimum of 10 subjects reading the same text. There are 1.6 million tokens in total (including repeated text read by different subjects). The total reading time (TRT) of each word is normalized across corpora and av- eraged over subjects, and then evenly mapped to {0, 1, ,,,,,,,111
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy, 2022. Memorizing transform- ers. In International Conference on Learning Repre- seniations.
Manzil 7aheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 11111283-17999
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713.
B Fixation Prediction Model
We train a fixation prediction model, which uses the same tokenizer and embedding layer as the main transformer language model (the embedding layer is frozen for the sake of generalization on unseen tokens). The model predicts fixation duration on teven- The original word-level fixation dura- tion is converted token-level in order to train the model. The conversion is slightly complex: TRT of a word first assigned to each character of it, then a small number is assigned to the last character of the word (mainly to give small values to punctua- tion). After tokenizing the word we obtain the span of each subword, and take the maximum value in the span to get the final token-level fixation data. We do it in this way because the tokenizer we are using only provides character-level span informa- tion. This is not a perfect solution but works fine in most cases.
Zhenhai Zhu and Radu Soricut. 2021. H- H-transformer- 1d: Fast mme-dimensional hierarchical attention for sequences. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1Ith International Joint Conference on Natu- ral Language Processing (Volume I: Long Pupers), pages 3801-3815.
The fixation prediction model consists of an em- bedding layer (TS embedding), two-layer bidirec- tional LSTM, and one- ame-hidden-layer MLP on top of it. The best model achieves an MSE of 4.02 on randomly held-out test set (25% of all data).
C Hard Concrete Distribution
The Hard Concrete distribution is based on Bi- nary Concrete distribution (Maddison al., 2016; Jang al., 2016), which is defined on the interval (0, 1) and can be regarded as a relaxed version of Bernoulli distribution (see Figure 2). The Binary Concrete distribution is parameterized by log a and

---

B. The location parameter log controls the prob- ability mass skewing towards 0 or 1, while 3 con- trols how sharp the probability density is, or the degree of approximation to a real Bernoulli distri- bution. The following function can be used to draw samples from this distribution:
In this paper, we follow the recommendation in (Louizos et al., 2017) and use 3 = 2/3, 0.1, < = 1.1 when training the model with the selector network. Thus the selector only predicts log a.
D Training Selector over Previous Snippets
Sigmoid (Oogg log (1 u) + log B)
To train the selector, the model maintains list of previous snippets. Two hyperparameters' / and Comp control the length of the list (the number of snippets stored) and the size of snippets respectively. Note that in this paper, Coup 128. At the end of the current step, a consecutive sub-sequence of hidden states and key-value pairs are randomly selected from all current hidden states [7 kn]] and all current keys and values [Kr,-.. k.,], [2h T.). [.,),,
(9)
where u is drawn from uniform distribution u 2(0 1), the sample drawn from Binary Concrete distribution. To obtain the Hard Concrete distribution, the Binary Concrete distribution first stretched to (7, interval, where y < 0 and <> 1. It is then rectified to [0, 1].
s s(< - 7) +
(10)
They do not receive gradient as well. If the list is not full, the selected snippet is then appended to the list, otherwise random old snippet is replaced by it. At the beginning of the next step, one random snippet in the list is selected. The stored hidden states are fed into the selector, and the correspond- ing keys and values are prepended to the main key and value matrix. The selector produces masks that control the attention to the snippet. Therefore, the selector is trained on a snippet of the previous context, and the time interval between that context and the current input is random. The probability of selecting a snippet from the kth previous step is (1 - 1) 1 which is a geometric distribution. Thus / should be set according to the average length of documents.
- min(1, main(1,max4 m0 3))
(11)
replaced
where a is the sample drawn from the Hard Con-
crete distribution. In the second step, the prob- ability mass between (7, 0) is "folded" to the 0 point, and the probability mass between (1,5) is also "folded" to the point at 1. The stretching and rectifying operations result in non-zero probabil- ity at the two endpoints, as well as a continuous curve between them. Figure 2 shows how param- eter log a affects the distribution. For more infor- mation about the probability density function and cumulative density function of Hard Concrete dis- tribution, see Appendix B of (Louizos et al., 2017). When using stochastic masks, the Lo norm is treated in expectation. Then Equation 2 becomes:
On the other hand, at the end of each step, the selector selects key-value pairs. But this process (12) does not update the selector. The self-attention layer then learns to attend whatever is selected. During testing, since the selector does not need (13) training, the list of snippets is always empty, and there are no snippets prepended to the current keys and values.
E||l ||||2||0|
E (R (2/)
j=1
Σ p(zy > 0)
j=1
Σ (1 - (0)
(14)
E Architecture of Selector and Replacer Network
where Qsy (-) is the cumulative density function of 8, which is the stretched distribution introduced in Equation 10. The last term is function of its parameters.
The selector network consists of one fully- connected layer, one ReLU activation function (Agarap, 2018), and one fully-connected layer se- quentially. The input size, hidden size, and output size are H, H/4,1 1 respectively, where H is the dimensionality of hidden states of the language model. For each attenntoe layer, there is a sepa-
"
(1-Q 11-Qz, Σ (log a, - log 1
(15)

---

rate selector network, so that the selection criterion is adapted the need of each layer.
%
%
1/2
F2
%
4
1
1
0
0
0
We use the same architecture for the replacer net- work, with the exception that we add time informa- tion. The time interval At is first converted into vector and then concatenated with the hidden states. In mathematical form, [j At 12d:-1 where de is the dimensionality for time embed- ding. In this paper, we use de = 8. So the input size, hidden size, and output size of replacer is H + 8, H/4, 4, 1, where H the dimensionality of the hidden states.
0
0
1
9
0
0
1
1
0
9/
0
0
4
9
0 0 1 19
3
]0
5
Figure 4: In order to further remove similar represen- tations, a pair-wise Euclidean distance matrix is calcu- lated. It is then converted to binary matrix f) whose 1 entries represent that the distance is small enough. It then converted to a strictly lower triangular matrix D. If the sum of a column is zero, the corresponding kcy-value pair kept.
F Detailed Description of RSI
the case of selector network, suppose that the output log a for _h_,-... ,h,] are greater than O, then their corresponding key-value pairs are se- lected, i.e., [,i,,... kn [i....... F,,,,, Vr E {1.... m where r is the index of the head, is the number of heads. Then a distance matrix D E IR is calculated. Each entry Dij the Euclidean distance between the con- catenated values, i... Concate(EE,,. mm and Concate(U,,1 (U,,,, ... m. The reason for choosing values instead of hidden states is that the imple- mentation is simpler. Then the distance matrix is compared with threshold and converted into binary matrix D, whose entries are equal 1 for distances less than the threshold, and 0 other- wise. Therefore, each row or column represents a collection of similar representations. Then its elements along and above the diagonal are set to be O, resulting in a strictly lower triangular matrix D. Finally, the sum of each column is calculated, if DE Li=1 Dij = 0, then kjrr. Tr are remained. If EL Dij O. the corresponding key-value pairs are discarded. The additional computation involved in these operations is mainly on the distance cal- culation. Since this operation is implemented with nigh-peeformance code by most popular deep learn- ing frameworks, and the distance is only calculated for selected representations, it does not add much extra computation.
that in practice, 77 is usually not a small value, so the vectors in the same cluster only bear a limited resemblance.
G Implementation Details
Batching and Processing Figure 5 shows how the model trained with batches of documents. all of our experiments, the documents are shuffled and then packed in this way. The models process a batch of segments at a time, each segment comes from a different document. The segments are al- ways aligned, i.e., the segments at the same position in the batch dimension are consecutive. Therefore. there separate caches for each position in the batch dimension. Because the selected number of key-value pairs different for each cache, we iter- ate over the batch dimension to update each cache respectively. The cache is emptied at the beginning of each epoch, as well as the beginning and the end of the evaluation. When the model encounters new document, e.g., from document A to docu- ment B in Figure 5, the cache is not emptied. It is unnecessary do because the document masks (Section 3) can prevent the model from attending the old document.
During training and validation, the model stops at the shortest row in Figure 5, i.e., at the end of document in this example, for the sake of effi- ciency. During testing, the model finishes all the documents, i.e., stops at the end of document D. The documents are packed in way such that the rows in Figure 5 have lengths that are as sim- ilar to one another as possible. This is important since some documents (especially in PG-19) are
Figure shows the case where vi, dg are similar and U2 U4 i/ are similar.
Again, like the coefficient for Lo norm, find constant threshold is not appropriate. The overall magnitude of distance varies across different layers. Therefore, we use threshold that is determined by the average distance. Specifically, threshold = n n Eig Dij, where v is a hyperparameter. Note

---

segment
document A
document B
document C
document
batch dimension
document
document F
document G
document
document
document
documentt
document
Figure 5: The documents are packed and split into multiple segments. The model processes batch of segments at time, and then moves to the next batch of segments (from left to right). In the same position along the batch dimension, i.e., for each row, the segments are continuous. All the models in this paper are trained and tested in this way.
extremely long, and uneven lengths cause a lot of waste.
epochs instead of training 60% of data for 1 epoch. The reason is that we would like to compare the performance when the models reach convergence. For experiments done on WikiText2 dataset, we train all models that use T5 Small for 50 epochs, with batch size of 32. We train models that use T5 base for 40 epochs (these models converge faster) with batch size of 16 and gradient accumulation of steps. The models are evaluated on validation set every epoch. The best checkpoints are then tested on the test set. For experiments done on the con- catenated CMU-DoG dataset, we train all models for 30 epochs, because find the models start to overfit after 15-20 epochs. we use batch size of 8 and accumulate gradient for 2 steps. We use the same evaluation process as WikiText2.
Implementation Details T5 Small has 8 layers, 6 heads of dimension of 64 in each layer, an embed- ding size of 512, an FFN hidden layer of size 1024. TS Base has 12 layers, 12 heads of dimension of 64 in each layer, am embedding size of 768, an FFN hidden layer of size 2048. The two models use GEGLU activation (Shazeer, 2020) in FFN layer. The two models use sentence-piece (Kudo and Richardson, 2018) tokenizer with vocabulary size of 32K.
In preliminary experiments, we fine-tune the T5 Small with XL cache and T5 Base with XL cache on truncated PG-19 dataset using learning rate of {1- 11-10-3,1 1- 10-10-4,35 5 10-5,3 2 1,00-,,, as well as constant learning rate and a linearly decaying learn- ing rate. We find that constant learning rate of 1 10 4 works best for both models, when trained for 100K steps. Therefore, we use for all the experiments presented in this paper. In the follow- ing part of this section, unless otherwise specified, the same setting applies to all models. We use A (Loshchilov and Hutter, 2018) optimizer, with weight decay of 0.01. We also use fp16 16-bit mixed precision training. We run all experiments Tesla T4 GPUs The models are implemented in Pytorch (Paszke et al., 2019) and based on Hug- gingface Transformer (Wolf et al., 2020).
For compressive cache, we implement the best configuration reported in (Rae et al., 2019), namely, ID convolution as the compression function trained with attention-recun reconstruction loss. We follow the compression rate reported in their experiments on PG-19 and use the compression rate of 2. The other experimental settings are the same as other models in this paper, c.g., fine-tuning instcad of training from scratch. Recall that we keep track of docu- ment IDs to use document masks, the document ID of the first hidden state in the sliding window of convolution operation is assigned as the document ID of the compressed hidden state.
For fixatio-bbsse selective cache, we transform the original fixation duration to values ranging from to 11, thus the prediction is also mostly in this range (See preprocessing of eye-tracking data in Appendix A). we then experiment with thresholds of 8, 9, 10 and find 9 works best, which means there are roughly 25% of key-value pairs are selected in each step. Given that Cimp 512, Csic 256, the selective cache only covers two previous steps. For selector-based selective cache, when train- ing the selector, the size of the context snippet
For experiments done on the truncated PG-19 dataset, we train the models for 100K steps (59.28 epochs). After 50K steps, the models are evaluated on the validation set every 10K steps, and the best checkpoints then tested the test set when the training is finished. We use batch size of 32 for models using T5 Small, and use batch size of steps for mod-
16 and gradient accumulation cls using T5 Base. Note that we train the models on % of training data from PG-19 9 for about 60

---

Csmp 128 throughout all experiments, which is removed during evaluation. we haven't tried other values for Ssmp because we assume does not af- fect the performance as long as it is not too small. Besides the cache size, another important hyperpa- rameter is the length of the list that stores previous snippets 1. Recall that in Section D, the probability of selecting a snippet from the kth previous step is geometric distribution with parameter 1. In gen- eral, It is better set larger for modeling longer dependency. We search the best hyperparameter from choices of {3, 5, 7 for PG-19 and Wiki- Text2 datasets, and from {6, 9, 12} for CMU-DoG dataset, using& model fine-tuned from T5 small. For PG-19, the best = 5; For WikiText2, the best / 3; For CMU-DoG, the best 1 9, and in this dataset the performance is almost the same with different choices. Another hyperparamter is X in Equation 4. In preliminary experiments, we find X' = 0.01 works the best in general among {0.03, 0.01, 0.003) and use it for all experiments involving Lo norm. As for the 1 that is used to control the threshold for RSI (See Section 3.4), we simply set 0.5 in all experiments.
methods are comparable to the compressive cache, and higher than vanilla and XL cache. While these numbers will change on different hardware, we believe the relative proportion will be largely con- sistent.
J Comparison to Simple Baselines
Other than fixation duration and neural networks, we also experiment with two less-sophisticated baselines. (1) The first one ("frreq") selects rare tokens. We calculate the token frequency on the training set, and select those tokens whose fre- quency is lower than the threshold into the selective cache. We determine the threshold so that, erage, around quarter of tokens in each step are selected (same as when using fixation duration). (2) The second one ("entropyy") uses information entropy. When doing auto-regressive LM, the lan- guage model estimates the distribution of the next token P(X; the entropy of the distribution is H(X,)) = - Ex,,s P(z.(=<>) log P(r, (=<s), where V denotes the vocabulary. If the entropy of X, is large enough, the observed next token will be selected. Intuitively, when the entropy is large, the model uncertain about the next token, it's hard to infer from the context, so seeing the next token provides new information. Similarly, we de- termine the threshold so that the same proportion of tokens are selected in each step.
As for replacement-baseel selective cache, adopt the same hyperparameters when possible. Concretely, I use Csmp 128, Cate 512, C.t 128, Cimp = 512. We also use = 5 for PG-19, = for WikiText2, I for CMU-DoG.
H Statistical Significance
We measure the statistical significance of the re- sults. We did 3 runs with different random seeds for 3 configurations, namely T5 Small + XL of size 512, T5 Small XL, CPR of size (256,256), T5 Small XI., SLC (sle, RST) of size (128, 384). We measure the standard deviation of the perplexity as shown in Table 2. Note that we train models on PG-19 dataset for only 10K steps, while we use the same experimental setting on other datasets. We can see the standard deviation is much smaller than the gap between different models.
The experimental results of the new baselines are shown in Table 4. From the table, we can see that the frequency a better selection criterion than the entropy in general. Compared to Table 1, the frequency and the fixation duration achieve sim- ilar performance. On PG-19, 19, the latter performs slightly better, while on CMU-DoG the former per- forms better. However, the frequency and fixation- based methods demonstrate different selection pat- terns. E.g.. when using frequency as selection crite- rion, the model selects almost all digits and selects more named entity-related tokens.
I Training and Inference Cost
We measure the number of parameters, speed and memory usage of different configurations, as shown in Table 3. For all these configurations, use the same hyperparameters (batch size, model size, etc.) and experiment on the same device (2 Tesla T4) to ensure fair comparisons. We can see the training and inference cost of the proposed
Note that the models with selector and replacer networks still show considerable improvements compared to these two baselines. Therefore, even though neural network-based selection tends se- lect rare tokens, aspects other than frequency also play an important role.

---

PG-19
WikiTex12
CMU-DoG
XL
34.49 + 0.059
19.95 + 0.055
20.13 : 0.058
XL,CPR
34.09 = 0.033
19.92 = 0.035
20.13 0.052
XL,SIC (sle,RSI)
33.14 + 0.051
19.05 + 0.010
18.94 0.037
Table 2: Mean and standard deviation of runs
Training
Inference
XL
2nd
#param
s/iter
Memory
s/iter
Memory
Vanilla Transformer
52.3 M
0.71
9.0 GB
0.12
2.9 GB
XL
512
52.3 M
0.84
10.5 GB
0.14
3.1 GB
XL, CPR
256
256
56.0 M
0.95
12.4 GB
0.16
3.4 GB
XL, SLC (fix)
256
256
52.3 M
0.93
GB
0.17
3.0 GB
XL, SLC (fix.RSI)
256
256
52.3 M
1.07
10.5
0.20
3.0 GB
XL, SLC (slc)
128
384
52.3 M
0.98
12.5
0.16
3.0 GB
XL, SLC (sIc,RSI)
128
384
52.3 M
1.05
12.5 GB
0.18
3.0 GB
XL, R-SLC
128
512
52.3 M
0.98
12.2 GB
0.15
3.1 GB
Table 3: Training and inference cost difference configurations. "#param" denotes the number of parameters, "sliter" stands for second per iteration, "Memory" stands for memory usage per GPU.
K Analysis of Selector Network
K.3 Difference across Layers
Figure shows examples of the selected tokens in other layers. We can see the selection ratio is different across layers, well as the "quality" of selection. Besides selecting named entities and rare tokens, no other meaningful patterns found. The pattern only appears in few layers. The pat- tern shown in the figure is consistent in the whole dataset.
K.1
Selected Tokens
Other than Figure 3, we show more examples in Figure 6. We can see the same pattern as described previously.
K.2 Effect on Token-Level Loss
We run two identical models which use selector and selective cache. One model runs normally, while the cached key-value pairs selective cache of the other model are randomly masked (50% probabil- ity). The model cannot attend to those key-value pairs, but those still occupy the room in selective cache. The differences in cross-entropy loss be- tween these two models are then calculated on token-level. In other words, measure whether the model makes better or worse predictions when predicting each token. Figure 7 shows the result- ing loss difference. Note that previous figures selected tokens, the color is associated with the input token, while in this figure the color is asso- ciated with the prediction target token. We pur- posely choose the input sequence that is from the same document the previous example, and few steps after that. Orange color means the loss increases after masking, green means the loss de- creases. Deeper color represents bigger values. The difference values that exceed 1.0 or -1.0 are repre- sented full color.
K.4 Performance CMU-DoG
interesting see how the selector performs CMU-DoG dataset as shown in Figure Recall that the document this dataset consists of one Wikipedia article and many conversations based on that article. Therefore, keeping the article in memory is considerable advantage when doing LM on this dataset.
It is obvious that the selector treats the Wikipedia article and the conversations very differently. The selection ratio drops drastically when coming to the conversation part (in the middle of the third block in Figure 9). This means that the selective cache stores the background article and receives minimal updates afterward. Therefore, the back- ground article Note that this beneficial pattern learned by the model itself, and there is no token type embedding or something similar explicitly distinguish two types of inputs. Furthermore, this example reveals a key difference between the selective cache and
be preserved for many steps.

---

PG-19
WikiTex12
CMU-DoG
XL
2nd
Small
Small
Base
Small
Base
XI. SLC (freq)
256
2566
29.53
19.86
16.25
19.85
16.66
XL, SLC (entropy)
256
256
29.91
19.88
16.24
20.24
17.02
Table 4: Test perplexity when using frequency ("freq") and information entropy ("entropy") as selection criterion.
Lagen the game and Sonic as masore for the company i Alternane verdons and ports = = S-@ hit version e = veriion of Soni- liedgehng was developed by Ancentt and releared in 1991 for seall 14888-30 38 bit consoles, the Master System and (ame Gear its plot and gamepley mechenics are siiler to the to 65-3 bitt version with different level themes and digttl assets The levd design is flater with no vertical looos and Sonic cannot 6 collect his rings after being ht. The gamehes a different soundtrack composed by crictune musican wal tohn and induding lis compostiion and adaptatione of music from the 16 wentol wes the thaa game released for Maste System iittn America. The Maste Banan version was (- rel releaced for will Wraaa Concal service in North America on August 2008 and Europe on Augusill The Game Scor vasion was 9 released for the Nacando 305 Virtua (oondi on June 2013 and included
uniokkable game in Sonik
Dreecer 5 Cut to
port, Sonic the rebassed for the Game babanna (GBa on Novenbur Mid 2006 in United Bulos to mark the game is ffteerth arriverary I undd sevard
and Windows and le Mesa
But for Pay
X0ox, and Windows
Sonciitee Hodothog Genadl-
faatue auch the abilit save
progres, level select option, and in Anrivenayy MMoil with the Spin Dadl move (not originaly implemented until Sonic
Hedgaton 2) I1s kowen slihtll zoomed and adepted the Gll siddesreen oxpect ruloo The
receved poor redeos with Metecriic score 33 peroent; the (realinng in slow form sate 1. nembed chunkyy muse, and promervation of the original gamaplay.
chel compleiit conpereed Compliation ielennes
conversion the Game
With its
Figure 6: The selected tokens in WikiText2 validation set. See the caption of Figure 3 for an explanation of colors. The model is T5 Small + XI., SLC (SIc.
sciencé and the now Itorature created in the last decades of the nineteenth century. And now once more return to reminiscent account of the Grand Canyon, for old mon love to talk of the past. The Flaveix Segion The Grand Canyon of Biittnn and the Marble Canyon constitte one great gorge carved by mighty river through high plateau On the northeent and north a line of cifes face the platwes by hold mecrpment of rock Climb thene clifs and you must aocend from 600 to 1,000 feet, but on their summit you will stand upon a platesu stretching away to the north Now turn to face the south and you will overlook the clif and what appeors to be a valley below from the foot of the caff the country rises to the south a great plateau through which the Marble and the Grand conyons are carved. This plateau terminates abruptly on the west by the Grand Wash Cimf which is high escarpment caused a "bult (as the geologist calls it. that is. the srata of sandstone and Emestone are broken off, and to the west of the fracture they drapped down several thouannd fewt, so that standing upon the edge of the platwau above the Grand Wash Cills you may look off the west into the valley of the Little Caleraa and where into
vast region of desert from which low volcanic
mountains rise that sown like purple mounds sand clad lands. On the wad the that run down into the Colorado, the raiload iscends the pletesu from the east
platwau breaks down in wey inegudar the south the platesu
pictureqque
mountains stand Southern Arizona, Where the platesy ends and the mountains begin not well- defined line. The platesu through which the Grand is region
interest
from
than thousand feet above
The Grand Platoau is composed of many subsidarr platoaus, each pine forest which intercapted by few meadows
having own with
feature.
the northeast Grand Canyork
lakellet is the
Figure 7: Difference in token-level loss when randomly masking 50% key-v pairs in selective cache in layer 4,5 and 6. Orange color means the loss increases after masking, green means the loss decreases. The deeper color represents bigger values. The model is T5 Small + XL, SLC (sle, RSI) trained on PG-19.
subsequent years land wonders] Major Powel's Several Trips same year lssaaed to explore Grand Canyorial the Colorado) together lower portionill Green Rivad this purposel employed four rowboats and made descent from what now Green River station through the whole coursedo canyons to the mouthial the Ro Virgin distanceal more than thousand mive| [llustration: From Kabab Planeed looking Southl the springläf 16701 again started with three bosts and descended the river to the Crossing of the fothers] where met pack trin and went out with partylal men to explore ways down imto the Grand Canyon from the north and devoted the summed fall winted and following spring to this undertating in the summerial 1871 returned to the rowboats and descended through Marble Canyon to the Grand Canyorial Arizonal and then through the greater par col the Grand Canyon issef) Subsequent years were than given to explorationial the country adjacent to the Grand Canyori On these trips Mi Gilwer the geologisd who had been with Lieutenant Whedle] and Capil ci El Duttoo] were my geologiaal companionss On the second boat tripl and during al the subsequent yeanial explor ation in this region] Prof Al H| Thampson was my geograchical companiort assisted by a numberiäf topographica (engineers in 1882 Mi ci DJ Walcott my assistant in the United States Geological Survey went with me into the deptheal the Grand Canyori We descended from the sumnitio the Kalbao Ratoau on the north by trai which we built down a side canyon in direction toward the mouthifl the Uitlee Colorado Rived The descent was made in the fail and a small party man was left with My Walcott in this regioniatt stupendous daptil to make studylaf the geology (il an important region labyrinthan gorged Hare with his party was shut up for the winted for it was known when
Layer previoulyy been seen by Doctor Newberry in 1858) Me Gilbert was the geologittaf expedition his studessil canyon region during have greatly to our inowledeeil
upper canyoncal that stream and the great canyonsaal
Liyer previussly been seen by Doctor Newbery in 1858, Mr. Gibert was the geologist of this expedition and his studies of the canyon region during this and subsequant years have added greatly to our knowledge of this land wonders Major Powel's Several Trips in this same your essayed explore the Grand Canyon of the Colorado, together with the canyors that stream and the great canyors of the lower portion of Green fiver. For this purpose employed four rowbosts and made the descent from what of
Green River station through the whole
the mouth of the Roo Virgin distance of more than thousand miles Illustation: from Kabbeb Platesu Looking fethers, where
South the of 1600 Grand started with three boats descended the river the Crossing fall, winter. and following spring this undertaking, the Grand ltselt Subsoquent
pack and went
party of
explore ways down rowboats descended through Marble Canyon the Grand
from the north and devoted the Artzona, then
1871 returned to the who had with Lieutenant Wheeler, and C. Dutton, Froo, C. Walcott, subsequent years exploration Geological Survey, with study the geology importent region of
the
then given geographical companion, assisted the Canyon, We Little Colorado
exploration
adjasent the
On these trips Mr. Gilbbrt the
geological companions. the
by number topogr engineers.
the
deecended Mbyrinnhiengggrre
by full small party men left Mr for was known
Kalbab
canyon this region stuperdous depthe
with his party, was shut up
Jayer prevouudy been seen by Doctor DoctorReesoo (eewbers in of Mr: Gilbert was the geologis of the Impostitti and his stud of thelanyon region during this and ou oubsequeet years have added greatly to our knowledge of thi
wendors) Major Powells Soveral Trips in this some yearl essayod to cxplore the Grand Canson oftine Calordda together with the upper canyoná of that stream and the great canyons of of portion of Green Rosal For this purpose employed four rowlocas and made the descent from what now
Green fivee station through the whole course A to the mouth ohe Bin Vinii distance more than thouaand miv ball suiorl From Kabab Planand Looking South inliie spring wfit 101
sharedd with these bosts and descended the river the Crosing of the fwthers where met
tain went out with party off8 71 returned tolfte and then through the greater of the Grand Canyon roolf Subsecuent been Licutenant
explore
the Grand Camjon from the north and devoted the
fall winter, and following spring to this undertaingll inllll
Moutennntiiiiaaee Mitiiii and Capt. C.E. Dutton then rowboets desconded through Marbie Cansana On tips Mr. Gilber the geologis regior Prul H. Thampaon my geographiue companion asitted loration country bry adjacant
the Grand
skgiial comperioon) the
ber hopegropphcall engineend 3 daring schaneuent yoars
Hatews north by aooidnnt
Welkett
Sumeei went
Grand We lloee
buit down canyon in direcionn toward descont was made in the fall and small party men with Wh-80 region stupendous deptio to make study of goology of on important rogion of lsbyriitt an gorgol Hore with his party was shut up for the wintef for it was known whon
descended from summitor
Figure& 8: The selected tokens in PG-19 validation set. See the caption of Figure 3 for explanation of colors.

---

layer document: cast Ray Scheder
Brooy
Brodd offiical response: a
sensationally effective action picture, scary
better because populated characters
daveloood
beings the
chear fully perverse scare movis over made... [with] more zust early picture, more electriiiy, [and] it's funny
sort way Spiglberg blessed
talent that is absurdly absent from most American fimmakers these days this actually knows how
story on screen. speaic
director's gifts some
the most lightening sequences inlland are those where we even see the
measure how the operates that not once do feel particular sympathy any
theltaan victime...
In moves Hoe land characters are simply functions the action... like stage hands who move too gruesome for children, and liell to furn the stomach of the impresiinable at any age I- It coerse grained exploitative work which depends excess impact. Ashore as bore. aeccowrdly staged and lumplyy written scare machine that works
fimms characters revealed terms the
props wound and deliver information when It's necessar
computer- like precision. the story.
You feel like rst, being given shock therapy directr
Spull berg genee thriller introdution
1975 American directed by and based on 's
layer die glant attades beachgoers on Amlly Haadd fetional s prometing the local police chief to hunt with the help offa and I professional Stwrkc humfer. The film stars
police cieff wremaad hunte Caint a andooniorrappee Mom Hoopes Dam loughtt the mayer or ariiiity Island and an willddd wife (iled movie name linst rating Romen
lomatoes 9/% with on average 92/10 Metaortics 6// IMBB B year 19/5 description A beach party takes place dusk on Ambt Hand where women named Ehrisss dosing
Wodlind goes doning in the ocead While she is violently pulled under The next day. her partal remains are found shore. The medical examiner ruling the death a ashari
attack leads to Malion Chier Maria brdd dosing thelbaches Mayor faam Vought overnuee him fearing it will ruin the town's summe Aconomy The now concurs with the
mayorss theory that Wordins was killdd a boating accidnt. Braad reluctanly accepts their conclusion until another fatal Shank attack occurs shorty after, A bounty is then placed
thelstark Irouuting
amateu Shark huninn
by. Local profesional shar mande Buiid offers his services for $10,000 Meanwhile, consulting locaa bgraphe Mall Hoopel
Miliiii remains and confimms her death from alihark attack. When local fishrmen catch large, the
prociains the beaches safe Hodper disputes being
the same predator confirming this after no human remains are found inside it H.Hoooel and Ball find hall sund vesual while searching the right wattee in
Underwater Hocbal retrivaeda stablel's tooth enbedded in the thiteeeee He drops other finding partil corple Narahi discuun Bll ody andittooe a clams that huge
great whiteshark resposible and reluues dose the beaches dlowing added whlty precautions. On the Fourth of Jiy weeland touritss fil the
Following
jraeii ank, the lhan actund lomeiny kiling bood and causing
Layer
go shock Brady finelly corvince
Whilellrdd lays down Cuan waits opportunity hookaiee Without wariing appears behind bost Quint enters The attaks
lanoontila barrel
drags berrel underwater and diappeers
slowly smking attempt approach Hoopel puts
Intending
lethally inject thelthar with using hypodiamic doee
thelaage Hooper Inject but manages toloscape tolthe
the boot direclly devours Quint la uood micniinng vosue stuffs a pressurize daoita int phark mouill dlimbing the mast shoots thetani with
Dunt's Idadtoyyng The nouttitiin
the shark Hoopai esuffaces, and and paddle touteity islandl and ng to meckkaa conversations user't
The movie
talk you about today Stevin Sprelber''s JAWS. has a review on Rodann omatoes
8/10 IMDS ratingt user2:
was worth
watch
usert: So do you agrae with the ratings
movi?? Many people find to be scary theller, others describe as being too graesomw What do you
gruesome and didntt agree with the rating But then. don't like violent movies user he But maybe you should give another try It's classic movie of action and suspense Strven Spad berg never desppoints Do you typiolly like Steven Spielber movies? user2 I'm not sure as whether
think? Do you like
of movel' user2 found it
layer ii like to give it another try or not, but then agsin liked ET usert: Well like one person put it. it's funny a Wood Alla sort of way. Since it's classic, it might not be exactly you remember Might be worth the watch even i just for the loughtl user2 Perhaps the humor is subtle I recall not really seeing to much humor it though. usert: Well suppose the film works for both types people. Some love it because of action and because it's scary thillle, while others find to be more humorous and funny, user2: That's trus, and it probably also has lot do with the types humor people lig don't caro for user't 1: Yash that' tru But course, most watch it the thril For example. there is scene in the movie where woman named Chiiite Walkil goss tipping in the While she's the water, shi's suddenly pullid undert It's unaspected and thriling one the reasons why
this movie, usert: honestly don't know what else say... This has been going way too long and honeetly sily.. I'm
return comversation usera how useri: refresh me?
the Hoopefull today" user2: h usent: Hello, user2: usent: was thinkin how remember what the done chatting with either somewhat jawe action movie?
abort the senversiion and still paid This temble user Ok
the original Jave user':
like action
definitely, dassic. vacation beach
packed today, 2:1 don't know much dragged under sounds sertlyingl usert: marine biologist police by hunt before
extrenely large white shark way
anyone
The first legendary
Figure 9:
The selected tokens in the concatenated CMU-DoG valid set. The input sequences shown are consecutive,
starting from the beginning of a document. See Figure 3 for the explanation of colors. The model is T5 Small + XL. SLC (SI.

---

compressive cache. The latter defines fixed com- pressive rate beforehand, while selective cache is adaptive to the varying density of information in the input and has varying compressive rates.
Meanwhile, another important advantage of using fixation duration is that it does not require training or fine-tuning together with the language model. So can be used to select key-value pairs for those very large pretrained language models that are too expensive to calculate gradients.
In addition, different from previous examples on PG-19 and WikiTex12, in this example the selector not only selects named entities or rare words, but also selects some common words such as "shark", "tank", even including verb "escape". This shows that the selection is necessarily restricted to named entities or rare words. The selector can further learn to select other kinds tokens such as keywords.
L Analysis of Replacer Network
L.1 Analysis of Replacement Process
Figure 10 shows how the replacement ratio changes over time. The peaks in the figure correspond the start of a new document. It seems that the selective cache can preserve items for infinite steps on CMU- DoG, since there are some nearly zero replacement ratios. We found this is the case for most layers. Figure I1 shows example of replacement. Both new key-value pairs and the cached ones are shown. They are aligned to show that "shops" and " are compared (more precisely, their correspond- ing inner representations), "districts" and "@" are compared, etc. Note that the "cache" rows contain non-continuous context, while "new" rows contain a continuous input sequence.
M Tokens Selected According to Fixation Duration
Figure 12 shows the selected token according to fixation duration. we can see the differences com- pared to those selected by the selector network. Humans are doing comprehension when reading, such as building the relationship map between char- acters and understanding the plot, and the long- fixated tokens are key information for that. As we can see, an important difference is that verbs compose a substantial proportion in those tokens selected by fixation duration, while they are ex- cluded by the selector network. We think those verbs are also helpful for predicting future tokens, but that requires higher-level abilities. It is possible that when combined with a more advanced or much larger language model which has the capacity to do complex reasoning and comprehension over pre- vious distant context, the fixation- based selection can bring about much larger performance gains.

---

1.0
0.8
0.0
50
100
150 step
20c
250
300
1.0
0.0
0.6
0.4
02
0.0
Figure 10: Replacement ratio over time. "Lx" denotes the layer number. Top: T5 Small + XL, R-SLC on PG-19 dataset. Bottom: T5 Base + XI., R-SLCC on the concatenated CMU-DoG dataset.
arche
people
saippl
Incoeparsred
Exurry
DOWTERAA Hookk Minnkrr city rtal Union Smition dele
slitions built Rog
date City
Cesser bilactic Clenrit Main Steee Sentin Hinxle Olemeer @
new
apanated by Une leoi Shows us bwail of
Us Mavivipul Shapa vrank of Contecdens is
l1 cliul of Us
Malihar Fire Peza StAnd
crche goventaan and
sound Ralhssd
2008 Gasy Iaaend
Arthary Eiyyni
Sim
1/200 cill. to 2004
-A
Edger
erung
en gency sariil
Wostoigii Inpartrent
Meniall operiees the Misiiss Hospiall Mesicdin
bouu/mom andl
Avcnus Secral
Loo.
lnd 11M6
lin rel
Jenno Cia
nov
lertt Aeaaae T0
new
Tates Rost
Service
operates the Mentciari
Nort Miet
and the Wens Mevitiiin Station pait. eetiinn
In
vade 3ollic
adeee
Ceoage
Word
Saxtion fow Kerem
The
arche
Dana FIr.
son
rsperxentaten from Werd
3 loder Kin touuton bS.. ropereetatte from Mard
1o
Hanty Hat
Servie
Carc fammon
city fines the on oftwart coreer 12 Morise and Pless
1937
and
rnprusarieton hom
Hen Eieel
cwand Desemoww Fur
Ne
Devel Goel teral
neW
the northeis soindr
compliion les part of the
Stare
Senor
(KGIk) and soors Sam Sonl Keissi
Block Cental
Figure 11: A example of replacement. Rows starting with "cache" depict the content of the selective cache, while rows starting with "new" show the new candidates from the input sequence. Tokens that are aligned to represent the one-to-one relationship. Highlighted tokens mean that their corresponding key-value pairs are kept after the replacement process. The configuration is T5 Small + XL, R- &LS on the WikiTeext2 dataset.
the greater part of the Grand Canyon Hoott Bulsonaan Jown were then glenn Inplooor on the country Indiacen to the Cran Lanyan Creesss
poolooa
who not been with Douitenont Whee and Cape C Dimon wer mh mooiociiall componionn ond Oh the second boot tip. and durtno the Buieeqque uen years onesploration In this
region Prot, A Thommcse was misasor
1880 Mr, C. Milsodi es sasizaall in the ted States Gullediall Survay, wonk with me into thediaat of the Gend Gaman We Beccendal from thisummt of Eiwiaaaa Kabbab Pataau on the north by tral which wa bullt down siile
companion bsssnad bya numoor otsooooroshicn onaineerd
amyoo an dresion lowand Hhe mouth of the Jille Rolanai Rive. The deaan was mede in the kill and small parly of man was lell with Mi. Vas tnil in regian
gerpes hew with lis party, was shut up for the wirtw, for was known when we hm that sn mnte on the summet of the platen works geeeen his return to the pper regin before the un should meil them the neet sprin Mr. We moll now the Director of the
iimee las deattl make a staly of thegeeing inn ol inpotant regian Bess
dimuiii the first trip in bosts many others have ansn aed to follow me and andlyerr by year ouc insociii have not with disaster some ha fadeom rals areiuueed on the banks of the Greon and the groves of others indiaaaton
Unidd States celioginn Survey. After thes year! made no an witn ale to my ligil and Gonnnt knoodedge of the Crand Canyen, thoughh
aidologg to the south and east throughout wide region of nuited puebi de and chighelling Since
intaaaale slong the cource of the
Co crado. In ael Phobiaad F M. Brown lost Hi. Budiiaal carty rallooad anolnwedd led by R a Btantoil started the haad of Marble Conyon and made their way down
the ther as they ccanded survey for alldinad along courss. Othe Iahonnaauus frovdiai have liated contini the Grand Canyon region and My. Mihuttll James has
duly ooe the region the intewnd of
Indluddd unlockable game in Sonk Adventure UX: Linector tot Windows...
Windows Sonic Moga Collection for Flaystation 2. Xbox, and
port. Sonic broteereeocaeheooo Resnehon
relessed the Gamee Boy Advance GEA) on Navember
gamaa Illwn th aniiessay lankaaed sunvel roo batle such es Uhe ebilly to sove game crcgriou
sulect option.
Aurivessayy Mode Suln
origialllinn inplemented Lit Sonic Hidiida 2). screen fillly soomeed in aaciinn
Waickonee asonct ruio. Thalgune
poor
Metamiii scoee perten tih sie complens seneed Mapoosamann comanica aeaun foe faemsa Sanii the linigenn has ported for
Game Advance leatling muskow frum rale Jrmmiee sturnyy
music and pon insoevation of giginallammelll
Camp leton weme
range
home and hendheedocneeee and bensona compunes hnsaaneared Irlonii Compl aboe (1955 for theeei Some Jam 1981 for the Segellamm and Game com.
Sonic Megelialemidd (2002), Sonic Mog Colemms (200)) Soric's 's Ultimate Gonnes Gal cetoo (209) for the Xbox 300 and Playstetion and Soni
octoo Sonkc the tndidddoonnd
2010) for the Nintondo DS Rediirral all the game an in eababaabl roward in the console vesions of sonk Senaroiofl
loacab rolaad
has been beslabll for all three maor sesanil 9 cmorrsion video gome tamnnt It was part of the Vil traaal Consill the service s 2209 mondua o and was relcosod for the Your liee kenil and Plagialion Nalmark shott Manee The wanw wednanand lor the Ped maa .Poi vieeo, and video @ sape Pa modeh in 2007 and
for Apple 100 unvic Compuilll (Ner and Pos wach moskes) in Aceil 2000. Seat
The selected tokens on PG-19. Bottom: The selected tokens on WikiText2. Those long-fixade tokens are selected, based on the fixation duration predicted by separate model. The input tokens of the selected key-value pairs are marked with blue. The RSI is not used here.
Figure 12: Top:


Reviews:
[{"review_id": "KeKA6IkDIx", "reviewer": "Reviewer_voB6", "summary": "", "questions": "A. Why choose the T5 decoder, which removes encoder attention instead of a natural decoder-only pre-trained model?\nB. Can you give more analysis or demonstration about the training cost as well as the inference cost? As you mentioned, the limitation of computational resources, is this method training heavily?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review! Here is our reply to your questions:\n\n* Why use T5 decoder?\n\n    Because we need pretrained language models with relative position embeddings, we intended to use Pretrained Transformer-XL (Dai et al, 2019), but the available sizes of Transformer-XL are too large. On the other hand, T5 is available in a wide range of sizes, its Small and Base size suit our budget. Even though taking only a part of it seems crude, we find it works well on LM tasks in general, even before fine-tuning. We will add a short justification for the choice of the model in the next version of the paper.\n\n*   Analysis about the training and inference cost\n\n    |   configurations  | #param | s/iter (train) | Memory (train) | s/iter (infer) | Memory (infer) |\n    | ------ | ------ | -------------- | -------------- | -------------- | -------------- |\n    | Vanilla | 52.3 M| 0.71           | 9.0 GB         | 0.12           | 2.9 GB         |\n    |XL 512   | 52.3 M| 0.84           | 10.5 GB        | 0.14           | 3.1  GB        |\n    |XL,CPR 256 256 | 56.0 M | 0.95      | 12.4 GB        | 0.16           | 3.4 GB         |\n    |XL,SLC (fix) 256 256| 52.3 M | 0.93   | 10.5 GB        | 0.17           | 3.0 GB         |\n    |XL,SLC (fix,RSI) 256 256| 52.3 M | 1.07 | 10.5 GB      | 0.20           | 3.0 GB         |\n    |XL,SLC (slc) 128 384| 52.3 M | 0.98  | 12.5 GB        | 0.16           | 3.0  GB        |\n    |XL,SLC (slc,RSI) 128 384| 52.3 M | 1.05 | 12.5 GB      | 0.18           | 3.0 GB         |\n    |XL,R-SLC 128 512 | 52.3 M | 0.98    | 12.2 GB        | 0.15           | 3.1 GB         |      \n\n\n    We measure the number of parameters, speed and memory usage of different configurations, as shown in the table above. For all these configurations, we use the same hyperparameters (batch size, model size, etc.) and experiment on the same device (2 Tesla T4) to ensure fair comparisons. “s/iter” is the time for each iteration or step, “Memory” is the memory usage per GPU. We can see the training and inference cost of the proposed methods are comparable to the compressive cache, and higher than vanilla and XL cache. While these numbers will change on different hardware, we believe the relative proportion will be largely consistent. We will add this information in the appendix of the paper.\n\nRegarding your concern about limited experiments, we can provide additional results. After submission of the paper we continue to run experiments using T5 Base on PG-19 dataset, here is the result. Note that the conclusions of the paper are unaffected. We will add this to Table 1 in the next version of the paper. \n\n|   configurations  | PG-19, Base |\n| ------ | ------ |\n| Vanilla | 25.24|\n|XL 512   | 24.38|\n|XL,CPR 256 256 | 24.18|\n|XL,SLC (fix) 256 256| 24.04 |\n|XL,SLC (fix,RSI) 256 256| 23.99 |\n|XL,SLC (slc) 128 384| 24.14 |\n|XL,SLC (slc,RSI) 128 384| 23.83 |\n|XL 640   | 24.43|\n|XL,CPR 384 256 | 24.21|\n|XL,R-SLC 128 512 | 23.63 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a sophisticated method for long-range language modeling. The improvement is obviously compared with Vanilla Transformer, Transformer-XL and CPR. The selective cache is well-designed with two insightful selection criterions. The case analysis is interesting.", "reasons_to_reject": "Experiments are limited. The paper didn't compare with other recently related work because of the different settings, as it only conducts experiments on a small dataset or subset.\nThe method is complex, and the effect of each design on performance gains is hard to attribute. The analysis experiments of the necessary design are needed.", "questions_to_authors": "A. Why choose the T5 decoder, which removes encoder attention instead of a natural decoder-only pre-trained model?\nB. Can you give more analysis or demonstration about the training cost as well as the inference cost? As you mentioned, the limitation of computational resources, is this method training heavily?", "paper_topic_and_main_contributions": "The topic of this paper is about long-range language modeling. This paper introduces the selective cache, which stores the selected key-value pairs from the previous context. It proposes two selection methods, the first one is based on eye-fixation duration, and the second one is based on training. And it proposes a method to remove similar instances for further improvement. Experiments are conducted on three datasets, including the subset of commonly used datasets for long-range language modeling.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Fine", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper needs more introduction about the compared baselines, and the contributions should be clarified more obviously."}, "extra_text": {}}, {"review_id": "6aBgervgat", "reviewer": "Reviewer_ziVP", "summary": "", "questions": "There are two main questions from my end:\n\n1. Why don't you compare to some less sophisticated and easer-to-enderstand baselines? For example, simply using the information entropy as the selection criteron.\n\n2. You mentioned that a pre-trained T5 model is used and further adapted. My question is, how comparable is the data condition then for different rows in Table1? I feel adding a column for \"data condition\" etc. would help with the clarity.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review! Here is our reply to your questions:\n\n* Comparison to less-sophisticated baselines\n\n    As you suggested, we experiment with two simpler methods. \n    \n    (1) The first one (\"freq\") selects rare tokens. We calculate the token frequency on the training set, and select those tokens whose frequency is lower than the threshold into the selective cache. We determine the threshold so that, on average, around a quarter of tokens in each step are selected (same as when using fixation duration). \n    \n    (2) The second one (\"entropy\") uses information entropy. When doing auto-regressive LM, the language model estimates the distribution of the next token $P(X_i|x_{<i})$, the entropy of the distribution is \n    $ H(X_i)  = - \\sum_{x_i \\in V} P(x_i|x_{<i}) \\log P(x_i|x_{<i}) $\n    \n    where $V$ denotes the vocabulary. If the entropy of $X_i$ is large enough, the observed next token will be selected. Intuitively, when the entropy is large, the model is uncertain about the next token, or it's hard to infer from the context, so seeing the next token provides new information. Similarly, we determine the threshold so that the same proportion of tokens are selected in each step.\n\n    A random sample of the selected token based on entropy (parentheses indicate selection): Kyra was (created) as a (separate) character from (the) (Se)er , who was (featured) as (the) (primary) antagonist in the (second) half of season four . (Carpenter) joked that she was surprised to be (considered) for such a minor part on the show as a (guest) star .\n\n   The experimental results of the new baselines are as follows:\n\n    | configurations | PG-19, Small | WikiText2, Small | WikiText2, Base | CMU-DoG, Small | CMU-DoG, Base | \n    | ------------- | ------------ | ---------------- | --------------- | -------------- | ------------- |\n    | XL,SLC (freq) 256 256 |29.53 |    19.86         |      16.25      |    19.85      |     16.66     |\n    | XL,SLC (entropy) 256 256 |29.91 |   19.88       |     16.24     |    20.24      |     17.02     |\n\n    From the table, we can see that the frequency is a better selection criterion than the entropy in general. Compared to Table 1 in the paper, the frequency and the fixation duration achieve similar performance. On PG-19, the latter performs slightly better, while on CMU-DoG the former performs better. However, the frequency and fixation-based methods demonstrate different selection patterns. E.g., when using frequency as selection criterion, the model selects almost all digits and selects more named entity-related tokens. \n\n    Note that the models with selector and replacer networks still show considerable improvements compared to the new baselines. Therefore, even though neural network-based selection tends to select rare tokens, aspects other than frequency also play an important role.\n\n* Data condition for different rows in Table 1\n\n    The data condition is the same for all rows in Table 1. The original T5v1.1 model is further adapted to LM objective, producing a new version T5v1.1-LM-adapted, released by google research. All our experiments are fine-tuning from this version, so the data condition is the same.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is of sufficient interest. Such a cache which essentially prunes irrelevant information while keeping important information from the history is an interesting direction to enable long-range modeling capabilities of the LMs.\n\nDecent PPL improvements are obtained with the proposed method.", "reasons_to_reject": "The proposed method is rather complicated, which requires pre-training the LSTM eye-fixation duration prediction network, the selector or the replacer network.\n\nAlthough qualitatively explained, the quatitative improvements from the proposed method remains somewhat unclear to me - yes we saw PPL improvemtns, but where do they really come from? Is it really like expected, that the improved PPL can be explained,", "questions_to_authors": "There are two main questions from my end:\n\n1. Why don't you compare to some less sophisticated and easer-to-enderstand baselines? For example, simply using the information entropy as the selection criteron.\n\n2. You mentioned that a pre-trained T5 model is used and further adapted. My question is, how comparable is the data condition then for different rows in Table1? I feel adding a column for \"data condition\" etc. would help with the clarity.", "paper_topic_and_main_contributions": "This paper extends upon the idea to cache relevant information from previous context to enable the learning long-range dependencies in language models.\n\nSpecifically, on top of the usual XL cache, the authors propose to include an additional secondary selective cache, which by some selection criterion, is populated by long-range relevant key-value pairs. They also proposed a differentiable proxy to overcome the non-differentiable nature of the selective cache. Three selection criteria are considered in this paper, namely, human eye-fixation duration on different words as predicted by a pre-trained LSTM network, a dedicated selector network, and a replacer netowrk.\n\nThe topic is of interest for EMNLP, because modeling long-range depencies remain a challenging direction to further improve the LMs.\nThe authors showed experimental results, that with their method, PPL can be improved on top of the vanilla transformer, as well as the XL model. Additionally, the authors provided detailed analyses to support that the selection process actually kept important words into the cache.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Eq. 2 is kind of \"floating\" on its own, I felt the smoothness of the reading experience was broken abruptly when I first reached L321. Maybe some connection sentences to explain what it is could help with the problem."}}, {"review_id": "E7ZqNujQJ1", "reviewer": "Reviewer_WTsa", "summary": "", "questions": "* In Section 3.2, total reading time is at the word level but the model operates at a subword (wordpiece/sentencepiece) level - how do you deal with this aspect?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review! \n\n* About your question\n\n    We convert the word-level TRT to subword level in a slightly complex way: For example, \"biggest, (6)\" represents a word \"biggest,\" and its Total Reading Time 6 (after converting to integers). \n\n    \"biggest, (6)\"\n\n    TRT is first assigned to each character, resulting \n\n    \"b (6)\", \"i (6)\", ... \", (6)\". \n\n    Then assign a small TRT to the last character,\n\n    \"b (6)\", \"i (6)\", ... \", (1)\". \n\n    After tokenizing the word we obtain the span of each subword,\n\n    \"big\", \"g\", \"est\", \",\" \n\n    And take the maximum value in the span.\n\n    \"big max(6,6,6)\", \"g max(6)\", \"est max(6,6,6)\", \", max(1)\"\n\n    The final result is\n\n    \"big (6)\", \"g (6)\", \"est (6)\", \", (1)\"\n\n    We do it in this way because the tokenizer we are using only provides character-level span information. This is not a perfect solution but it works fine in most cases. \n\n* About your comments\n    * In Table 1, we use different XL and secondary cache sizes for different configurations. As mentioned in the paper, we experiment with (256, 256) and (128, 384) for each configuration and choose the best one to report. In general, we find the perplexity of SLC (fix) 128 384 > SLC (fix) 256 256 > SLC (slc) 256 256 > SLC (slc) 128 384. If we choose the same cache size for all configurations, then the difference between SLC (fix) and SLC (slc) will depend on our choice. (256, 256) produces a smaller difference while (128, 384) produces a larger difference. So we treat the cache sizes as hyperparameters and search for the best one for each configuration.\n    We do the same when comparing R-SLC to CPR, we search the best hyperparameter from (128, 512), (256, 384), (384, 256) for CPR, while using (128, 512) for R-SLC.\n\n    * When experimenting with the replacer network, as mentioned in the paper, the R-SLC cache size needs to be greater or equal to the input size, i.e., >= 512. Meanwhile, XL cache plays an important role so it is necessary to have a minimum size of it. Thus we use (128, 512) for R-SLC and we have to run the baseline models with the total cache size of 640. Using (0, 512) for R-SLC would result in bad performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Given the immense popularity of large language models in the NLP community, the techniques proposed in this work can help improve performance of language models while lowering computation, which is meaningful and will likely benefit several researchers without access to compute resources.\n", "reasons_to_reject": "* Some of the experimental comparisons in the paper are not fully convincing. See comments above.", "questions_to_authors": "* In Section 3.2, total reading time is at the word level but the model operates at a subword (wordpiece/sentencepiece) level - how do you deal with this aspect?\n", "paper_topic_and_main_contributions": "This paper proposes three strategies to build a selective cache of key-value pairs (or \"states\") in a transformer XL style language model which enables the model to  make more effective use of longer histories using a fixed size cache. The strategies include 1) selecting words based on eye-fixation duration, 2) using a selector network which learns to prune key-value pairs and 3) a refinement of the selector network called replacement network which learns to remove key-value pairs from the cache. Results on PG-19, Wikitext and CMU-DoG datasets show that all 3 strategies yield improvements over Transformer XL and Compressive Cache (Rae et al. 2019). The replacement network strategy performs the best. Additionally the paper reports small improvements by clustering similar tokens and selecting only one key-value pair from each cluster. \n\nStrengths:\n* Proposes 3 novel approaches to improve the effectiveness of a transformer LM by caching important key-value pairs without increasing the context over which attention is computed.\n* Results on three tasks show that the proposed approach yield perplexity improvements over transformer XL  (Dai et al. 2019) and compressive cache (Rae et al. 2019)\n* Demonstrates that the approaches tend to store named entities and rare tokens in the cache, an intuitive result.\n\nWeaknesses:\n* Some experimental comparisons are weak/incomplete. See comments below\n\nComments:\n* In Table 1, the results from selector network (slc) do not use the same XL and secondary cache size as the SLC (fix). While the total cache size (XL+2nd) for slc is the same as the baseline (512), it would have been preferable if the slc used the exact same cache sizes for XL/2nd. This is also the case for the replacement network (R-SLC).\n* In Table 1, the result from the replacement network is not reported for the cache size of 512. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper proposes an efficiency improvement method for long-range transformers. While the reviewers found the proposed method interesting, they also highlighted the complexity of it and questioned whether the conducted experiments were enough to explain the complexity of the proposal.
Decision: Accept-Findings