DODO : Dynamic Contextual Compression for Decoder-only LMs (2024)

Abstract: Transformer-based language models (LMs) are inefficient in long contexts. We propose DODO , a solution for context compression. Instead of one vector per token in a standard transformer model, DODO represents text with a dynamic number of hidden states at each layer, reducing the cost of self-attention to a fraction of typical time and space. Moreover, off-the-shelf models such as LLAMA can be adapted to DODO by efficient parameter tuning methods such as LoRA. In use, DODO can act as either an autoregressive LM or a context compressor for downstream tasks. We demonstrate through experiments in language modeling, question answering, and summarization that DODO retains capabilities in these tasks, while drastically reducing the overhead during decoding. For example, in the autoencoding task, DODO shrinks context at a 20x compression ratio with a BLEU score of 98% for reconstruction, achieving nearly lossless encoding.

heading: 1 Introduction
text: Transformer-based LMs (Vaswani et al., 2017) suffer from quadratic computational complexity w.r.t. sequence length, making it challenging to scale to long sequences. Proposed solutions (Tay et al., 2022) include sparsifying attention patterns (Beltagy et al., 2020; Ding et al., 2023) or approximating the attention computation with kernel methods (Choromanski et al., 2021). However, not all these approaches are proven effective for NLP tasks (Qin et al., 2023), and very few of them are applied to large language models (LLMs), such as LLaMA (Touvron et al., 2023a).
We propose DODO , a solution for dynamic contextual compression for decoder-only LMs. While a standard transformer represents a text with vector sequences of the same length as tokens,
∗Work done in part during Guanghui Qin’s internship at Microsoft Research.
NLP is concerned with human language, …
the intuition of DODO is to use a smaller, variable number of vectors as a contextual representation. Past research indicates that a subset of token embeddings, named nuggets , in an encoder with global attention may carry enough information to reconstruct surrounding context (Qin and Van Durme, 2023), and upon inspection those authors observed these nuggets tended to account for preceding text. This suggests a decoder-only model might be dynamically capable of deriving such a representation online (Fig. 1). To enable DODO requires addressing a selection process that is not differentiable: we adopt the straight-through estimator (Bengio et al., 2013) to make the model end-to-end trainable.
Past work on context compression, such as Ge et al. (2024) and Mu et al. (2023), appends fixed additional tokens. DODO grows the representation with sequence length and re-uses existing token embeddings. Moreover, unlike pattern-based methods that evenly chunk the text (Rae et al., 2020), experiments show that DODO spontaneously learns to use textual delimiters as nuggets , naturally splitting the text into subsentential units (Section 4.3).
DODO supports causal masking and can be naturally used as an autoregressive LM. We experimentally demonstrate that DODO can achieve a perplexity score lower than the original LM with restricted memory, outperforming the baseline model of Rae
9961
et al. (2020). For tasks with a fixed context, e.g. long-form QA, DODO works as a context compressor: It encodes a token sequence into a shorter vector sequence, achieving a configurable compression ratio. In experiments on autoencoding, we demonstrate that DODO can achieve near lossless encoding with a compression ratio as high as 20x, a marked improvement over ICAE (Ge et al., 2024). After fine-tuning, DODO is effective in downstream NLP tasks such as question answering (QA) and summarization, where it performs on par with or even better than the original LMs while achieving a compression ratio as high as 10x.
In summary, we propose DODO for contextual compression for decoder-only transformers. It learns to subselect a fractional number of tokens as context representation. A straight-through estimator ensures that DODO is differentiable and can be trained with the next-token prediction objective. DODO achieves a remarkable compression ratio of up to 20x and is shown to be effective in tasks such as autoencoding, language modeling, and applications including QA and summarization.

heading: 2 Approach
text: In this paper, we study the language modeling problem p(wt | w<t), where wi ∈ V is a sequence of tokens and V is the vocabulary. The common Transformer (Vaswani et al., 2017) approach encodes a token sequence w1:n into a sequence of vectors and then predicts the next token: ( xL1 ,x L 2 . . . ,x L n ) = Transformerθ(w1:n), (1)
p(wn+1 | w1:n) ∼ LMHeadθ(xLn), (2)
where θ is the parameter, L is the number of transformer layers, xLt ∈ Rd is the hidden state of the t-th token in the L-th layer, d is the hidden state dimension, and LMHead is a feedforward neural network that defines a categorical distribution over the vocabulary. In the decoder-only transformers, xl+1t is encoded by attending to past token representation in the l-th layer:
xl+1t = Attnθ(x l t,x l 1:t), l = 1, 2, . . . , L−1 (3)
where the Attn function takes query and key (value) vectors as arguments. Eq. (3) can be inefficient with long sequences as its computation grows quadratically with the sequence length. In this paper, we aim to answer: Can we find an alternative method to efficiently approximate xlt ?

heading: 2.1 Representing texts with DODO
text: In Eq. (3), context information up to the t-th token is encoded into t vectors as hidden states. Intuitively, we can reduce the computational overhead by controlling the size of hidden states. Formally, we want to encode t tokens w1:t into k vectors: (zl1, . . . , z l k), where k ≤ t. Following prior work (Qin and Van Durme, 2023) we refer to these vectors as nuggets . Then xl+1t is derived by
xl+1t = Attnθ(x l t, z l 1:k), l = 1, 2, . . . , L−1. (4)
Please note that k is not a fixed number (Zhang et al., 2022; Ge et al., 2024) but a dynamic number that depends on the input sequence w1:t. We will discuss the choice of k later.
We observe that xl1:t encodes the information of tokens w1:t, thus one may derive zl1:k from x l 1:t. We therefore select zl1:k by subselecting vectors from xl1:t. Formally, we have (c.f. §3.3 in Zeng et al., 2023b and §3.1 in Qin and Van Durme, 2023):
{zl1, . . . , zlk} = {xli | αi = 1, 1 ≤ i ≤ t}, (5) p(αi = 1) = σ(Scorerφ(x ι i)), (6)
where αi is a binary variable indicating if xli is selected, p(αi = 1) refers to a Bernoulli distribution, Scorerφ is a feedforward neural network parameterized by φ, and σ is the sigmoid function. Scorerφ takes as input xιi, the hidden state of wi in the ι-th layer, where ι is a hyperparameter. 1 That is, tokens that were assigned with higher scores by Scorer is more likely be selected as nuggets .
Note that ι in Eq. (6) does not depend on l, thus it selects the same set of indices for all the layers. In the remainder of this paper, we abstract the process of Eqs. (1) and (4) to (6) into a Dodo operator:
z1:L1:k = Dodoθ,φ(w1:t), 1 ≤ k ≤ t. (7)
We may omit the superscript and use zi (xi) to indicate z1:Li (x 1:L i ), the i-th nuggets in all layers.
So far, we only assume that k is a dynamic number depending on w1:t. In general, we set k to be roughly proportional to t, controlled by a compression ratio r ≈ t/k. Depending on the task, k can either grow with t when w1:t is incrementally observed (Section 2.2), or be strictly proportional to t when w1:t is fully observed (Section 2.3).
1We empirically set ι = 3 in all experiments.

heading: 2.2 DODO as an autoregressive LM
text: Not all efficient LMs support causal masking (Peng et al., 2022). Many context compression methods (Mu et al., 2023; Ge et al., 2024) only apply to fixed-sized texts. However, each hidden state zi in nuggets only conditions on its past tokens. Thus DODO can be naturally integrated into an autoregressive LM, where tokens w1:t are sequentially fed into an LM. Instead of saving all past hidden states x1:t, DODO only retains a subset of tokens as nuggets , which are selected by Scorer. The stochastic selection process in Eq. (5) is made deterministic by settings a threshold Λ in Eq. (6):
αi = 1 {Scorerφ(xιi) > Λ} , (8)
where 1 is the indicator function. That is, token wi is retained as nuggets zj if its score is above the threshold Λ. Because Eq. (8) does not depend on future tokens, z1:k can be autoregressively encoded with causal masking.
To set a proper threshold Λ, we define a compression ratio r ≥ 1 and let r ≈ t/k. That is, Λ should be set such that after t tokens are fed into DODO , roughly k ≈ t/r hidden states xi’s should be selected as zj’s. In practice, we estimate the threshold Λ by running a trained Scorerφ on sampled tokens. 2
Parameter configuration Intuitively, as a compressed representation, zj should encode a broader range of tokens than xi does. We therefore separate their attention parameters: Once a token wt is selected by Eq. (8), it uses Attnϕ to attend past tokens. Otherwise, it uses Attnθ.
A mixed resolution Though z1:k is more efficient than x1:t, information loss is inevitable during the subselection process. Intuitively, the tokens closer to the target token wt+1 contain more relevant information. We propose to revise Eq. (4) with a mixed resolution, where xt attends to recent τ tokens without compression. Suppose we split the sequence w1:t at index (t− τ), we have
xl+1t = Attnθ ( xlt, [ zl1:k;x l t−τ :t ]) , (9)
z1:k = Dodoϕ,φ(w1:t−τ ) (10)
where z1:k are the compressed representation of w1:t−τ , [ ; ] indicates the concatenation of vector
2Training Scorerφ requires a determined Λ, but setting Λ needs a trained Scorerφ. To prevent the chicken-and-egg problem, we initialize the Scorerφ here from Section 2.3.
sequences, and τ is a hyperparameter. An illustration of our method can be seen in Fig. 2.
Learning To train DODO as an autoregressive LM, we estimate the parameters (θ, ϕ, φ) to maximize the log likelihood of p(w1:n):
max θ,ϕ,φ
∑
w1:n∈D
n−1∑
i=1
log p(wi+1 | w1:i), (11)
whereD is the corpus and p(wi+1 | w1:i) is defined by Eqs. (2), (9) and (10).
Learning with Eq. (11) can be inefficient: The computation cannot be parallelized on the sequence dimension because they have different splitting index (i − τ). As an efficiency optimization, we chunk the texts into segments, and tokens in a segment share the same splitting index.

heading: 2.3 DODO as a contextual compressor
text: In some tasks, such as long-form question answering, a fixed segment text, say w1:n, is treated as the context and is fully observed before the text generation. In this case, one can use DODO as an encoder 3 to encode the input text into hidden states z1:k where k ≤ n.
Formally, suppose w1:n and y1:m are the input and output sequences separately, the probability distribution of y1:m is defined as
p(yi | y<i, w1:n) ∼ LMHeadθ ( yLi ) , (12)
yl+1i = Attnθ ( yli, [ zl1:k;y l 1:i ]) , (13)
where we slightly abuse the notation to use yi as the hidden states of token yi. Refer to Fig. 3 for an illustration of Eq. (13).
3We use the term “encoder” because it encodes an input sequence. It is technically a decoder-only transformer model.
Decoder Generation Text
……
Encoder Input Text top-k
…… … …
Encoder Side Decoder Side ……
Nonparametric
Figure 3: DODO as context compressor. From left to right, Encoder side: Dodoϕ encodes texts into vectors representations; Scorer: Scorerφ computes a score for eaceh encoder token and then select the top-k tokens as nuggets ; Decoder side: Language model LMθ autoretressively decodes text conditioned on nuggets .
Because n, the number of input tokens, is known, we could maintain a fixed compression r = n/k by setting k = ⌈n/r⌉. We therefore make the stochastic selection in Eq. (6) deterministic by:
{z1, . . . , zk} = TopK(x1:n, s1:n, k), (14) si = Scorerφ(x ι i), (15)
where TopK selects k vectors from x1:n with the highest si, the score of token wi. 4
Parameter configuration We assign separate parameters to the attention modules in the encoder and decoder: The parameters of the encoder (decoder) are indicated by ϕ (θ).
Learning To train DODO as an encoder, we learn it through maximum likelihood estimation:
max θ,ϕ,φ
∑
w,y∈D
m∑
i=1
log p (yi | y<i, w1:n) ,
where input and output sequence pairs (w1:n, y1:m) are sampled from a corpus D, and the next-token probability is defined by Eqs. (12) to (15).

heading: 2.4 Learning with straight-through estimator
text: The selection of z is discrete: the selection process, Eqs. (8) and (14), is not differentiable. Here we show how to back-propagate the gradients so the parameter φ in Scorerφ can be learned.
Previous work proposed approaches to make TopK differentiable (e.g., Xie et al., 2020 and Sander et al., 2023). To avoid unnecessary complexity, we adopt the biased but simpler straightthrough estimator of Bengio et al. (2013). Suppose
4Because xi only encodes texts before wi, the last token wn is always selected to the information in w1:n is completely encoded in z1:k.
the token xj attends to the compressed representation zi, and let ξi,j denote the logit of the attention token xi to the compressed hidden state zj . Then we have (c.f. §3.2 in Qin and Van Durme, 2023 and §2.2 in Jang et al., 2017):
ξli,j = ( WQx l j )⊤ ( WKz l i ) , (16)
∂ℓ ∂si ←
∑
j
L∑
l=1
∂ℓ
∂ξli,j , (17)
where WQ and WK are parameters of the selfattention, and ∂ℓ/∂si is set to be the aggregation of the gradients of ξli,j from future tokens in all layers. Intuitively, Scorerφ learns to select tokens that are more attended by future tokens. To implement Eq. (17), we replace ξli,j in Eq. (16) with:
ξ l i,j = ξ l i,j + si − StopGrad(si), (18)
where the StopGrad(si) detaches si from backward pass and ensures that the addition of si to ξLi,j does not affect the forward pass.

heading: 3 Overall experiment setup
text: We adopt the decoder-only transformer architecture of LLAMA (Touvron et al., 2023a,b) as our base model. For the autoencoding experiment, we use the checkpoint of LLaMA-7B following the baseline model ICAE (Ge et al., 2024). We use the checkpoint of LLaMA-2-7B for the autoregressive language modeling experiments (Section 5) and LLaMA-2-7B-chat (Section 6) for the downstream NLP tasks.
We adopt LORA (Hu et al., 2022) with a rank of 32 to fine-tune the parameters of the LM, namely
θ and ϕ. We adopt the implementation of huggingface/PEFT packakge (Sourab Mangrulkar et al., 2022). More specifically, we fix the original parameters of LLAMAand add two LORA adapters for θ and ϕ respectively. Different adapters are activated for the computation of compressing and decoding of DODO . We disable the adapters to produce the features to Scorer.
We employ mixed precision to save GPU memory. The training is scaled up to 16 NVIDIA V100 cards with DeepSpeed (Rasley et al., 2020). See Appendix B for further training details, including hyperparameters, and parameter counts.

heading: 4 Autoencoding experiment

heading: 4.1 Task, dataset, and experiment setups
text: In this section, we use DODO as a context compressor (Section 2.3) and apply it to the autoencoding task. As a comparison, we use In-Context AutoEncoder (Ge et al., 2024, ICAE) as a baseline model. In this task, a model is asked to reconstruct the input text from a compressed representation. Following ICAE, we fine-tune the LLaMA-7B model on the Pile (Gao et al., 2020) dataset. We manually split the corpus into train, dev, and test splits, and train the model until convergence.
As stated in Section 2.3, we use DODO to compress the input text into fewer hidden states z, and then use the LM to decode the input sequence. The size of hidden states z, i.e. k, is set to be proportional to the length of the input sequence: k = n/r, and we set r = 20 and 10. We prepend a trainable soft token to the decoding sequence to signal the model to reconstruct inputs (Ge et al., 2024).
The key idea of ICAE is to append 128 tokens to the input sequence as “memory slots,” and train the decoder to reconstruct the input from the memories:
(m̃1, m̃2, . . . , m̃128) = LM ([w1:n;m1:128])
p(wi+1 | w1:i) = LM ([w1:i; m̃1:128]) .
We measure using BLEU (Papineni et al., 2002) score on pairs of input and decoded texts. 5

heading: 4.2 Experiment results
text: In Fig. 4 we see DODO has comparable performance with the ICAE baseline for short sequences and better performance for long sequences. Moreover, DODO successfully handles longer inputs:
5We report ICAE results per the §3.3.1 in Ge et al. (2024).
performance improves on longer sequences because the number of nuggets is proportional to the sequence length, unlike ICAE’s constantsized memory. Despite its variable memory, DODO maintains an advantage over ICAE in computational time and space. First, DODO encodes sequences more efficiently: while ICAE always appends 128 tokens, DODO reuses a fraction of the already-encoded tokens. Also, DODO uses fewer tokens than ICAE: even for the longest sequences, DODO only uses 25 or 50 tokens, while ICAE uses 128 for all sequences. 6 Lastly, DODO is more efficient than ICAE during decoding because it uses fewer tokens and does not need to re-encode them. In short, compared to the baseline, DODO demonstrates comparable or better performance, successful handling of long sequences, and much more efficient encoding and decoding.
We also conducted experiments on languages other than English. For more details, readers may refer to Appendix F.

heading: 4.3 DODO selects clausal text delimiters
text: In Section 2.1, we employ Scorer to pick out nuggets , but what are the actual tokens selected? We empirically sampled 128 documents with 50k tokens and run the Scorer from the checkpoint in Section 4 with a compression ratio of 10, and the results are shown in Fig. 5. Readers may refer to Appendix C for case studies on sampled texts. From Fig. 5, we observe similar phenomena as Qin and Van Durme (2023), where the tokens preferred by DODO are mostly clausal text delimiters, such as punctuation marks and conjunction words. This
6DODO uses all layers while ICAE only uses the last layer. However, ICAE needs to encode their memory tokens into hidden states during decoding, while DODO can save this step.
phenonenon is further discussed in Section 7.2.

heading: 5 Autoregressive LM experiment

heading: 5.1 Experiment setup
text: In this task, the model is asked to autoregressively decode a sequence of texts. We therefore use DODO as an autoregressive LM (Section 2.2). We introduce a baseline method Compressive Transformers (Rae et al., 2020) (denoted by COMPRESSIVE ), which evenly chunks the text into segments and uses a pooling algorithm 7 to compress the hidden states of each segment into a single vector. We also conduct experiments with the original LLAMA, denoted by FULL . In experiments, COMPRESSIVE has the save compression ratio as DODO does. FULL does not support compression, so we limit its context length to make sure all models use the same number of hidden states.
We use the Pile (Gao et al., 2020) and WikiText103 (Merity et al., 2017) as the corpus. We randomly split the Pile into train, dev, and test sets, where the test set contains 100k tokens. All models are initialized from the checkpoint Llama-2-7b, and trained on the training set of the Pile until convergence. The compression ratio for DODO and COMPRESSIVE is 10x. The evaluation is conducted on the test set of the Pile and WikiText-103.
Perplexity (PPL) is used as the evaluation metric. Following previous work, we exclude the words that are defined as out-of-vocabulary by Merity et al. (2017) from the evaluation on WikiText-103. Because WikiText-103 is a tokenized corpus, we take production over the probabilities of subwords for each complete word to measure the word PPL. Note our algorithm underestimates the model performance for the complete word PPL.
We illustrate the intuition of DODO via an exam7In experiments, we adopt the mean pooling.
ple in Fig. 6. For such an example, DODO should retain both topical and explicit vocabulary information (e.g., the underlined text) in the compressed history, in order to be less surprised by subsequent text such as bolded there.

heading: 5.2 Experiment results
text: The experiment results are shown in Table 1. We conduct experiments with 3 context configurations, where an LM has access to up to 64, 128, or 256 past hidden states. For DODO and COMPRESSIVE , the first 32, 64, or 128 states are compressed representation of the past 320, 640, or 1280 tokens. DODO outperforms both COMPRESSIVE and FULL , showing that with a restricted size of hidden states, DODO is an effective method to encode history information.

heading: 6 Downstream task experiments
text: We pick downstream tasks where a document as context is followed by a query. The model is asked to encode the document and decode the answer conditioned on the document encoding and question. In these tasks, we use DODO as a context compressor (Section 2.3), and we set the compression r = 5 or 10. To train DODO to perform these tasks, we consider 2 scenarios. a) Fine-tuning: DODO is trained on the training set of the downstream tasks. b) Zero-shot: DODO is trained on normal texts randomly sampled from the Pile and directly tested on the downstream task. In this case, each text is split into 2 parts, containing up to 512 and 128 tokens, and the model is asked to decode the second part conditioned on the encoding of the first part.
We consider the tasks of question answering and summarization. Datasets used in this section are SQuAD (Rajpurkar et al., 2016) and CNN/DailyMail v3.0.0 (See et al., 2017) for summarization. Their statistics are listed in Table 2.
We use the following baseline methods: • FULL : Results of the original LM.
• NODOC : LM is used to do the task without any documents. Only the question is provided.
• LMSUMM : Use the LM to summarize the text into fewer tokens with prompts, which asks the LM to compress the texts into 10% of its length. LM uses the summary instead of documents to do the task. (Appendix D.1) 8
8In practice, LM uses 10.9% of its original length to summarize the text on average, counted by subwords.

heading: 6.1 Question answering
text: In SQuAD a model is asked to extract a phrase from the passage to answer the query. We reformulate this problem as a text-to-text task instead of annotation and prompt the model to answer the question (Appendix D.2). We use accuracy to evaluate the model performance. As the model tends to generate tokens more than the answer itself or using different forms (e.g. using “two” instead of “2”), we normalize the output to match the answer. Readers may refer to Appendix E for the algorithm used to calculate the accuracy.
We consider all models: FULL , LMSUMM , DODO , and NODOC (Table 3). All models are evaluated in a zero-shot manner without finetuning. FULL and DODO easily outperform the NODOC and LMSUMM , and we observe that LMSUMM often omits details that are needed by the question. The performance of DODO can be improved by lowering its compression ratio, and the performance of DODO (r = 5) is close to FULL , confirming a compressed representation can still support LLM reasoning.

heading: 6.2 Summarization
text: CNN/DailyMail contains news articles, where a model is required to generate a short summary. As no query is involved, we propose a prompt as a statement of the task requirement (Appendix D.3).
We consider FULL and DODO (r = 10). FULL is evaluated in both zero-shot and fine-tuning settings and DODO is fine-tuned. The results are shown in Table 4. We find that DODO can achieve similar or even better performance than FULL after compression. We speculate that as the context of CNN/DailyMail is long, this may lead the LM to be “lost in the middle” (Liu et al., 2024), whereas the nuggets generated by DODO is only 10% of the original length and perhaps less susceptible. This is an interesting avenue for future exploration.

heading: 7 Discussion
text: 7.1 The selection of nuggets
In DODO , Scorer selects k vectors out of n candidates at each layer of the transformers. We adopt a solution of hard selection because of its simplicity. Some alternatives, such as soft attention and soft
top-k operator, require either additional parameters or advanced machine learning techniques. Hard selection learns to naturally split the text, which contrasts some pooling strategies that evenly split the text (c.f. Section 5).
NUGGET selection is learned through the residual connection introduced in Section 2.4. With gradient signal from the self-attention, Scorer tends to select the tokens that are mostly attended by the decoder. Isolating the other parts of the model, how can we evaluate the performance of Scorer itself ?
To simplify the discussion, let I be the selection conducted by Scorer. We use I∗ to denote the theoretically optimal nuggets selection, which is defined as the selection that achieves the best performance in a task, e.g. the lowest perplexity in the LM task. To evaluate I, we ask: How similar are I and I∗ ? What is their performance gap?
Unfortunately, finding the optimal selection I∗ is a non-trivial combinatorial problem, so we propose a greedy algorithm to approximate I∗ . Due to the space limit, we leave the details of this algorithm and our experiment design to Appendix A. As the results, the overlapping between I and I∗ is roughly 75.3%, meaning the nuggets selected by Scorer are very close to the theoretical optimal se-
lection. Replacing I∗ with I will sacrifice 7.9% of the performance in terms of LM perplexity, so we conclude that Scorer, though not being optimal, can achieve a near-optimal performance through the straight-through estimator.

heading: 7.2 DODO favors clausal text delimiters
text: In Section 4.3, we observed that DODO favors clausal text delimiters as the nuggets tokens, similar to the findings of Qin and Van Durme (2023). We have the following assumptions: • Clausal text delimiters are used as “summariza-
tion tokens” during pretraining. The LM was pretrained to predict the next token, and predicting the text delimiters was equivalent to predicting the ending of a clause/sentence. Therefore, the LM learned to store contextual information in the delimiters, such as punctuation marks.
• Scorer was biased to frequent tokens. Except for the clausal text delimiters, DODO also prefers the token “the”, which hints that the straightthrough estimator in Section 2.4 might bias Scorer to select frequently appeared tokens.

heading: 8 Related work

heading: 8.1 NUGGET text representation
text: DODO can be viewed as a natural extension of NUGGET on decoder-only transformers. They are similar regarding the vector subselection (Section 2.1) but different in architecture and applications. From the perspective of architecture, different from NUGGET that reduces the last-layer representation of a transformer encoder, DODO reduces the memory and computation of self-attention in a transformer decoder. Also, DODO replaces the residual connection used by NUGGET with straightthrough estimator (Section 2.4), which naturally cancels the side-effect of the residual connection in the forward pass. From the perspective of applications, because DODO supports causal masking, it can be used for autoregressive language modeling without re-computation. NUGGET , instead, is more suitable for text similarity measurement.

heading: 8.2 Scaling the context length of transformers
text: Scaling transformers to long sequences is a popular topic in the NLP community (Tay et al., 2022). Existing work includes sparsify the attention patterns (Beltagy et al., 2020; Zaheer et al., 2020; Khalitov et al., 2023; Ding et al., 2023; Ainslie et al., 2023; Rae et al., 2020), employing lowrank or kernel methods to approximate the attention matrix computation (Choromanski et al., 2021; Katharopoulos et al., 2020), or applying recurrence (Dai et al., 2019; Yang et al., 2019; Bulatov et al., 2022). Another line of work tries to extrapolate the ability of LMs to long contexts, such as using linear bias (Press et al., 2022) or rotary position embeddings (Su et al., 2024). Recently, Bertsch et al. (2023); Tworkowski et al. (2023) applied kNN search to select a subset of tokens for attention at each layer of an encoder-decoder transformer, effectively extending the attention range of transformers. Zeng et al. (2023b) proposed to compress the context by prioritizing the “VIP tokens”, which are important to certain tasks and can be saved in specialized data structure.
Past work on efficient transformers, as shown above, mainly improves the efficiency of the selfattention. DODO instead addresses a language representation problem: It shortens the length of the sequences in the space of hidden states. From this perspective, the idea of DODO is orthogonal to most of the efficient self-attention methods, and thus can be jointly applied with most of them, e.g. kNN based methods (Tworkowski et al., 2023).
In the context of large language models, recent work focuses on compressing the prompt tokens into soft embeddings (Mu et al., 2023; Wingate et al., 2022) or encoding the supporting documents (Ge et al., 2024; Chevalier et al., 2023) into fewer vectors. LLMLingua (Jiang et al., 2023) is a coarse-to-fine prompt compression method that allocates different compression ratios over various prompt components. Some recent work tries to train LLMs with longer contexts, such as Li et al. (2023), GLM (Zeng et al., 2023a), and Claude 2 (Anthropic, 2023). Notably, Xiong et al. (2023) continue to train LLAMA to study the relationship between model performance and context length.
Researchers also explored retrieval-based methods that infuse knowledge into LM decoding, some notable work in this field includes FiD (Izacard and Grave, 2021), REALM (Guu et al., 2020), KNNLM (Khandelwal et al., 2020), and RAG (Lewis
et al., 2020). From the angle of the LLMs, Zheng et al. (2023) found that providing contexts to LLMs can help them generate truthful answers.

heading: 9 Conclusion
text: In this work, we propose DODO , a method for contextual compression for decoder-only transformers. In language modeling (Section 5) and summarization (Section 6.2), DODO is shown to generate a highly condensed representation of the context, while the results in autoencoding (Section 4) and question answering (Section 6.1) reflect that the details of the contexts can be recovered from nuggets . Moreover, in Section 6.1 we show that DODO trained with text continuation preserves the capability of instruction following. This demonstrates LLMs can encapsulate more of their input into fewer hidden states than previously realized, suggesting a new direction for efficient foundation models. Future work will explore more specialized versions of this proposal for optimizing results on individual applications, such as in dialog, supervised fine-tuning, reinforcement learning with human feedback, and in-context learning.
Ethical statement and limitations
Used artifacts In this work, we used the publicly released codes and checkpoints of LLAMA. Per the license attached to LLAMA, we agree not to re-distribute their parameters and limit the usage of the models for research purposes only.
Potential societal risks Because we only trained LLAMA on general texts, we do not think that our paper will have any additional societal impacts beyond the checkpoints, except for the privacy issues mentioned below.
Privacy issues on the datasets Our method further fine-tunes LLAMA on the Pile (Gao et al., 2020). Given the size of the Pile (Gao et al., 2020) is huge (around 800GB), we are unable to conduct effective investigations on the privacy issue on the corpus. We refer readers to Gao et al. (2020) for the discussion of the potential issues inside the data.

heading: Acknowledgment
text: We thank Ho-Lam Chung and Canwen Xu for their thoughtful discussion. We thank William Fleshman for his valuable feedback on the writing.
This work has been supported by the U.S. National Science Foundation under grant no. 2204926.
Any opinions, findings, conclusions, or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: D Prompts used in the paper
text: Here we list all the prompts used in Section 6.
D.1 Compress texts with LMs The prompt used by the LMSUMM method to generate a summary for a given text is:

 Please summarize the following text into $WORD words: $TEXT 

We replace $WORD with ⌈n · r⌉, where n is the number of words (counted by spaces) and r is a desired ratio (in Section 6, r is 10).
D.2 Question answering on SQuAD
In the SQuAD experiment (Section 6.1), a prompt is used to answer a question given a document:

 $DOCUMENT Based on the provided document, answer the following question: $QUESTION 

We replace $DOCUMENT with the context document and $QUESTION with the question.
D.3 Summarization
In the summarization experiment (Section 6.2), we use the following prompt:

 $DOCUMENT Please summarize the above document in one sentence. 

We replace $DOCUMENT with the document to be summarized.

heading: E Normalization algorithm for SQuAD answers
text: The output of the language model tends to have tokens other than the answer or have different forms. For each pair of model output and SQuAD answer, we apply the following rules:
• Convert all English numbers to digits. E.g. convert “two” to “2”.
• Replace all punctuation marks with spaces.
• Remove side spaces on both sides.
• Lowercase the string.
After these steps, a program is used to check if the model output contains the answer. We restrict the model to generate up to 64 tokens in case they generate many tokens to hit the answer. 9

heading: F Multilingual autoencoding experiments
text: For the autoencoding experiment, we adopt the architecture of LLAMAand the checkpoint of LLaMA-7B (Touvron et al., 2023a) and fine-tune the model on the Pile dataset (Gao et al., 2020). Both pretraining and fine-tuning corpus are heavily biased towards English, but the tremendous size of LLAMAenables it to process languages other than English. In this section, we conduct experiments to test the multilingual capability of DODO .
We adopt the checkpoint of DODO in Section 4 with a 10x compression ratio without further finetuning. We sampled 8 languages: Bulgarian, German, English, French, Italian, Dutch, Polish, and Russian. 10 For each language, we sampled 100 documents from the RedPajama corpus (Together
9They rarely do, as they are not optimized to cheat SQuAD. 10We did not consider non-Indo-European languages, such as Chinese and Japanese, because we found that many characters are out-of-vocabulary for LLAMA.
Computer, 2023). We truncate the document if it is longer than 512 tokens. We use BLEU (Papineni et al., 2002) and perplexity as our metrics.
The results are shown in Table 6. We can observe that DODO can still process other languages, even if it was fine-tuned on an English-only corpus.

heading: Future Work
text: Future work will explore more specialized versions of this proposal for optimizing results on individual applications, such as in dialog, supervised fine-tuning, reinforcement learning with human feedback, and in-context learning