SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jesse Mu, Xiang Lisa Li, Noah Goodman
muj@cs.stanford.edu, {xlisali,ngoodman}@stanford.edu
Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to  $40\%$  FLOPs reductions,  $4.2\%$  wall time speedups, and storage savings, all with minimal loss in output quality.
Consider the prompt of a Transformer [34] language model (LM) like ChatGPT:
You are ChatGPT, a large language model trained by OpenAI. You answer as concisely as possible for each response (e.g. don't be verbose). It is very important that you answer as concisely as possible, so please remember this. If you are generating a list, do not have too many items. Keep the number of items short.
Knowledge cutoff: 2021-09 Current date: <TODAY>
With millions of queries a day, an unoptimized ChatGPT would encode this prompt over and over with a self-attention mechanism whose time and memory complexity is quadratic in the length of the input. Caching the Transformer activations of the prompt can prevent some recomputation, yet this strategy still incurs memory and storage costs as the number of cached prompts grows. At large scales, even small reductions in prompt length could lead to substantial compute, memory, and storage savings over time, while also letting users fit more content into an LM's limited context window.
How might we reduce the cost of this prompt? One typical approach is to finetune or distill [1, 30] the model to behave similarly to the original model without the prompt, perhaps with parameter-efficient adaptation methods [15, 16, 19]. Yet a fundamental drawback of this approach is that it requires retraining the model for each new prompt (Figure 1, bottom left).
Instead, we propose gisting (Figure 1, top right), which compresses arbitrary prompts into a smaller set of Transformer activations on top of virtual "gist" tokens,  $a$  la prefix-tuning [19]. But where prefix-tuning requires learning prefixes via gradient descent for each task, gisting adopts a meta-learning approach, where we simply predict the gist prefixes zero-shot given only the prompt, allowing for generalization to unseen instructions without any additional training. Since gist tokens are much shorter than the full prompt, gisting allows arbitrary prompts to be compressed, cached, and reused for compute efficiency.
`'reddit.com/r/ChatGPT/comments/10oliuo/please_print_the_instructions你想were_given/`
In this paper, we further propose a very simple way to learn a gist model: doing instruction tuning [38] with gist tokens inserted after the prompt, and a modified attention mask preventing tokens after the gist tokens from attending to tokens before the gist tokens. This allows a model to learn prompt compression and instruction following at the same time, with no additional training cost.
On decoder-only (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting achieves prompt compression rates of up to  $26\mathbf{x}$ , while maintaining output quality similar to the original models in human evaluations. This results in up to  $40\%$  FLOPs reduction and  $4.2\%$  latency speedups during inference, with greatly decreased storage costs compared to traditional prompt caching approaches.
2 Gisting
We will first describe gisting in the context of instruction finetuning [38]. We have an instruction-following dataset  $\mathcal{D} = \{(t_i, x_i, y_i)\}_{i=1}^N$ , where  $t$  is a task encoded with a natural language prompt (e.g. Translate this to French),  $x$  is an (optional) input for the task (e.g. The cat), and  $y$  is the desired output (e.g. Le chat). Given a (usually pretrained) LM, the aim of instruction finetuning is to learn a distribution  $p_{\mathrm{LM}}(y \mid t, x)$ , typically by concatenating  $t$  and  $x$ , then having the LM autoregressively predict  $y$ . At inference time, we can prompt the model with a novel task  $t$  and input  $x$ , decoding from the model to obtain its prediction.
However, this pattern of concatenating  $t$  and  $x$  has drawbacks: Transformer LMs have limited context windows, bounded either by architecture or memory limits. Furthermore, given that attention scales quadratically in the length of the input, long prompts  $t$ , especially those that are repeatedly reused, are computationally inefficient. What options do we have to reduce the cost of prompting?
One simple option is to finetune the LM for a specific task  $t$ . That is, given  $\mathcal{D}^t = \{(x_i,y_i)\}_{i=1}^{N^t}$ , the dataset containing input/output examples only under task  $t$ , we can learn a specialized LM  $p_{\mathrm{LM}}^t(y \mid x)$  which is faster because it does not condition on  $t$ . Parameter-efficient finetuning methods such as prefix-/prompt-tuning [18, 19] or adapters [15, 16] promise to do so at a fraction of the cost of full finetuning, and newer methods like HyperTuning [25] eliminate gradient descent entirely, instead predicting the parameters of the specialized model directly from  $\mathcal{D}^t$ . Yet problems with these methods still remain: we must store at least a subset of model weights for each task, and more importantly, for each task  $t$ , we must collect a corresponding dataset of input/output pairs  $\mathcal{D}^t$  to adapt the model.
Gisting is a different approach that amortizes both (1) the inference-time cost of prompting  $p_{\mathrm{LM}}$  with  $t$  and (2) the train-time cost of learning a new  $p_{\mathrm{LM}}^t$  for each  $t$ . The idea is to learn a compressed version of  $t$ ,  $G(t)$ , such that inference from  $p_G(y \mid G(t), x)$  is faster than  $p_{\mathrm{LM}}(y \mid t, x)$ . In LM terms,  $G(t)$  will be the key/value activations on top a set of gist tokens, smaller than the number of tokens in  $t$ , yet still inducing similar behavior from the LM. Also known as a Transformer prefix [19],  $G(t)$  can then be cached and reused for compute efficiency. Crucially, we expect  $G$  to generalize to unseen tasks: given a new task  $t$ , we can predict and use the gist activations  $G(t)$  without any additional training.
2.1 A Context Distillation Perspective
An alternative way to view gisting is through the lens of distillation of an already instruction-tuned LM  $p_{\mathrm{LM}}(y \mid t, x)$ . Askell et al. [1] and Snell et al. [30] define context distillation as the process of finetuning a new LM  $p_{\mathrm{CD}}^t$  to mimic the original LM without the prompt ("context")  $t$ , via the loss
$$
\mathcal {L} _ {\mathrm {C D}} \left(p _ {\mathrm {C D}} ^ {t}, t\right) = \mathbb {E} _ {x} \left[ D _ {\mathrm {K L}} \left(p _ {\mathrm {L M}} (y \mid t, x) \| p _ {\mathrm {C D}} ^ {t} (y \mid x)\right) \right]. \tag {1}
$$
(a) Decoder-only (e.g. GPT, LLaMA)
(b) Encoder-decoder (e.g. T5)
Figure 2: Gist Masking. Attention mask modifications for (a) decoder-only and (b) encoder-decoder Transformer LMs to encourage prompt compression into gist tokens  $\langle \mathrm{G1} \rangle < \mathrm{G2}$ . In these tables, cell  $(r, c)$  shows whether token  $r$  can attend to token  $c$  during self- or cross-attention.
The insight to be gained from this perspective is that we do not need any external data  $\mathcal{D}$ : this KL objective can be approximated by finetuning  $p_{\mathrm{CD}}^t$  on a synthetic sampled dataset  $\hat{\mathcal{D}}^t = \{(\hat{x}_i, \hat{y}_i)\}$  where  $(\hat{x}_i, \hat{y}_i) \sim p_{\mathrm{LM}}(\cdot | t)$ . This is precisely the approach taken by recent work [1, 7, 30], including Wingate et al. [40], who notably learn to compress a single discrete prompt into a soft prompt via gradient descent, similar to this paper.
However, we differ from this prior work in that we are not interested in distilling just a single task, but in amortizing the cost of distillation across a distribution of tasks  $T$ . That is, given a task  $t \sim T$ , instead of obtaining the distilled model via gradient descent, we use  $G$  to simply predict the gist tokens ( $\approx$  parameters) of the distilled model, in the style of HyperNetworks [13] and HyperTuning [25]. Our "meta" distillation objective is thus (with changes highlighted in blue):
$$
\mathcal {L} _ {G} \left(p _ {G}, T\right) = \mathbb {E} _ {t \sim T, x} \left[ D _ {\mathrm {K L}} \left(p _ {\mathrm {L M}} (y \mid t, x) \| p _ {G} (y \mid G (t), x)\right) \right]. \tag {2}
$$
In the experiments we describe below, we train on synthetic instruction-following data sampled from instruction-tuned variants of GPT-3 [3, 23]. Thus, these experiments can indeed be seen as a form of context distillation for the GPT-3 series models.
3 Learning Gisting by Masking
Having just described the general framework of gisting, here we will explore an extremely simple way of learning such a model: using the LM itself as the gist predictor  $G$ . This not only leverages the pre-existing knowledge in the LM, but also allows us to learn gisting by simply doing standard instruction finetuning while modifying the Transformer attention masks to enforce prompt compression. This means that gisting incurs no additional training cost on top of standard instruction finetuning!
Specifically, we add a single gist token  $g$  to the model vocabulary and embedding matrix. Then, given a (task, input) pair  $(t,x)$ , we concatenate  $t$  and  $x$  with a set of  $k$  copies of  $g$  in between:  $(t,g_{1},\ldots ,g_{k},x)$ , e.g. Translate French:  $<G1>$ $<G2>$  The cat. The model is then restricted such that input tokens after the gist tokens cannot attend to any of the prompt tokens before the gist tokens (but they can attend to the gist tokens). This forces the model to compress the prompt information into the gist prefix, since the input  $x$  (and output  $y$ ) cannot attend to the prompt  $t$ .
Figure 2 illustrates the required changes. For decoder-only LMs such as GPT-3 [3] or LLaMA [33] that normally admit an autoregressive, causal attention mask, we simply mask out the lower-left corner of the triangle (Figure 2a). For encoder-decoder LMs (e.g. T5; [28]) with a bidirectional encoder followed by an autoregressive decoder, two changes are needed (Figure 2b). First, in the encoder, which normally has no masking, we prevent the input  $x$  from attending to the prompt  $t$ . But
2Again, the gist token is the same from  $g_{1}$  to  $g_{k}$ ; what changes is the activations on top of each token.
we must also prevent the prompt  $t$  and gist tokens  $g_{i}$  from attending to the input  $x$ , since otherwise the encoder learns different representations depending on the input. Finally, the decoder operates as normal, except during cross-attention, we prevent the decoder from attending to the prompt  $t$ .
Overall, these masking changes are extremely simple and can be implemented in roughly 10 source lines of code. See Appendix A for a sample PyTorch implementation which can be used as a drop-in replacement for attention masking in deep learning libraries such as Hugging Face Transformers [41].
4.1 Data
A dataset with a large variety of tasks (prompts) is crucial to learn gist models that can generalize. To obtain the largest possible set of tasks for instruction finetuning, we create a dataset called Alpaca+, which combines the Self-Instruct [36] and Stanford Alpaca [31] instruction tuning datasets, each consisting of  $(t,x,y)$  tuples sampled from OpenAI's text-davinci-001 and text-davinci-003 variants of GPT-3, respectively. In total, Alpaca+ has 130,321 examples, with 104,664 unique tasks  $t$ , 48,530 unique inputs  $x$ , and anywhere from 0-5 inputs per task (0.64 on average).
Note that  $\sim 59\%$  of tasks in Alpaca+ have no inputs (e.g. Write me a poem about frogs), in which case we simply omit the input  $x$ . While it is less interesting to cache such prompts since they are not input-dependent, they still serve as valuable training signal for learning prompt compression. Overall, while Alpaca+ is noisy and imperfect, Wang et al. [36] and Taori et al. [31] nevertheless show that models trained on such data achieve comparable performance to the original models from which the data is sampled, making this a promising testbed for studying gisting.
From Alpaca+ we hold out 3 validation splits: 1000 Seen prompts (with unseen, non-empty inputs); 1000 Unseen prompts (with non-empty inputs); and the 252 hand-written Human prompts and completions used in Wang et al. [36], of which  $83\%$  have non-empty inputs. The latter two splits test generalization to unseen instructions, with the Human split posing a stronger out-of-distribution (OOD) challenge: the average training prompt has  $\sim 20$  tokens, compared to  $\sim 26$  in the human split.
To demonstrate gisting across multiple Transformer LM architectures, we experiment with LLaMA-7B [33], a decoder-only GPT-style model with  $\sim 7\mathrm{B}$  parameters, and FLAN-T5-XXL [8], an encoder-decoder T5 model [28] with 11B parameters. For each of these models, we train models with a varying number of gist tokens  $k\in \{1,2,5,10\}$ , using the modified attention masks described in Section 3. To assess how the model is learning prompt compression, we calibrate performance against upper- and lower-bound baselines and a simple discrete compression strategy:
Positive Control. As an upper bound on performance, we train a model with a single gist token, but without any modifications to the attention mask. This is akin to doing standard instruction finetuning.
Negative Control. As a lower bound on performance, we train a model without access to the task  $t$ . This is similar to a "random gist token" baseline, which allows us to measure how the model would do if it failed to compress any information into the gist prefix.
Discrete Compression with TF-IDF. An alternative approach to compression is simply using fewer discrete tokens to express the same task. Achieving compression rates similar to gisting, however, requires compression far beyond any threshold of fluency. Nevertheless, as a baseline, we compute TF-IDF statistics over the set of instructions in the Alpaca+ training set to extract the most relevant keyword in each instruction. Some examples from the training set include (see Appendix G for more):
Write a letter to your boss asking for an increase in salary → salary
Given two integers, find their average  $\rightarrow$  average
We then replace each instruction in Alpaca+ with the first subword token from each keyword, resulting in compression rates equivalent to a model trained with a single gist token. Similarly to the positive control, we do standard instruction finetuning over Alpaca+ with these compressed instructions.
For full training, data, and compute details, and a link to code, see Appendix B.
Our evaluation uses a combination of automated metrics and AI- and human-assisted evaluation:
ROUGE-L. We first use ROUGE-L, a simple lexical overlap statistic [20], used in previous open-ended instruction finetuning work [37, 38]. The text-davinci- $\{001, 003\}$  completions are used as references, except for the Human split, where we use the gold-standard human reference.
ChatGPT. Next, we use ChatGPT-3.5 [22] to compare the outputs of our models to the positive control. While this is an imperfect metric, it allows for much faster and cheaper evaluation than human experiments, with an arguably more meaningful semantic signal than ROUGE-L. Recent work has found that ChatGPT can be used for text annotation and evaluation [12, 17, 35] with near-human performance, and similar model-based evaluations have been conducted with recent LMs [5, 11, 31].
Specifically, given a task  $t$ , input  $x$ , and outputs from two models  $(y_{1}, y_{2})$  identified only as Assistants A and B, ChatGPT was asked to choose which assistant response is better, explaining its reasoning in Chain-of-Thought fashion [39]. If the models produced the same output, or were equally bad, ChatGPT was allowed to call a tie. We gave examples of desired outputs in ChatGPT's prompt, and randomized the order of presentation between the models for each query to avoid order effects. The full prompt given to ChatGPT and evaluation details are in Appendix C. Using these outputs, we measure the win rate of a model against the positive control: a win rate of  $50\%$  indicates that the model is of comparable quality to a model that does no prompt compression.
Human eval. Finally, after prototyping with ChatGPT, we select the best gist compression models and do a Human evaluation on a random subset of 100 of the 252 examples in the Human validation split. For each of the 100 examples, we recruited 3 US or UK-based, English-fluent annotators from Prolific, and asked them to rate model outputs in the same style as the ChatGPT evaluation above (see Appendix D for full details, including the annotation interface). The only difference is that human participants were allowed to select "I Don't Know" in cases where they had inadequate domain knowledge to accurately judge the responses, e.g. if the question was a coding question; we drop these responses ( $\sim 10\%$ ) during analysis. With this human evaluation, we are not only interested in evaluating our final models, but also validating whether ChatGPT can be used as a reliable replacement for human annotation on this task.
ROUGE-L and ChatGPT evaluations for LLaMA-7B and FLAN-T5-XXL, with varying numbers of gist tokens, are shown in Figure 3. Models were generally insensitive to the number of gist tokens  $k$ : compressing prompts into a single token prefix did not substantially underperform larger prefixes. In fact, having too many gist tokens hurts performance in some cases (e.g. LLaMA-7B, 10 gist tokens), perhaps because the increased capacity enables overfitting to the training distribution. Thus, we use the single gist token models for the rest of the experiments in the paper, and report the exact numbers for the single-token models, with the positive, negative, and TF-IDF baselines, in Table 1.
On Seen instructions, gist models attain near-identical ROUGE and ChatGPT performance as their positive control models (48.6% and 50.8% win rates for LLaMA-7B and FLAN-T5-XXL, respectively). But we are most interested in generalization to unseen tasks, as measured by the other two splits. On Unseen prompts within the Alpaca+ distribution, we again see competitive performance: 49.7% (LLaMA) and 46.2% (FLAN-T5) win rates against the positive controls. It is on the most challenging OOD Human split where we see slight drops in win rate to 45.8% (LLaMA) and 42.5% (FLAN-T5), though these numbers are still quite competitive with the positive control. Finally, gist compression is vastly superior to discrete compression; the TF-IDF models in Table 1 only marginally outperform the negative control models across the board.
Table 2 shows the human evaluation results on the Human validation split, comparing the single gist token models to the positive control. Overall, human annotators agree with ChatGPT, with average win rates of  $52.3\%$  (vs.  $48.0\%$ ) for LLaMA-7B and  $40.6\%$  (vs.  $42.0\%$ ) for FLAN-T5-XXL. Importantly, this agreement persists at the level of individual responses. The average pairwise Cohen's  $\kappa$  among human annotators is .24 for LLaMA-7B and .33 for FLAN-T5-XXL. Because humans will often arbitrarily choose one response over another even for samples of equal quality, these numbers
Table 1: Results for single gist tokens. ROUGE-L and ChatGPT scores for Gist, TF-IDF, and positive/negative controls. Parentheses are scores normalized between positive/negative controls.
Model	Seen	Unseen	Human
ROUGE-L	ChatGPT %	ROUGE-L	ChatGPT %	ROUGE-L	ChatGPT %
LLaMA-7B	Pos	58.0 (100)	50.0 (100)	48.1 (100)	50.0 (100)	27.0 (100)	50.0 (100)
Gist	57.8 (99.2)	48.6 (92.4)	46.6 (91.0)	49.7 (98.8)	23.9 (75.4)	45.8 (84.9)
TF-IDF	38.1 (24.5)	34.5 (16.2)	34.0 (15.6)	29.3 (15.9)	16.5 (16.7)	24.6 (8.6)
Neg	31.5 (0)	31.5 (0)	31.4 (0)	25.4 (0)	14.4 (0)	22.2 (0)
FLAN-T5-XXL	Pos	50.6 (100)	50.0 (100)	45.7 (100)	50.0 (100)	23.9 (100)	50.0 (100)
Gist	48.9 (93.2)	50.8 (103.9)	43.8 (88.6)	46.2 (84.4)	21.7 (80.9)	42.5 (63.2)
TF-IDF	32.0 (25.9)	35.9 (30.5)	34.3 (31.3)	31.0 (22.1)	13.5 (9.6)	28.4 (-5.9)
Neg	25.5 (0)	29.7 (0)	29.1 (0)	25.6 (0)	12.4 (0)	29.6 (0)
(a) LLaMA-7B
(b) FLAN-T5-XXL
Figure 3: Varying the number of gist tokens. ROUGE-L and ChatGPT scores for (a) LLaMA-7B and (b) FLAN-T5-XXL for different gist tokens. Dashed lines indicate positive and negative control performance. Error bars are  $95\%$  exact binomial confidence intervals, splitting ties equally between models [10] and rounding down in favor of the positive control in case of an odd number of ties. Compression factors are calculated by computing the average token length of the validation split and dividing by the number of gist tokens.
Table 2: Human evaluation results. Win rate and inter-annotator agreement of single token gist models over positive control according to 3 Human annotators (H1-H3), their average, and ChatGPT (95% confidence intervals in parentheses), for 100 out of 252 examples in the Human validation split.
Model	Gist Win % over Pos	Agreement (Cohen's κ)
H1	H2	H3	Human (H1-H3)	ChatGPT	Human	ChatGPT
LLaMA-7B	51.1	44.5	59.8	52.3 (46.1, 58.4)	48.0 (38.0, 58.2)	.24	.29
FLAN-T5-XXL	43.0	41.9	37.2	40.6 (34.6, 46.8)	42.0 (32.2, 52.3)	.33	.29
are fairly low; however, ChatGPT shows similar levels of agreement, with average  $\kappa$  across each of the 3 human annotators at .29 for both models. These results, paired with the similar overall win rates, show that using ChatGPT is similar to simply recruiting an additional human annotator, and corroborates the broader results in Figure 3. See Appendix D for more human evaluation results, including a breakdown of agreement across annotators, and Appendix G for examples of instructions, model outputs, and human/ChatGPT judgments in the Human validation split.
Since our aim is having the gist models mimic the original models, one might ask how often the gist model is identical to the positive control. Figure A.3 in Appendix E shows how often this happens: for Seen tasks (but unseen inputs), the gist model outputs exactly match the positive control nearly  $50\%$  of the time. This drops to  $\sim 20 - 25\%$  for Unseen tasks and  $\sim 10\%$  for the OOD Human tasks.
Overall, our results show that gist models can reliably compress prompts, even to some degree those that lie outside the training distribution, especially for decoder-only LMs (LLaMA). Encoder-decoder LMs like FLAN-T5 show slightly worse OOD performance; one possible reason is that gist masking inhibits the bidirectional attention flow in the encoder, which is more challenging to adjust to than simply limiting the history of a decoder LM. Future work will be needed to investigate this hypothesis.
5.1 Failure cases
While the responses from the gist models are often indistinguishable from the positive control, we nevertheless do see failure cases where the gist token models do not perfectly compress the prompt. We give two examples where this occurs in the Human validation split: first, when the instruction contains specific details, e.g. phrases that must be copied verbatim into the output (in the snippets below, the names of Assistants A/B are hidden to ChatGPT):
Instruction: Give the news title a category. Pick a category from the list of News & Buzz, Travel, Style, Arts & Culture, Politics, Tech, and Science & Health. <G1>  
Input: The #Banksy Exhibit in Cambridge, MA is absolutely terrific.  
Assistant A (FLAN-T5-XXL gist): Culture  
Assistant B (pos control): Arts & Culture
ChatGPT choice: B