Title: Learning to Compress Prompts with Gist Tokens
Venue: NeurIPS 2023
Accepted: yes
Keywords: language models

instruction finetuning

prompt compression

distillation

context distillation

prompting

soft prompting

efficiency
Abstract: Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

Learning to Compress Prompts with Gist Tokens

Abstract

Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting , which trains an LM to compress prompts into smaller sets of 'gist' tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

1 Introduction

Consider the prompt of a Transformer [34] language model (LM) like ChatGPT: 1

You are ChatGPT, a large language model trained by OpenAI. You answer as concisely as possible for each response (e.g. don't be verbose). It is very important that you answer as concisely as possible, so please remember this. If you are generating a list, do not have too many items. Keep the number of items short. Knowledge cutoff: 2021-09 Current date: <TODAY>

With millions of queries a day, an unoptimized ChatGPT would encode this prompt over and over with a self-attention mechanism whose time and memory complexity is quadratic in the length of the input. Caching the Transformer activations of the prompt can prevent some recomputation, yet this strategy still incurs memory and storage costs as the number of cached prompts grows. At large scales, even small reductions in prompt length could lead to substantial compute, memory, and storage savings over time, while also letting users fit more content into an LM's limited context window.

How might we reduce the cost of this prompt? One typical approach is to finetune or distill [1, 30] the model to behave similarly to the original model without the prompt, perhaps with parameter-efficient adaptation methods [15, 16, 19]. Yet a fundamental drawback of this approach is that it requires retraining the model for each new prompt (Figure 1, bottom left).

Instead, we propose gisting (Figure 1, top right), which compresses arbitrary prompts into a smaller set of Transformer activations on top of virtual 'gist' tokens, a la prefix-tuning [19]. But where prefixtuning requires learning prefixes via gradient descent for each task, gisting adopts a meta-learning approach, where we simply predict the gist prefixes zero-shot given only the prompt, allowing for generalization to unseen instructions without any additional training. Since gist tokens are much shorter than the full prompt, gisting allows arbitrary prompts to be compressed, cached, and reused for compute efficiency.

p

LM

(

y

|

x

)

p

LM

(

y

|

x

)

p

LM

(

y

|

x

)

p

LM

(

y

|

t, x

)

Translate this to French: The cat <SEP> Le chat

Translate this to French: The cat <SEP> Le chat

p

G

(

y

|

G

(

t

)

, x

)

<G1> <G2> The cat <SEP> Le chat

Solve the math equation:

Summarize the article:

Translate this to French:

Solve the math equation:

Summarize the article:

Translate this to French:

Prompting

Finetuning/Distillation

Gisting

finetune

predict

t

1

t

2

t

3

Solve the math equation:

Summarize the article:

Translate this to French:

In this paper, we further propose a very simple way to learn a gist model: doing instruction tuning [38] with gist tokens inserted after the prompt, and a modified attention mask preventing tokens after the gist tokens from attending to tokens before the gist tokens. This allows a model to learn prompt compression and instruction following at the same time, with no additional training cost.

On decoder-only (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting achieves prompt compression rates of up to 26x , while maintaining output quality similar to the original models in human evaluations. This results in up to 40% FLOPs reduction and 4.2% latency speedups during inference, with greatly decreased storage costs compared to traditional prompt caching approaches.

2 Gisting

We will first describe gisting in the context of instruction finetuning [38]. We have an instructionfollowing dataset D = { ( t i , x i , y i ) } N i =1 , where t is a task encoded with a natural language prompt (e.g. Translate this to French ), x is an (optional) input for the task (e.g. The cat ), and y is the desired output (e.g. Le chat ). Given a (usually pretrained) LM, the aim of instruction finetuning is to learn a distribution p LM ( y | t, x ) , typically by concatenating t and x , then having the LM autoregressively predict y . At inference time, we can prompt the model with a novel task t and input x , decoding from the model to obtain its prediction.

However, this pattern of concatenating t and x has drawbacks: Transformer LMs have limited context windows, bounded either by architecture or memory limits. Furthermore, given that attention scales quadratically in the length of the input, long prompts t , especially those that are repeatedly reused, are computationally inefficient. What options do we have to reduce the cost of prompting?

One simple option is to finetune the LM for a specific task t . That is, given D t = { ( x i , y i ) } N t i =1 , the dataset containing input/output examples only under task t , we can learn a specialized LM p t LM ( y | x ) which is faster because it does not condition on t . Parameter-efficient finetuning methods such as prefix-/prompt-tuning [18, 19] or adapters [15, 16] promise to do so at a fraction of the cost of full finetuning, and newer methods like HyperTuning [25] eliminate gradient descent entirely, instead predicting the parameters of the specialized model directly from D t . Yet problems with these methods still remain: we must store at least a subset of model weights for each task, and more importantly, for each task t , we must collect a corresponding dataset of input/output pairs D t to adapt the model.

Gisting is a different approach that amortizes both (1) the inference-time cost of prompting p LM with t and (2) the train-time cost of learning a new p t LM for each t . The idea is to learn a compressed version of t , G ( t ) , such that inference from p G ( y | G ( t ) , x ) is faster than p LM ( y | t, x ) . In LM terms, G ( t ) will be the key/value activations on top a set of gist tokens , smaller than the number of tokens in t , yet still inducing similar behavior from the LM. Also known as a Transformer prefix [19], G ( t ) can then be cached and reused for compute efficiency. Crucially, we expect G to generalize to unseen tasks: given a new task t , we can predict and use the gist activations G ( t ) without any additional training .

2.1 A Context Distillation Perspective

An alternative way to view gisting is through the lens of distillation of an already instruction-tuned LM p LM ( y | t, x ) . Askell et al. [1] and Snell et al. [30] define context distillation as the process of finetuning a new LM p t CD to mimic the original LM without the prompt ('context') t , via the loss

Autoregressive Decoder Mask

Translate

French:

<G2>

The

cat

Translate

French:

<G2>

The

cat

Standard Mask                    Gist Mask

(a) Decoder-only (e.g. GPT, LLaMA)

Bidirectional Encoder Mask

Translate

French:

<G2>

The

cat

Translate

French:

<G2>

The

cat

Decoder Cross-Attention Mask

Translate

French:

<G2>

The

cat

Le

chat

(b) Encoder-decoder (e.g. T5)

The insight to be gained from this perspective is that we do not need any external data D : this KL objective can be approximated by finetuning p t CD on a synthetic sampled dataset ˆ D t = { (ˆ x i , ˆ y i ) } where (ˆ x i , ˆ y i ) ∼ p LM ( · | t ) . This is precisely the approach taken by recent work [1, 7, 30], including Wingate et al. [40], who notably learn to compress a single discrete prompt into a soft prompt via gradient descent, similar to this paper.

However, we differ from this prior work in that we are not interested in distilling just a single task, but in amortizing the cost of distillation across a distribution of tasks T . That is, given a task t ∼ T , instead of obtaining the distilled model via gradient descent, we use G to simply predict the gist tokens ( ≈ parameters) of the distilled model, in the style of HyperNetworks [13] and HyperTuning [25]. Our 'meta' distillation objective is thus (with changes highlighted in blue ):

In the experiments we describe below, we train on synthetic instruction-following data sampled from instruction-tuned variants of GPT-3 [3, 23]. Thus, these experiments can indeed be seen as a form of context distillation for the GPT-3 series models.

3 Learning Gisting by Masking

Having just described the general framework of gisting, here we will explore an extremely simple way of learning such a model: using the LM itself as the gist predictor G . This not only leverages the preexisting knowledge in the LM, but also allows us to learn gisting by simply doing standard instruction finetuning while modifying the Transformer attention masks to enforce prompt compression. This means that gisting incurs no additional training cost on top of standard instruction finetuning!

Specifically, we add a single gist token g to the model vocabulary and embedding matrix. Then, given a (task, input) pair ( t, x ) , we concatenate t and x with a set of k copies of g in between: ( t, g 1 , . . . , g k , x ) , e.g. Translate French: <G1> <G2> The cat . 2 The model is then restricted such that input tokens after the gist tokens cannot attend to any of the prompt tokens before the gist tokens (but they can attend to the gist tokens). This forces the model to compress the prompt information into the gist prefix, since the input x (and output y ) cannot attend to the prompt t .

Figure 2 illustrates the required changes. For decoder-only LMs such as GPT-3 [3] or LLaMA [33] that normally admit an autoregressive, causal attention mask, we simply mask out the lower-left corner of the triangle (Figure 2a). For encoder-decoder LMs (e.g. T5; [28]) with a bidirectional encoder followed by an autoregressive decoder, two changes are needed (Figure 2b). First, in the encoder, which normally has no masking, we prevent the input x from attending to the prompt t . But

we must also prevent the prompt t and gist tokens g i from attending to the input x , since otherwise the encoder learns different representations depending on the input. Finally, the decoder operates as normal, except during cross-attention, we prevent the decoder from attending to the prompt t .

Overall, these masking changes are extremely simple and can be implemented in roughly 10 source lines of code. See Appendix A for a sample PyTorch implementation which can be used as a drop-in replacement for attention masking in deep learning libraries such as Hugging Face Transformers [41].

4 Experiments

4.1 Data

A dataset with a large variety of tasks (prompts) is crucial to learn gist models that can generalize. To obtain the largest possible set of tasks for instruction finetuning, we create a dataset called Alpaca+, which combines the Self-Instruct [36] and Stanford Alpaca [31] instruction tuning datasets, each consisting of ( t, x, y ) tuples sampled from OpenAI's text-davinci-001 and text-davinci-003 variants of GPT-3, respectively. In total, Alpaca+ has 130,321 examples, with 104,664 unique tasks t , 48,530 unique inputs x , and anywhere from 0-5 inputs per task (0.64 on average).

Note that ~59% of tasks in Alpaca+ have no inputs (e.g. Write me a poem about frogs ), in which case we simply omit the input x . While it is less interesting to cache such prompts since they are not input-dependent, they still serve as valuable training signal for learning prompt compression. Overall, while Alpaca+ is noisy and imperfect, Wang et al. [36] and Taori et al. [31] nevertheless show that models trained on such data achieve comparable performance to the original models from which the data is sampled, making this a promising testbed for studying gisting.

From Alpaca+ we hold out 3 validation splits: 1000 Seen prompts (with unseen, non-empty inputs); 1000 Unseen prompts (with non-empty inputs); and the 252 hand-written Human prompts and completions used in Wang et al. [36], of which 83% have non-empty inputs. The latter two splits test generalization to unseen instructions, with the Human split posing a stronger out-of-distribution (OOD) challenge: the average training prompt has ~20 tokens, compared to ~26 in the human split.

4.2 Models

To demonstrate gisting across multiple Transformer LM architectures, we experiment with LLaMA7B [33], a decoder-only GPT-style model with ~7B parameters, and FLAN-T5-XXL [8], an encoderdecoder T5 model [28] with 11B parameters. For each of these models, we train models with a varying number of gist tokens k ∈ { 1 , 2 , 5 , 10 } , using the modified attention masks described in Section 3. To assess how the model is learning prompt compression, we calibrate performance against upper- and lower-bound baselines and a simple discrete compression strategy:

Positive Control. As an upper bound on performance, we train a model with a single gist token, but without any modifications to the attention mask. This is akin to doing standard instruction finetuning.

Negative Control. As a lower bound on performance, we train a model without access to the task t . This is similar to a 'random gist token' baseline, which allows us to measure how the model would do if it failed to compress any information into the gist prefix.

Discrete Compression with TF-IDF. An alternative approach to compression is simply using fewer discrete tokens to express the same task. Achieving compression rates similar to gisting, however, requires compression far beyond any threshold of fluency. Nevertheless, as a baseline, we compute TF-IDF statistics over the set of instructions in the Alpaca+ training set to extract the most relevant keyword in each instruction. Some examples from the training set include (see Appendix G for more):

Wethen replace each instruction in Alpaca+ with the first subword token from each keyword, resulting in compression rates equivalent to a model trained with a single gist token. Similarly to the positive control, we do standard instruction finetuning over Alpaca+ with these compressed instructions.

For full training, data, and compute details, and a link to code, see Appendix B.

4.3 Evaluation

Our evaluation uses a combination of automated metrics and AI- and human-assisted evaluation:

ROUGE-L. We first use ROUGE-L, a simple lexical overlap statistic [20], used in previous openended instruction finetuning work [37, 38]. The text-davinci-{001,003} completions are used as references, except for the Human split, where we use the gold-standard human reference.

ChatGPT. Next, we use ChatGPT-3.5 [22] to compare the outputs of our models to the positive control. While this is an imperfect metric, it allows for much faster and cheaper evaluation than human experiments, with an arguably more meaningful semantic signal than ROUGE-L. Recent work has found that ChatGPT can be used for text annotation and evaluation [12, 17, 35] with near-human performance, and similar model-based evaluations have been conducted with recent LMs [5, 11, 31].

Specifically, given a task t , input x , and outputs from two models ( y 1 , y 2 ) identified only as Assistants A and B, ChatGPT was asked to choose which assistant response is better, explaining its reasoning in Chain-of-Thought fashion [39]. If the models produced the same output, or were equally bad, ChatGPT was allowed to call a tie. We gave examples of desired outputs in ChatGPT's prompt, and randomized the order of presentation between the models for each query to avoid order effects. The full prompt given to ChatGPT and evaluation details are in Appendix C. Using these outputs, we measure the win rate of a model against the positive control: a win rate of 50% indicates that the model is of comparable quality to a model that does no prompt compression.

Human eval. Finally, after prototyping with ChatGPT, we select the best gist compression models and do a Human evaluation on a random subset of 100 of the 252 examples in the Human validation split. For each of the 100 examples, we recruited 3 US or UK-based, English-fluent annotators from Prolific, and asked them to rate model outputs in the same style as the ChatGPT evaluation above (see Appendix D for full details, including the annotation interface). The only difference is that human participants were allowed to select "I Don't Know" in cases where they had inadequate domain knowledge to accurately judge the responses, e.g. if the question was a coding question; we drop these responses (~10%) during analysis. With this human evaluation, we are not only interested in evaluating our final models, but also validating whether ChatGPT can be used as a reliable replacement for human annotation on this task.

5 Results

ROUGE-L and ChatGPT evaluations for LLaMA-7B and FLAN-T5-XXL, with varying numbers of gist tokens, are shown in Figure 3. Models were generally insensitive to the number of gist tokens k : compressing prompts into a single token prefix did not substantially underperform larger prefixes. In fact, having too many gist tokens hurts performance in some cases (e.g. LLaMA-7B, 10 gist tokens), perhaps because the increased capacity enables overfitting to the training distribution. Thus, we use the single gist token models for the rest of the experiments in the paper, and report the exact numbers for the single-token models, with the positive, negative, and TF-IDF baselines, in Table 1.

On Seen instructions, gist models attain near-identical ROUGE and ChatGPT performance as their positive control models (48.6% and 50.8% win rates for LLaMA-7B and FLAN-T5-XXL, respectively). But we are most interested in generalization to unseen tasks, as measured by the other two splits. On Unseen prompts within the Alpaca+ distribution, we again see competitive performance: 49.7% (LLaMA) and 46.2% (FLAN-T5) win rates against the positive controls. It is on the most challenging OOD Human split where we see slight drops in win rate to 45.8% (LLaMA) and 42.5% (FLAN-T5), though these numbers are still quite competitive with the positive control. Finally, gist compression is vastly superior to discrete compression; the TF-IDF models in Table 1 only marginally outperform the negative control models across the board.

Table 2 shows the human evaluation results on the Human validation split, comparing the single gist token models to the positive control. Overall, human annotators agree with ChatGPT, with average win rates of 52.3% (vs. 48.0%) for LLaMA-7B and 40.6% (vs. 42.0%) for FLAN-T5-XXL. Importantly, this agreement persists at the level of individual responses. The average pairwise Cohen's κ among human annotators is .24 for LLaMA-7B and .33 for FLAN-T5-XXL. Because humans will often arbitrarily choose one response over another even for samples of equal quality, these numbers

(b)

Seen

Unseen

Human

Model

ROUGE-L

ChatGPT %

ROUGE-L

ChatGPT %

ROUGE-L

ChatGPT %

LLaMA-

Pos

58.0 (100)

50.0 (100)

48.1 (100)

50.0 (100)

27.0 (100)

50.0 (100)

7B

Gist

57.8 (99.2)

48.6 (92.4)

46.6 (91.0)

49.7 (98.8)

23.9 (75.4)

45.8 (84.9)

TF-IDF

38.1 (24.5)

34.5 (16.2)

34.0 (15.6)

29.3 (15.9)

16.5 (16.7)

24.6 (8.6)

Neg

31.5 (0)

31.5 (0)

31.4 (0)

25.4 (0)

14.4 (0)

22.2 (0)

FLAN-

Pos

50.6 (100)

50.0 (100)

45.7 (100)

50.0 (100)

23.9 (100)

50.0 (100)

T5-XXL

Gist

48.9 (93.2)

50.8 (103.9)

43.8 (88.6)

46.2 (84.4)

21.7 (80.9)

42.5 (63.2)

TF-IDF

32.0 (25.9)

35.9 (30.5)

34.3 (31.3)

31.0 (22.1)

13.5 (9.6)

28.4 (-5.9)

Neg

25.5 (0)

29.7 (0)

29.1 (0)

25.6 (0)

12.4 (0)

29.6 (0)

20x10x

4x

2x

40

50

1

2

5

10

Compression factor

ROUGE-L

Seen

16x 8x

3x

2x

35

40

45

1

2

5

10

Compression factor

Unseen

26x13x

5x

3x

16

20

24

1

2

5

10

Compression factor

Human

20x10x

4x

2x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

ChatGPT win rate

16x 8x

3x

2x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

26x13x

5x

3x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

(a)

LLaMA-7B

20x10x

4x

2x

25

30

35

40

45

50

1

2

5

10

Compression factor

ROUGE-L

Seen

15x 8x

3x

2x

30

35

40

45

1

2

5

10

Compression factor

Unseen

26x13x

5x

3x

12

15

18

21

24

1

2

5

10

Compression factor

Human

20x10x

4x

2x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

ChatGPT win rate

15x 8x

3x

2x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

26x13x

5x

3x

20%

25%

30%

35%

40%

45%

50%

1

2

5

10

# gist tokens

FLAN-T5-XXL

are fairly low; however, ChatGPT shows similar levels of agreement, with average κ across each of the 3 human annotators at .29 for both models. These results, paired with the similar overall win rates, show that using ChatGPT is similar to simply recruiting an additional human annotator, and corroborates the broader results in Figure 3. See Appendix D for more human evaluation results, including a breakdown of agreement across annotators, and Appendix G for examples of instructions, model outputs, and human/ChatGPT judgments in the Human validation split.

Since our aim is having the gist models mimic the original models, one might ask how often the gist model is identical to the positive control. Figure A.3 in Appendix E shows how often this happens: for Seen tasks (but unseen inputs), the gist model outputs exactly match the positive control nearly 50% of the time. This drops to ~20-25% for Unseen tasks and ~10% for the OOD Human tasks.

Overall, our results show that gist models can reliably compress prompts, even to some degree those that lie outside the training distribution, especially for decoder-only LMs (LLaMA). Encoder-decoder LMs like FLAN-T5 show slightly worse OOD performance; one possible reason is that gist masking inhibits the bidirectional attention flow in the encoder, which is more challenging to adjust to than simply limiting the history of a decoder LM. Future work will be needed to investigate this hypothesis.

5.1 Failure cases

While the responses from the gist models are often indistinguishable from the positive control, we nevertheless do see failure cases where the gist token models do not perfectly compress the prompt. We give two examples where this occurs in the Human validation split: first, when the instruction contains specific details , e.g. phrases that must be copied verbatim into the output (in the snippets below, the names of Assistants A/B are hidden to ChatGPT):

While it is unclear why only the gist models exhibit this behavior, these issues can likely be mitigated with more careful sampling techniques.

i

Average KV Cache Length = 26

ii

Average KV Cache Length = 1

6 Compute, Memory, and Storage Efficiency

Finally, we return to one of the central motivations of this paper: what kind of efficiency gains does gisting enable? To answer this question, we compare the compute requirements (CUDA wall time and FLOPs) during inference with the single-token gist models using different caching strategies:

No caching : just encoding the full prompt t .

Instruction caching : caching the activations of the uncompressed instruction t (keys and values for all layers) into what is called the KVcache . This is the most common caching behavior for Transformer inference [4, 26] and is supported in libraries like Hugging Face Transformers [41]. However, it is only applicable to decoder-only models, since in models with bidirectional encoders like T5, the instruction representations t depend on the input x .

Gist caching : Compressing the prompt into the gist prefix G ( t ) .

Table 3 displays the results of profiling a single forward pass through the model (i.e. one step of decoding with a single input token) with PyTorch [24] 2.0, averaged across the 252 Human instructions. Gist caching improves significantly over unoptimized models, with 40% FLOPs savings and 4-7% lower wall time for both models. Note that at these (relatively) small scales, the wall time improvements are smaller than the FLOPs reductions because much of the inference latency is caused by moving tensors from high-bandwidth memory (HBM) to the chip compute cores, i.e. what Pope et al. [26] call the 'memory time'. Larger sequence lengths and batch sizes will lead to additional speedups, as the overall latency becomes dominated by the actual matrix computations.

For LLaMA-7B, the picture is more nuanced when compared to caching the full instruction. Compute improvements of gist caching are smaller: a negligible decrease in FLOPs (0.11%) and a modest 1% speedup in wall time. This is because the FLOPs required for a Transformer forward pass is dominated by processing of the new input tokens, rather than self-attention with the KV cache. For example, a forward pass through LLaMA-7B with a single input token and a 2000-length KVcache is only ~10% more expensive than the same forward pass with no KV cache-see Appendix F for more details. Nevertheless, this small decrease in FLOPs leads to a disproportionate decrease in wall time (1%), likely because the self-attention computations are slower relative to their FLOPs contribution.

At large scales and with heavily reused prompts, a 1% latency speedup can still accumulate into significant cost savings over time. More importantly, however, there are key benefits of gist caching over instruction caching besides latency: compressing 26 tokens into 1 gives more space in the input context window, which is bounded by absolute position embeddings or GPU VRAM. For example, for LLaMA-7B, each token in the KV cache requires 1.05 MB storage. 3 While the total contribution of the KV cache relative to the memory needed for LLaMA-7B inference is negligible at the prompt lengths we tested, an increasingly common scenario is developers caching many prompts across a large number of users, where storage costs quickly add up. In these scenarios, gisting allows caching of up to 26x more prompts than full instruction caching, using the same amount of storage!

7 Additional Related Work

Gisting builds upon past work in (parameter-efficient) instruction finetuning and context distillation, as discussed in Section 2. Here we will outline some additional connections to related work:

Adapting LMs without Backprop. As mentioned in Section 2, gisting can be viewed as a way to adapt LMs without gradient descent, by predicting the the prefix ( ≈ parameters) of the adapted model. Similarly, HyperTuning [25] predicts the prefix of a model for a task using (input, output) pairs for that task. If HyperTuning is a 'few-shot' adaptation method, predicting the prefix from few-shot examples, then gisting can be seen as a 'zero-shot' version, predicting the prefix from the language instruction alone. Gisting also has the additional benefit over HyperTuning of being conceptually simpler: instead of training a separate LM to predict the prefix, the LM itself is used as the HyperNetwork [13], with only a tiny change to the attention masks needed for prompt compression.

Compression and memory in transformers. The idea of 'compressing' prompts is closely related to previous attempts at storing past representations to improve memory and long-range sequence modeling in Transformers [9, 21, 27, 42, 43]. In particular, the Compressive Transformer [27] compresses transformer activations into a smaller compressed memory using a learned convolutional operator. Gisting can be seen as a variant of the Compressive Transformer with 3 key differences. First, the compression function is not a separately learned function, but the LM's own self-attention mechanism, controlled by an input-dependent gist token. Second, the compression function is learned jointly with instruction finetuning via the standard language modeling loss, not a specialized auxiliary reconstruction loss as in [27]. Finally, our task of interest is not long-range sequence modeling, but caching and reusing instruction following prompts for efficiency reasons.

Sparse attention mechanisms. By restricting attention masks, gisting draws inspiration from efficient/sparse attention methods in Transformers (see [32] for review). For example, some sliding window attention mechanisms [2, 6] may remove the need to keep the entire KV cache around during decoding, but these more general methods are not optimized for caching arbitrary parts of the input sequence of varying length, which prompt compression demands. In light of this, gisting can be viewed as an input-dependent sparse attention mechanism specifically aimed at improving efficiency of the prompting workflow now commonly used in LMs.

8 Discussion and Limitations

In this paper we presented gisting, a framework for prompt compression in LMs, and a simple way of implementing gist models by modifying Transformer attention masks that incurs no additional cost over standard instruction finetuning. Gisting can be seen either as a modified form of instruction finetuning or a method for (meta-)context distillation of an LM. Gist models can compress unseen OODprompts up to 26x while maintaining output quality, resulting in up to 40% FLOPs reduction and 4.2% wall clock speedups over unoptimized models, and enabling new methods for prompt caching in encoder-decoder models. While wall-time improvements for decoder-only LMs are smaller, gisting nevertheless enables caching 1 order of magnitude (26x) more prompts relative to full instructions.

Gisting is a promising method for improving LM efficiency, but carries some limitations. While gisting seems to succeed in capturing the 'gist' of instructions (hence the name), achieving such compression necessarily results in some loss of nuance of the original instruction; Secton 5.1 illustrates a concrete failure case we observed. Since the behavior of LMs on edge cases is already not well understood, it is especially important for practitioners to carefully evaluate whether the compute/accuracy tradeoff of gisting is sufficiently safe and robust for their use case, before deployment.

Nevertheless, we believe gisting opens several interesting directions for future work. First, the masking method presented here can be easily integrated into existing instruction finetuning workflows, but another exciting approach is to retrofit an existing, frozen LM by training a smaller model to compress prompts, if finetuning the larger LM is inconvenient. Second, the largest efficiency gains from gisting will result from compressing very long prompts, for example k -shot prompts for large k that may even exceed a single context window. Finally, compression performance can likely be improved through 'gist pretraining': first learning to compress arbitrary spans of natural language, before then learning prompt compression. Such objectives could be devised by inserting gist tokens into other pretraining objectives, perhaps during language modeling or T5's span corruption objective.

Acknowledgments and Disclosure of Funding

We thank the Stanford Alpaca team, especially Xuechen Li, with assistance with Alpaca data and finetuning, and Gabriel Poesia for the codebase used to collect human evaluations. Additional thanks to Xiuyu Li for help debugging and ensuring reproducibility of the open-source codebase. JM is supported by the Open Philanthropy AI Fellowship and XLL is supported by the Stanford Graduate Fellowship.

References

A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05150 , 2020.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

C. Chen. Transformer inference arithmetic. https://kipp.ly/blog/ transformer-inference-arithmetic/ , 2022.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality. https://vicuna.lmsys.org/ , 2023.

R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

E. Choi, Y. Jo, J. Jang, and M. Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint arXiv:2206.11349 , 2022.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, 2019.

W. J. Dixon and F. J. Massey Jr. Introduction to statistical analysis. McGraw-Hill, 1951.

X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley. edu/blog/2023/04/03/koala/ .

F. Gilardi, M. Alizadeh, and M. Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 , 2023.

D. Ha, A. Dai, and Q. V. Le. HyperNetworks. In International Conference on Learning Representations , 2017.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

F. Huang, H. Kwak, and J. An. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. arXiv preprint arXiv:2302.07736 , 2023.

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, 2021.

X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.

C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating Wikipedia by summarizing long sequences. In International Conference on Learning Representations , 2018.

OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt , 2022.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , pages 27730-27744, 2022.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 32, 2019.

J. Phang, Y. Mao, P. He, and W. Chen. HyperTuning: Toward adapting large language models without back-propagation. arXiv preprint arXiv:2211.12485 , 2022.

R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102 , 2022.

J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2020.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified Text-to-Text Transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3505-3506, 2020.

C. Snell, D. Klein, and R. Zhong. Learning by distilling context. arXiv preprint arXiv:2209.15189 , 2022.

R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/ stanford_alpaca , 2023.

Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6):1-28, 2022.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou. Is ChatGPT a good NLG evaluator? A preliminary study. arXiv preprint arXiv:2303.04048 , 2023.

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 5085-5109, 2022.

J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , 2022.

D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 5621-5634, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/ 2022.findings-emnlp.412 .

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6 .

Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations , 2022.

H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv, N. Duan, and W. Chen. Poolingformer: Long document modeling with pooling attention. In International Conference on Machine Learning , pages 12437-12446. PMLR, 2021.

A Example PyTorch Implementation of Gist Masking

See Listing A.1 for a sample annotated implementation of gist masking. This PyTorch implementation relies on basic NumPy-style tensor operations and can thus be adapted easily to a framework like JAX.

B Data, Training, Evaluation, and Compute Details

Code, data, and model checkpoints are available at https://github.com/jayelm/gisting .

Data. For LLaMA-7B, we used a maximum sequence length of 512 tokens during training and evaluation, except with the Human validation split, where the maximum length was increased to 768 (the Human instructions are longer). Examples longer than this length are truncated from the end. For FLAN-T5-XXL, we set a maximum input length (task t + input x ) of 128 and a maximum output length of 256, except again for the Human split, where the maximum input and output lengths were both set to 384. For both models, we set a maximum generation length of 512 tokens. These lengths were chosen such that < 1% of examples across the board were truncated during training and evaluation for both models.

Training. Full hyperparameters for training runs are located in Table A.1. These parameters were adapted from previous published work finetuning LLAMA/FLAN-T5. For LLaMA-7B, parameters are identical to those used in training Alpaca Taori et al. [31]. For FLAN-T5-XXL, parameters are identical to those used in training T k -INSTRUCT [37], except with a 5e-5 learning rate, as used in the T k -INSTRUCT GitHub repository, 4 rather than the 1e-5 learning rate in the paper.

LLaMA-7B was trained for 3000 steps, while FLAN-T5-XXL was trained for 16000 steps. Since there are about 130k examples in Alpaca+, given the batch sizes in Table A.1 this corresponds to about ~3 epochs and ~2 epochs of training, respsectively. These numbers, again, are identical to Taori et al. [31] and Wang et al. [36]. We note that the training time is relatively flexible; for example, we did not see substantial gains training beyond 1 epoch for FLAN-T5-XXL.

Evaluation. During evaluation and benchmarking, we simply greedily decoded the most likely sequence. We saw limited gains from beam search with beam size B = 4 .

Compute. Experiments were run on a cluster machine with 4xA100-SXM4-80GB NVIDIA GPUs, 480GB RAM, and 16 CPUs, using PyTorch 2.0 [24], Hugging Face Transformers [41], and DeepSpeed [29]. Training runs take about ~7 hours to complete for LLaMA-7B and ~25 hours for FLAN-T5XXL. Benchmarking results were obtained on the same machine, but using just 1 of the A100 GPUs.

C Details of ChatGPT Evaluation

We used the ChatGPT API, specifically the chatgpt-3.5-turbo engine, to run our ChatGPT evaluation experiments over a period of 2 weeks between March 27 and April 7, 2023.

The full prompt given to ChatGPT is located in Listing A.2, and contains 4 examples of desired output from ChatGPT, including preferring factually accurate responses (Example 1), preferring responses that follow the instruction, even if imperfect (Examples 2 and 3), and examples of models

being equally wrong (Examples 4). For the two models under comparison, we randomized the order of presentation of each model as either Assistant A or Assistant B, to avoid order effects.

ChatGPT was instructed to only respond in JSON format, outputting first a reason key followed by a choice key, to encourage chain-of-thought reasoning [39]. On rare occasions ( < 0 . 25% of the time), ChatGPT would output a response that did not conform to the requested JSON format (e.g. it would just give an unstructured paragraph). In these cases we manually went through and converted these responses to JSON, without altering ChatGPT's reasoning.

In total, we collected ~22.5k judgments from ChatGPT for an estimated cost of $29.28. The full outputs for each model across the Alpaca+ validation splits, as well as ChatGPT's responses and choices, are available in the code link above.

D Additional Human Evaluation Details and Results

D.1 Experimental Details

For each of the 100 examples randomly selected from the Human validation split, we recruited 3 US or UK-based, English-fluent annotators from Prolific, an online crowdsourcing platform. Experiments were IRB approved under a generic human experiments IRB given to the authors.

The annotation interface given to Prolific crowdworkers is located in Figure A.1. To verify task comprehension, participants were shown two simple examples before the main body of the task (Figure A.2), and were required to answer correctly before proceeding. We compensated participants USD $14.35/hour for an estimated cost (including Prolific fees) of USD $141.64.

D.2 Additional Results

See Table A.2 for a breakdown of Cohen's κ between human annotators and ChatGPT. We used a weighted version of Cohen's κ with linear weights, since the response scale is ordinal (e.g. 'tie' is a closer judgment to 'pos control win' than 'gist win').

E Exact Match Results

See Figure A.3 for a plot of exact match rates for the gist and positive control models (as measured by exact string match).

F Additional FLOPs details

The FLOPs required for a Transformer formward pass with varying KV cache lengths can be estimated by modifying existing equations to account for self-attention back to the KV cache. As an example, we modify the FLOPs equations used for computing FLOPs in the Chinchilla paper (Appendix F in [14]). Let seq_len_with_past = seq_len + kv_cache_len. Then the modified Transformer FLOPs equations are:

Embeddings

2 × seq_len × vocab_size × d_model

Judging AI Assistants

Progress: 12/50

User Instruction: Write a good Twitter Bio. Try to cover all the provided main points in your generated bio.

User Input: Name: Jane Main points: - Architect - Interested in Tech Written in first person Use casual tone

Assistant A: Jane is an architect who is interested in tech. She loves to solve problems and create beautiful things.

Assistant B: I'm Jane, an architect who loves to explore the latest tech trends. I'm always looking for new ways to make our lives easier and more convenient. Follow me for the latest tech news, tips, and inspiration!

Which Assistant is more helpful?

Remember:

When making your judgment, please consider whether the response

attempts to follow the provided instruction, and

is factually accurate.

If you can't decide , e.g. because the responses are the same or equally good or bad, you may call a tie, but please use this response sparingly.

Finally , if you do not understand the question being asked: you may still be able to make

an educated guess about which Assistant is better, for example if one assistant is clearly

not following the directions. If so, please do! Otherwise, click the

"I don't know" button. There is no need to look up additional information or spend a huge amount of time on an example; just click "I don't know.".

A

B

TIE

I DON'T KNOW

$

%

7,(

,GLYPH<c=3,font=/AAAAAB+font000000002a088ebd> '21GLYPH<c=10,font=/AAAAAB+font000000002a088ebd> 7 GLYPH<c=3,font=/AAAAAB+font000000002a088ebd> . 1 2 :

Attention (Single Layer)

Key, query, and value projections : 2 × 3 × seq_len × d_model × (key_size × num_heads)

Key and query logits : 2 × seq_len × seq_len_with_past × (key_size × num_heads)

Softmax : 3 × num_heads × seq_len × seq_len_with_past

Softmax @ query reductions : 2 × seq_len × seq_len_with_past × (key_size × num_heads)

Final linear : 2 × seq_len × (key_size × num_heads) × d_model

Dense Block

2 × seq_len × (d_model × ffw_size + d_model × ffw_size)

Final Logits

2 × seq_len × d_model × vocab_size

Total Forward Pass FLOPs

embeddings + num_layers × (attention_single_layer + dense_block) + final_logits

It can be seen that only 3 operations in each attention layer depend on the KV cache size, and they take up a relatively insignificant amount of FLOPs. As an illustrative example, Figure A.4 shows the relative FLOPs contributions within a single layer of attention for LLaMA-7B, assuming a 2000-length KV cache and a single input token. Operations dependent on the KV cache constitute at most ~10% of the total attention layer FLOPs; the rest are used in KQV projections and dense layers for processing the single new input token.

Given a KV cache compression rate of 26, as observed in our Human validation split, the Chinchilla equations predict a relative improvement of Gist caching of 0.12%. This is extremely close to the 0.11% improvement actually observed in Table 3. These results show that optimizing the KV cache size does not actually lead to huge compute speedups during Transformer inference, at least for relatively small prompt lengths. Nevertheless, there are clear memory and storage benefits to be gained from prompt compression, as discussed in Section 6.

G Additional Examples of Model Outputs

See Listing A.3 for additional examples of instructions, inputs, model completions (for LLaMA-7B and FLAN-T5-XXL across Pos, Gist, TF-IDF, and Neg conditions), human judgments, and ChatGPT judgments (+reasons). Examples are not cherry-picked; we enumerate the first 10 examples of the Alpaca+ Human validation split.

0%

10%

20%

30%

40%

50%

Seen

Unseen

Human

% Pos Control Exact Match

LLaMA-7B

FLAN-T5-XXL

dense block

180 M (51.9%)

kqv projections

101 M (29%)

final linear

34 M (9.7%)

kq logits

16 M (4.7%)

softmax query

reductions

16 M (4.7%)

softmax

192 k (0.1%)

depends on KV cache size

processing for single input token

Example 1

Example 6

Instruction: Write a to-do list based on the given information. <G1> Input: Daily tasks in the pharmacy

Reference Output: - Police sidewalk and entrance for litter

-Assess window displays to make sure they are not outdated or faded

Clean all doors and remove outdated and unnecessary signs

-Fill bags at each checkout area

-Clean and straighten checkout counters

-Tidy up seating/waiting area

-Face (pull product forward) in all departments

Check one department a day for short and outdated product

-Empty all trash receptacles

Vacuum (when the store is closed or only if customer traffic allows for it during store hours) TF-IDF Word : information

LLaMA-7B

Pos : -Check inventory of medications

-Reorder medications

-Prepare patient profiles

Enter patient information into the computer system

-Verify patient information

-Monitor expiration dates of medications

-Clean and restock shelves

Answer telephone and provide assistance as needed

-Process prescriptions

-Refill prescriptions

-Monitor inventory of supplies

-Stock shelves

-Clean and restock shelves

Answer telephone and provide assistance as needed

Gist

: - Check inventory and order new supplies

-Refill prescriptions

-Answer customer questions

-Process insurance paperwork

-Clean the pharmacy

-Stock shelves

-Count pills

-Take customer phone calls

-Monitor expiration dates on medications

-Update patient profiles

-Perform administrative tasks

-Review patient charts

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Maintain inventory

-Monitor controlled substances

-Perform quality control tests

-Review patient profiles

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Maintain inventory

-Monitor expiration dates on medications

-Update patient profiles

-Perform administrative tasks

-Review patient charts

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Stock shelves

-Count pills

-Take customer phone calls

-Monitor expiration dates on medications

-Update patient profiles

-Perform administrative tasks

-Review patient charts

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Maintain inventory

-Monitor expiration dates on medications

-Update patient profiles

-Perform administrative tasks

-Review patient charts

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Maintain inventory

-Monitor expiration dates on medications

-Update patient profiles

-Perform administrative tasks

-Review patient charts

-Contact doctors about prescriptions

-Answer questions from customers

-Process payments

-Maintain inventory

-Monitor expiration dates TF-IDF : 1.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDue to the computationally inefficient nature of long prompts for language models (LMs) today, this work proposes one framework to compress prompts into a smaller set of \"gist\" virtual tokens. Different from existing work that distills the context for a single NLP task, this work focuses on a distribution of tasks by predicting the gist tokens (via LMs learned embedding) for each task, seamlessly built upon instruction tuning. The design of their generator G would be the key to their success of the generality to gist tokens. The experiments show the effectiveness of their designs.\n\nSTRENGTHS:\n1. As soft prompts limit themselves in the computational efficiency, and fine-tuning is limited by the bulk nature by retraining to adapt LMs to each specific task, this work combines the strengths of these two, by learning some virtual tokens seamlessly upon the supervised instruction tuning (a common practice in current LLM deployment and serving). To me, the simple design working well for both training-time and inference-time is elegant and interesting;\n2. The compression ratio looks really promising, making their technical contributions solid and interesting to some practitioners;\n3. The paper is nearly very structured, and the experiments on some instruction tuning datasets look convincing.\n\nWEAKNESSES:\nI still have some confusions about its working mechanisms, regarding unseen prompts, and # of gist tokens (newly added vocabulary to the LM). \n\nFirstly, is it learning one gist token (concatenate k of them in training) per task, and for new task, we do need to add more vocabularies? So it would be like one task one gist token?\n\nSecondly, for unseen and seen prompts, is the training of gist tokens like \"average of the context distillation loss\" over all seen prompts for one particular task, and for unseen prompt, but still the same task type, you simply re-use existing gist tokens (the gist token embeddings would be updated by the cross-attention by the unseen prompts and learned embeddings in inference, though no updates)? Hope my understandings are correct. \n\nI am happy to raise my scores if you can help me address them, and make me feel more interested.\n\nQUESTIONS:\nSee my weaknesses part. Clarifying these in your paper clearly would at least be useful to some readers.\n\nLIMITATIONS:\nIn addition to what they claimed in their paper,\n\nThe starting point of their proposal is significant to the community, as deploying LLMs with structured, templated prompts are becoming the common practice to a wide range of NLP, or general AI tasks, reducing the computational budgets would be beneficial to a lot of practitioners. However, their framework still require the access to the model logits in their in-context distillation loss, restricting the impacts to white-box LMs. And more precisely, how to reduce the costs for black-box LLMs is more interesting to others.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
Prompting is the current way of using LLMs, but it occupies the context spaces. Instead of training the LLMs (e.g. fine-tuning), the paper presents a way to compress the prompt into gist tokens, which can be efficiently cached and reused. The method shows 26x compression rate, 40% FLOP reduction and wall clock time speed up.

STRENGTHS:
(1) The idea is well motivated and straight-forward.
(2) The implementation through masking is well derived, simple, and well illustrated. 
(3) The baselines for experiments are well proposed.
(4) The experiment results support the proposed method well (Neg < TF-IDF < Gist <Pos) and is intuitive.
(5) The failure cases are shown and well explained.

WEAKNESSES:
(1) The method achieve 4% wall-clock time reduction, which is not significant, especially when it compromises the accuracy. The claim that "longer sequence length and larger batch sizes can lead to higher speedup" (Line 267) is not shown quantatively. 
(2) Compressing from 26 tokens to 1 token is not significant, compared to usually 2K context length of the model.

QUESTIONS:
Please address the weakness above. In particular,  when does prompt has longer sequence length? What would be the corresponding speedup?

LIMITATIONS:
The compression is lossy. The reviewer thinks this is significant because the benefits are not clear yet.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of wasted compute and waste context window space from repeatedly encoding a prompt.\n\nThe authors propose gisting, in which gist tokens are inserted after the prompt, and the attention mask is modified such that tokens after the gist tokens cannot attend to tokens before the gist tokens. This forces the gist tokens to encode and compress the prompt.\n\nExperiments are conducted with two models, LLaMA 7B and FLAN-T5-XXL 11B using a dataset the authors create from Self-Instruct and Stanford Alpaca. A hold-out set of 1000 seen, 1000 unseen, and 252 human prompts are used for evaluation.\n\nThe evaluation is conducted using a ROGUE-L score, ChatGPT3.5 model evaluation, and human evaluation score conducted on a subset of the human prompts.\n\nCompared to an “upper-bound” of training a model with a single gist token, but no modifications to the attention mask, gisting scores near this “upper-bound” using both ROGUE-L and ChatGPT3.5.\n\nHuman evaluation results show gisting winning 52.3% and 40.6% of the time over the “upper-bound” baseline for LLaMA and FLAN-T5-XXL respectively.\n\nCompression rates from gisting are material compared with no caching strategy, but more modest compared with caching the full instruction.\n\nSTRENGTHS:\nGisting is simple to implement, requiring only adding one or more gist tokens, and a few modifications in attention masks\n\nGisting learns prompt compression and instruction following at the same time, incurring no additional training\n\nGisting can be used even with unseen prompts and has some generalization capability\n\nExperiments use relatively large models and a relatively large dataset\n\nPaper is clearly written\n\nWEAKNESSES:\nThe evaluation scheme is the primary weakness of the paper. Human evaluators are critical given the nature of the tasks in the dataset (which consist of many open-ended generation tasks) to check how reliable ChatGPT (along with the prompt the authors present in the Appendix) is as an evaluator. We see that human evaluators have low agreement (kappa of 0.24 and 0.33, Table 2). The overall experimental results would be more convincing if human agreement was high. For example, it would be interesting to see results on a high human agreement subset (say tasks with clear objective answers) to see if Gist’s Win % over Pos was still near 50%. Furthermore, Negative Control scores ~25-30% using ChatGPT on various validation splits, which is high considering that ~59% of the tasks in Alpaca+ have no input, and the task is the only information used to generate the output. It is plausible that human evaluators have very high agreement on those outputs generated with Negative Control, but with much lower scores than ChatGPT scores, which would indicate ChatGPT + author’s prompt as an unreliable evaluator. Presented as is, it is hard to tell whether 1) the gisting process is doing well at compressing the task t, resulting in a similar performance as using the original prompt or 2) there is a large amount of randomness inherent in the evaluation process (from factors such as differing human preferences), resulting in a near 50% score (as measured by ChatGPT). These observations, along with the odd fact that reducing the compression rate does not materially improve the scores as measured by ChatGPT casts into doubt the experimental results.  \n\nAs Section 5.1 shows, for some tasks, specific details need to be preserved to successfully accomplish the task. However, the authors claim that increasing the number of gisting tokens k does not help with performance. Furthermore, the choice of the number of gisting tokens is fixed for all tasks prior to training. Therefore, this method does not allow for dynamically trading off compression levels and model output quality.  It would be interesting to see evaluation results on tasks requiring differing levels of specificity.\n\nThe efficiency gains of Gisting is modest compared to instruction caching\n\nQUESTIONS:\nCould you please clarify “having too many gist tokens hurts performance in some cases … the increased capacity enables overfitting to the training distribution”. In the extreme case, having enough gist tokens to copy the longest prompt t, we would expect equal performance between gisting and using original task instructions t.\n\nCould you please clarify the tie-breaking process referenced in this sentence: “Due to the subjectivity of breaking ties in a forced-ranking task…”. This is fairly important given that this is a proposed explanation for the low agreement numbers. Based on the prompt given to ChatGPT, instructions given to human evaluators, and evaluation examples found at the end of the Appendix, it appears no tie-breaking has occurred.\n\nLIMITATIONS:\nGisting loses some nuance in the original instructions.\n\nThe compression level is not dynamic relative to the task.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents \"gisting\", which learns language models to compress instruction prompts into smaller sets of compressed context -- that includes special '<GIST>' tokens and the activation stacks above these tokens. Compressing instruction prompts allows saving context windows and saving compute for encoding instruction prompts. The task settings that this paper considers include:\n* [instruction prompt][input][output] --> [compressed instruction prompt][input][output]\n* [instruction prompt][output] --> [compressed instruction prompt][output]\nThe experimental results show that it is indeed possible to represent instruction prompts with much shorter compressed context while having only marginal performance loss. The authors evaluated the gisting model against uncompressed model on instruction following tasks comprehensively with ROUGE-L, preference of ChatGPT, and human evaluation.\n\nSTRENGTHS:\n* Novelty: compressing instruction prompts to save context windows and save compute for encoding instruction prompts is a novel and useful idea that can have good impact on real-world applications.\n* Very solid and well-designed experiments and evaluations: the authors present comprehensive evaluations by ROUGE-L (which is lexical), ChatGPT (which can be considered an automatic semantic evaluation), and human. Furthermore, when showing the human evaluation (Table 2), which is based on a subset of evaluation set, it is compared with ChatGPT results side-by-side and shows that outcomes are consistent and inter-annotator agreement level is similar.\n* Writing clarity: the paper is very easy to follow in most of the part. Figures, plots, and tables are well-polished.\n\nWEAKNESSES:\nThere are some descriptions in the paper such as **G(t) will be a set of soft gist tokens smaller than the number of tokens in t ...** (line 66-68), **Gisting compresses prompts into “gist tokens”** (Figure 1 caption), etc, suggesting that the gist **tokens** contains compressed information. However, if I understand the paper and code correctly, a gist token is really just a special additional token <GIST> in vocabulary and the gist token embedding does not change according to task t. What really contains the compressed information is those activations above the gist tokens. \n\nIn other words, gisting compresses prompts into gist \"activations\" instead of gist \"tokens\" (are inputs to the model). G(t) are activations, not \"soft gist tokens\". When I think of soft tokens, I would think of the type of soft tokens as in prompt tuning, and this seems clearly different. Please correct me if I'm wrong but I think these descriptions are incorrect and misleading and need to be fixed before the paper can be published.\n\nQUESTIONS:\nHow is compression factor (in figure 3) estimated? It doesn't seem to be explained in the paper.\n\nLIMITATIONS:\nLimitations are properly discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}