Learning to Compress Prompts with Gist Tokens
Authors: Jesse Mu
Xiang Lisa Li
Noah Goodman
Venue: NeurIPS 2023 poster (2023)
Keywords: language models
instruction finetuning
prompt compression
distillation
context distillation
prompting
soft prompting
efficiency
TL;DR: Train an LM to compress prompts up to 26x with "gist tokens", leading to compute, memory, and storage savings.

Abstract: Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

Learning to Compress Prompts with Gist Tokens
Jesse Mu, Xiang Lisa Li, Noah Goodman Stanford University
sttefossttaoodd.coom edu, (xlisali, ]
Abstract
Prompting is the primary way to utilize the multitask capabilities language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with additional cost standard instruction finetuning simply modifying Transformer attention masks to encourage prompt compression. On decoder (ILaMA-7BB) and encoder-decoder (FLAN-T5-XXL)) LMs, gisting enables to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.
Introduction
Consider the prompt of a Transformer 34 language model (LM) like ChatGPT
You ChatGPT, large language model trained by OpenAI. You answer concisely possible for each response (e.g. don't verbose). It very important that you answer as concisely as possible, so please remember this. If you are generating a list, do not have too many items. Keep the number of items short.
Knowledge cutoff: 2021-09 Current date: <TODAY>
With millions of queries day, an unoptimized ChatGPT would encode this prompt over and over with a self-attention mechanism whose time and memory complexity is quadratic the length of the input. Caching the Transformer activations of the prompt can prevent some recomputation, yet this strategy still incurs memory and storage costs as the number cached prompts grows. At large scales, even small reductions in prompt length could lead to substantial compute, memory, and storage savings over time, while also letting users more content into an LM's limited context window.
How might we reduce the cost of this prompt? One typical approach is to finetune distill (0 the model behave similarly the original model without the prompt, perhaps with parameter-efficient Yet a fundamental drawback of this approach is that it requires retraining the model for each new prompt (Figure] [ bottom left).
adaptation methods 15 16 19
Instead, we propose gisting (Figure | top right), which compresses arbitrary prompts into a smaller of Transformer activations on top virtual "gist" tokens, a la prefix-tuning (19). But where prefix- tuning requires learning prefixes via gradient descent for each task, gisting adopts a meta-learning approach, where we simply predict the gist prefixes zero-shot given only the prompt, allowing for generalization to unseen instructions without any additional training. Since gist tokens are much shorter than the full prompt, gisting allows arbitrary prompts to be compressed, cached, and reused for compute efficiency.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Prompting
Pudyy 1 x)
Gisting
Paly I G()) x)
math equation:
equation:
Summarize the article:
Enstare Summarize the articlo:
Translate this French:
prekit Translate this French:
FinetuningDDistilltiionn
Piuly1 +)
Figure I: Prompting retains the multitask capabilities of LMs, but is inefficient. Finetuning'distillaiion is more efficient, but requires training model for each task. Gisting compresses prompts into activations on top of "gist tokens", saving compute and generalizing to novel tasks at test time. Each vertical rectangle represents stack of Transformer activations.
Pi
Solve the math equation:
Pinty (x)
Summarize the article:
Translate this to Fronch:
The oat <SEPs Le chat
In this paper, we further propose very simple way learn a gist model: doing instruction tuning 38 with gist tokens inserted after the prompt, and modified attention mask preventing tokens after the gist tokens from attending to tokens before the gist tokens. This allows a model to learn prompt compression and instruction following at the same time, with no additional training cost.
On decoder-only (LLaMA-7B) and encoder-decoder ---XN-)--XXI) LMs, gisting achieves prompt compression rates of up to 26x, while maintaining output quality similar to the original models in human evaluations. This results in up 40% FLOPs reduction and 4.2% latency speedups during inference, with greatly decreased storage costs compared to traditional prompt caching approaches.
Gisting
We will first deseribe gisting in the context of instruction finetuning (38. We have instruction- following dataset D = (t,, Ti : y:) where t is a task encoded with a natural language prompt (e.g. Translate this to French), x: is an (optional) input for the task (e.g. The cat), and y is the desired output (e.g. Le chat). Given a (usually pretrained) LM, the aim of instruction finetuning is to learn a distribution Pry(y 1, typically by concatenating l and x, then having the LM autoregressively predict y. At inference time, we can prompi the model with a novel task / and input x, decoding from the model obtain its prediction.
However, this pattern of concatenating and æ has drawbacks: Transformer LMs have limited context windows, bounded either by architecture or memory limits. Furthermore, given that attention scales quadratically in the length of the input, long prompts f, especially those that are repeatedly reused, computationally inefficient. What options do we have reduce the cost of prompting?
One simple option is finetune the LM for specific task t. That is, given D' {2 y:) W the dataset containing input/output examples only under task t, we can learn a specialized LM P x) which is faster because it does not condition on t. Purameter-efficient finetuning methods such as preix-/promptt-unigg [18, [99 or adapters [15 16 promise to do so at fraction of the cost of full finetuning, and newer methods like HyperTuunig 25 eliminate gradient descent entirely, instead predicting the parameters of the specialized model directly from D'. Yet problems with these methods still remain: we must store at least subset of model weights for each task, and more importantly, for each task f, we must collect a corresponding dataset of input/output pairs D' to adapt the model.
Gisting is a different approach that amortizes both (1) the inference-time cost of prompting PLM with I and (2) the train-time cost of learning a new pi for each f. The idea is to learn a compressed version of t, G(t), such that inference from Pa (y G(t), 2) is faster than Py x, In LM terms, G(t)) will be the key/value activations on top a set of gist tokens, smaller than the number of tokens in t, yet still inducing similar behavior from the LM. Also known as Transformer prefix (19) G(t) can then be cached and reused for compute efficiency. Crucially, we expect G to generalize to unseen tasks: given a new task we can predict and use the gist activations G(()) without any additional training.
2.1
A Context Distillation Perspective
An alternative way to view gisting is through the lens of distillation of an already instruction-tuned LM (y 4, x). Askell et al. [1] and Snell et al. (30 define context distillation as the process of finetuning a new LM Poo to mimic the original LM without the prompt ("context") t, via the loss
Lev(Pen, t) = E,, Dau (PLM t, II PCD (v [))]]
(1)

---

Bidirectional Encoder Mask
Transiate
French:
<Gt>
<62>
The
cat
Autoregressive Decoder Mask
French:
Translate
The
cat
<G1>
Translate
<G2>
Fronch:
<G1>
cat
Decoder Cross-Attention Mask
Translate
French:
<G1>
<G2>
cat
chat
Standard Mask
Gist
(a) Decoder-only (e.g. GPT, LLaMA)
(b) Encoder-decodert (c.g. T5)
Figure Gist Masking Attention mask modifications for (a) decoder-only and (b) encoder-decoder Transformer LMs to encourage prompt compression into gist tokens <G1> <G2>, In these tables, cell (r, c) shows whether token can attend token during self- cross-attention.
The insight be gained from this perspective that we do not need any external data D: this KL objective can be approximated by finetuning Pen on synthetic sampled dataset D {(0, úi) where (d;, )t) Pra(( (). This is precisely the approach taken by recent work 7 30 including Wingate et al. [40], who notably learn to compress single discrete prompt into a soft prompt via gradient descent, similar to this paper.
However, we differ from this prior work in that we are not interested in distilling just single task, but in amortizing the cost of distillation across distribution of tasks T. That is, given task t ~ T, instead of obtaining the distilled model via gradient descent, use G simply predict the gist tokens (= parameters) of the distilled model, in the style of HyperNetworks [13] and HyperTuning Our "meta" distillation objective thus (with changes highlighted in blue):
25
(p T) EENT, a (prm(y (, || Pa-ly G(t),
In the experiments we describe below, we train on synthetic instruction-following data sampled from instruction-uue connti variants of GPT-3 23. Thus, these experiments indeed be form of context distillation for the GPT-3 series models.
3 Learning Gisting by Masking
Having just described the general framework of gisting, here we will explore an extremely simple way of learning such model: using the LM itself as the gist predictor G. This not only leverages the pre- existing knowledge in the LM, but also allows us to learn gisting by simply doing standard instruction finetuning while modifying the Transformer attention masks to enforce prompt compression. This means that gisting incurs no additional training cost on top of standard instruction finetuning!
Specifically, we add a single gist token g to the model vocabulary and embedding matrix. Then, given (task, input) pair (t, we concatenate and x with a set of copies of g between: (t. .1......................... -k. 2), e.g. Translate French: <G1> <G2> The cat The model is then restricted such that input tokens after the gist tokens cannot attend to any of the prompt tokens before the gist tokens (but they can attend to the gist tokens). This forces the model to compress the prompt information into the gist prefix, since the input i (and output y) cannot attend to the prompt 4.
Figure Jiirustratssstess the required changes. For decoder-only LMs such as GPT-3 K or LLaMA [ that normally admit an autoregressive, causal attention mask. we simply mask out the lower-left corner of the triangle (Figure 2 For encoder-decoder LMs (e.g. T5; (28)) with a bidirectional encoder followed by an autoregressive decoder, changes are needed (Figure 2p). First, in the encoder, which normally has no masking, we prevent the input from attending to the prompt t. But
Again, the gist token is the same from to ge: what changes the activations top of each token.

---

we must also prevent the prompt i and gist tokens 9i from attending to the input 2, since otherwise the encoder learns different representations depending on the input. Finally, the decoder operates as normal, except during cross-attention, we prevent the decoder from attending to the prompt I.
Overall, these masking changes are extremely simple and can be implemented in roughly 10 source lines of code, See Appendix for sample PyTorch implementation which can be used as drop-in replacement for attention masking in deep learning libraries such Hugging Face Transformers (4]
4 Experiments
4.1 Data
A dataset with a large variety of tasks (prompts) is crucial to learn gist models that can generalize. To obtain the largest possible set of tasks for instruction finetuning, we create dataset called Alpaca+, which combines the Self-Instruct [36] and Stanford Alpaca (31] instruction tuning datasets, each consisting of (t, z, y) tuples sampled from OpenAl's text-davinci-
and text-davinci-00s variants of GPT-3, respectively. In total, Alpaca+ has 130,321 examples, with 104,664 unique tasks t, 48,530 unique inputs x, and anywhere from 0-5 inputs per task (0.64 on average).
Note that -59% of tasks in Alpaca+ have no inputs (e.g. Write me poem about frogs), in which case we simply omit the input z. While is less interesting to cache such prompts since they are not input-dependent, they still serve as valuable training signal for learning prompt compression. Overall, while Alpaca+ noisy and imperfect, Wang al. (36] and Taori et al. [31] nevertheless show that models trained on such data achieve comparable performance to the original models from which the data sampled, making this promising testhed for studying gisting.
From Alpaca+ we hold out 3 validation splits: 1000 Seen prompts (with unseen, non-empty inputs); 1000 Unseen prompts (with non-empty inputs); and the 252 hand-written Human prompts and completions used in Wang et al. [36). of which 83% have non-empty inputs. The latter two splits test generalization to unseen instructions, with the Human split posing a stronger -fitt--iistriuttin (OOD) challenge: the average training prompt has -20 tokens, compared to -26 in the human split.
4.2 Models
To demonstrate gisting across multiple Transformer LM architectures, we experiment with LLaMA- 7B (33), decoder-only GPT-style model with -BB parameters, and FLAN-TS-XXL [ an encoder- decoder T5 model (28) with 1IB parameters. For each of these models, we train models with varying number of gist tokens k E 21,2,5, 5, 10), using the modified attention masks described in Section 3 To assess how the model is learning prompt compression, calibrate performance against upper- and lower-bound baselines and a simple discrete compression strategy:
Positive Control. As an upper bound on performance, we train a model with a single gist token, but without any modifications the attention mask. This is akin to doing standard instruction finetuning.
Negative Control. As a lower bound on performance, we train a model without access to the task t. This is similar to "random gist token" baseline, which allows to measure how the model would do it failed to compress any information into the gist prefix.
Discrete Compression with TF-IDE. An alternative approach to compression is simply using fewer discrete tokens to express the same task. Achieving compression rates similar to gisting, however, requires compression far beyond any threshold of fluency. Nevertheless, as a baseline, we compute TF-IDF statistics over the set of instructions the Alpaca+ training set to extract the most relevant keyword in each instruction. Some examples from the training set include (see Appendix G for more);
Write a letter to your boss asking for an increase in salary Given two integers, find their average average
> salary
We then replace each instruction in Alpaca+ with the first subword token from each keyword, resulting in compression rates equivalent to model trained with single gist token. Similarly the positive control, we do standard instruction finetuning over Alpaca+ with these compressed instructions.
For full training, data, and compute details, and a link to code, Appendix B

---

4.3 Evaluation
Our evaluation uses combination of automated metries and AI- and duman-assiste evaluation:
We first use ROUGE-L, simple lexical overlap statistic 220), used in previous open- ended instruction finetuning work [37 38 The text-davinei- 0033 completions are used as references, except for the Human split, where we use the gold-standard human reference.
ROUGE-L.
ChatGPT. Next, we use ChatGPT-3.5 22 to compare the outputs of our models to the positive control. While this is imperfect metric, it allows for much faster and cheaper evaluation than human experiments, with an arguably more meaningful semantic signal than ROUGE-L. Recent work
has found that ChatGPT can be used for text annotation and evaluation 12 17 35 with near-human performance, and similar model-based evaluations have been conducted with recent LMs [5] 31
Specifically, given a task f, input x, and outputs from two models (y1 M2) identified only as Assistants and B, ChatGPT was asked to choose which assistant response is better, explaining its reasoning in Chain-of-Thought fashion [39]. If the models produced the same output, or were equally bad, ChatGPT was allowed to call tie. We gave examples of desired outputs in ChatGPT's prompt, and randomized the order of presentation between the models for each query to avoid order effects. The full prompt given to ChatGPT and evaluation details are in Appendix C Using these outputs, we measure the win rate of a model against the positive control: win rate of 50% indicates that the model is of comparable quality to a model that does no prompt compression.
Human eval. Finally, after prototyping with ChatGPT, select the best gist compression models and do a Human evaluation on a random subset of 100 of the 252 examples in the Human validation split. For each of the 100 examples, from Prolific, and asked them to rate model outputs in the same style as the ChatGPT evaluation above (see Appendix D for full details, including the annotation interface). The only difference is that human participants were allowed to select Don't Know" " in cases where they had inadequate domain knowledge to accurately judge the responses, e.g. if the question was coding question; we drop these responses (-10%) during analysís. With this human evaluation, we are not only interested in evaluating our final models, but also validating whether ChatGPT can be used reliable replacement for human annotation on this task.
recruited 3 US UK-based English-fluem annotators
5 Results
ROUGE-L and ChatGPT evaluations for LLaMA-7B and FLAN-TS-XXL, with varying numbers of gist tokens, are shown in Figure Models were generally insensitive to the number of gist tokens A: compressing prompts into a single token prefix did not substantially underperform larger prefixes. In fact, having too many gist tokens hurts performance in some cases (c.g. LLaMA-7B, 10 gist tokens), perhaps because the increased capacity enables overfitting to the training distribution. Thus, we use the single gist token models for the rest of the experiments in the paper, and report the exact numbers for the single-token models, with the positive, negative, and TF-IDF baselines, in Table
On Seen instructions, gist models attain rear-identical ROUGE and ChatGPT performance as their positive control models (48.6% and 50.8% win rates for LLaMA-7B and FLAN-TS-XXL, respectively). But we are most interested in generalization unseen tasks, as measured by the other two splits. Unseen prompts within the Alpaca+ distribution, we again see competitive performance: 49.7% (LLaMA) and 46.2% (FLAN-T5) win rates against the positive controls. It is on the most challenging OOD Human split where we see slight drops in win rate to 45.8% (LLaMA) and 42.5% (FLAN-T5), though these numbers are still quite competitive with the positive control. Finally, gist compression is vastly superior to discrete compression; the TF-IDF models in Table [ only marginally outperform the negative control models actoss the board.
Table shows the human evaluation results on the Human validation split, comparing the single gist token models to the positive control. Overall, human annotators agree with ChatGPT, with average win rates of 52.3% (vs. 48.0%) for LLaMA-7B and 40.6% (vs. 42.0%) for XXXL- Importantly, this agreement persists at the level of individual responses. The average pairwise Cohen's among human annotators is .24 for ILLMA-7B and .33 for FLAN-TS-XXI. Because humans will often arbitrarily choose one response over another even for samples of equal quality, these numbers
5

---

Table 1: Results for single gist tokens. ROUGE-L and ChatGPT scores for Gist, TF-IDF, and positive/negative controls, Parentheses are scores normalized between positive/negative controls.
Seen
Unseen
Human
Model
ROUGE-L
ChatGPT %
ROUGE-L
ChatGPT
ROUGE-L
ChatGPT %
LLaMA-
Pos
58.0 (100)
50.0 (100)
48.1 (100)
50.0 (100)
27.0 (100)
50.0 (100)
7B
Gist
57.8 (99.2)
48.6 (92.4)
46.6 (91.0)
49.7 (98.8)
23.9 (75.4)
45.8 (84.9)
TE-IDF
38.1 (24.5)
34.5 (16.2)
(15.6)
29.3 (15.9)
16.5 (16.7)
24.6 (8.6)
Neg
31,5 (0)
31,5 (0)
31.4 (0)
25.4 (0)
(0)
22.2 (0)
FLAN- T5-XXL
Pos
50.6
(100)
45.7 (100)
50.0 (100)
23.9 (100)
50.0 (100)
Gist
48.9 (93.2)
50.8 (103.9)
43.8 (88.6)
46.2 (84.4)
21.7 (80.9)
42.5 (63.2)
TF-IDF
32.0 (25.9)
35.9 (30.5)
34.3
31.0 (22.1)
(3.6)
28.4 (-5.9)
Neg
20)
29.7 (0)
29.1
25.6
12)4
29.6 (0)
Seen
Unseen
Human
Compression
Compression factor
Compression factor 5x
20x20x
4x
2x
16x 8x
3x
2x
26x13x
3x
ROUGE-L
45
24
40
20
35
20x10x
4x
2x
16x 8x
3x
26x13x
5%
3x
ChatGPT win rate
50%
50%
50%
40%
40%
40%
35%
35%
35%
25%
25%
25%
20%
20%
20%
10
1 2
10
10
gist tokens
# gist tokens
gist tokens
(a) LLaMA-7B
Seen
Unseen
Human
Compression factor
Compression factor
Compression factor
20x10x
4x
2x
15x 8x
3x
2x
26x13x
5x
3x
50
ROUGE-L
45
40
35
30
1 2
10
2
20x10x
4x
2x
15x 8x
3x
26x13x
5%
3x
ChatGPT win rate
50%
50%
50%
40%
40%
40%
35%
35%
35%
25%
25%
25%
20%
20%
20%
10
2
10
10
gist tokens
# gist tokens
gist tokens
(b) FLAN-TS-XXL
Figure 3: Varying the number of gist tokens. ROUGE-L L and ChatGPT scores for (a) LLaMA-7B and (b)
FLAN-T5-XXL. for different gist tokens. Dashed lines indicate positive and negative control performance.
Erro bars 95% exact binomial confidence intervals, splitting ties equally between models [10] and rounding down in favor of the positive control in of odd number of ties. Compression factors calculated by computing the average token length of validation split and dividing by number gist tokens.

---

Table 2: Human evaluation results. Win and inter-annotator agreement of single token gist models over positive control according to Human annotators (HI-H[), their average, and ChatGPT (95% confidence intervals in parentheses), for 100 out of 252 examples in the Human validation split.
Gist Win % over Pos Human (HI-H13
Agreement (Cohen's x)
Model
H1
H2
H3
ChatGPT
Human
ChatGPT
LLaMA-7B
51.1
44.5
59.8
52.3 (46.1, 1, 58.4)
48.0 (38.0, 58.2)
.24
.29
FLAN-T5-XXL.
43.0
41.9
37.2
40.6 (34.6, 46.8)
42.0 (32.2, 52.3)
.33
.29
fairly low; however, ChatGPT shows similar levels of agreement, with average across each of the 3 human annotators at .29 for both models. These results, paired with the similar overall win rates, show that using ChatGPT is similar to simply recruiting an additional human annotator, and corroborates the broader results in Figure 3 See Appendix for more human evaluation results, including a breakdown of agreement across annotators, and Appendix for examples of instructions. model outputs, and human/ChatGPTT judgments in the Human validation split.
Since aim is having the gist models mimic the original models, one might ask how often the gist model is identical to the positive control. Figure A.3 in Appendix D shows how often this happens: for Seen tasks (but unseen inputs), the gist model outputs exactly match the positive control nearly 50% of the time. This drops to -00-2%% for Unseen tasks and - -00% for the OOD Human tasks.
Overall, results show that gist models reliably compress prompts, that lie outside the training distribution, especially for decoder-only LMs (LLaMA). Encoder-decodee LMs like FLAN-T5 show slightly worse OOD performance; inhibits the bidirectional attention flow in the encoder, which is more challenging to adjust to than simply limiting the history of decoder L.M. Future work will be needed to investigate this hypothesis.
to some degree those
possible reason is that gist masking
5.1 Failure cases
While the responses from the gist models are often indistinguishable from the positive control, we nevertheless do see failure cases where the gist token models do not perfectly compress the prompt. We give two examples where this oceurs in the Human validation split: first, when the instruction contains specific details, e.g. phrases that must be copied verbatim into the output (in the snippets below, the names of Assistants A/B are hidden to ChatGPT):
Give the news title a category. Pick category from the list News Buzz, Travel, Style, Arts & Culture, Politics, Tech, and Science & Health. <G1> Input: The #Banksy Exhibit in Canbridge, MA is absolutely terrific. Assistant A (FLAN-TS-XXL gist): Culture
Instruction:
Assistant (pos control): Arts Culture
ChatGPT choice: B
ChatGPT reason: The news title is about an art exhibit, so Assistant B's response of 'Arts & Culture' is accurate and specific than Assistant A's 'Culture'.
We also
cases of runaway generations that do not occur with the positive control model:
Instruction:
Provide example of how table of contents be generated automatically in LATEX document. Make sure your example contains necessary commands. <G1> Assistant A (pos control): Ntableofcontents \chapter(Intrrouuttion) \chapter(Theory) Achapter (Experiments) \chapter(Conclusion) Assistant В (LLaMA-7B gist): \tableofcontents \chapter(Introduction)) Esection[The Basics)
\subsection (The Basics] (subsubsection(The Basics) \paragraph(The Basics) \subparagraph (The Basics) [repeats for hundreds of takens] Nsubparagraph(The Basics)
ChatGPT choice: A
ChatGPT reason: Assistant A provides simple and correct example of generating a table of contents in LaTeX. Assistant B's example is unnecessarily long and does not follow the standard structure of a table of contents.
While it unclear why only the gist models exhibit this behavior, these issues can likely be mitigated with more careful sampling techniques.
7

---

Table 3: Gist efficiency improvements. For different caching strategies (None, Instruction, Gist), we record CUDA wall time and GFI.OPs (+ std dev). Then we report the absolute/relative improvement of Gist Caching over these alternative strategies, with 95% confidence intervals in parentheses.
Caching Strategy Instruction'
Absolute/Relative A
Model
Metric
None
Gist
vs. None
vs. Instruction
LLaMA-
CUDA time (ms) 4
23.4 6.88
22.1 6.58
21.8 6.55
1.60 (1.29, 1.90)
.221 (.140, .302)
7B
6.8% (5.5, 8.1)
1.0% (.63, 1.4)
GFLOPs +
915 = 936
553 + 900
552 + 899
362 (337,387)
.607 (.48,, .766)
40% (37, 42)
.11% (.08, .14)
FLAN- T5-XXL
CUDA time (ms)
31.0 -1: 5.31
N/A
29.7 -1: 5.07
1.30 (1.10, 1.51) 4.2% (3.6,
N/A
GFLOPs +
716 + 717
N/A
427 + 684
289 (268, 310)
N/A
40% (37, 43)
'Average KV Cache Length 26
" "Average KV Cache Length
Compute, Memory, and Storage Efficiency
Finally, we return to one of the central motivations of this paper: what kind of efficiency gains does gisting enable? To answer this question, we compare the compute requirements (CUDA wall time and FLOPs) during inference with the single-token gist models using different caching strategies:
1. No caching: just encoding the full prompt I.
2. Instruction caching: caching the activations of the uncompressed instruction (keys and values for all layers) into what is called the KV cache. This is the most common caching behavior for Transformer inference [ 26 and is supported in libraries like Hugging Face Transformers [T. However, it is only applicable to decoder-only models, since in models with bidirectional encoders like T5, the instruction representations t depend on the input x.
3. Gist caching:
Compressing the prompt into the gist prefix G(I).
Table 3 displays the results of profiling single forward pass through the model (i.e. one step of decoding with a single input token) with PyTorch (4 2.0, averaged across the 252 Human instructions. Gist caching improves significantly over unoptimized models, with 40% FL.OPs savings and 4-7% lower wall time for both models. Note that at these (relatively) small scales, the wall time improvements are smaller than the FLOPs reductions because much of the inference latency is caused by moving tensors from high-bandwidth memory (HBM) to the chip compute cores, i.c. what Pope et al. [66] call the "memory time". Larger sequence lengths and batch sizes will lead to additional speedups, as the overall latency becomes dominated by the actual matrix computations.
For LLaMA-7B, the picture is more nuanced when compared to caching the full instruction. Compute improvements of gist caching are smaller: a negligible decrease in FL.PPs (0. 1 11%) and a modest 1 5% speedup in wall time. This is because the FL.PPs required for Transformer forward pass is dominated by processing of the new input tokens, rather than self self-attention with the KV cache. For example, forward pass through LLaMA-7B with single input token and 2000-length KV cache is only -10% more expensive than the same forward pass with no KV cache-see see Appendix F for more details. Nevertheless, this small decrease in FLOPs leads to disproportionate decrease in wall time (1%), likely because the self-attention computations are slower relative to their FLOPs contribution.
At large scales and with heavily reused prompts, a 1% latency speedup can still accumulate into significant cost savings over time. More importantly. however, there are key benefits of gist caching over instruction caching besides latency: compressing 26 tokens into gives more space in the input context window, which bounded by absolute position embeddings or GPU VRAM. For example, for I.LaMA-B,, each token in the KV cache requires 1.05 MB storage While the total contribution of the KV cache relative to the memory needed for L.LMA-7B inference is negligible the prompt lengths we tested, an increasingly common scenario is developers caching many prompts across large number of users, where storage costs quickly add up. In these scenarios, gisting allows caching of up to 26x more prompts than full instruction caching, using the same amount of storage!
24 (1p32 bytes) 2 (keys+values) 32 (num layers) 32 (num attn heads) x 128 (head dim) 1.05 MB.
8

---

7 Additional Related Work
Gisting builds upon past work in (parameter-efficieni instruction finetuning and context distillation, as discussed in Section Here we will outline some additional connections to related work:
Adapting LMs without Backprop. As mentioned in Section 2 gisting can be viewed as a way to adapt LMs without gradient descent, by predicting the the prefix ( parameters) of the adapted model. Similarly, HyperTuning (25] predicts the prefix of a model for a task using (input, output) pairs for that task. If HyperTuning is a "few-shot" adaptation method, predicting the prefix from few-shot examples, then gisting can be seen as a "zero-shot" version, predicting the prefix from the language instruction alone. Gisting also has the additional benefit over HyperTuning of being conceptually simpler: instead of training a separate LM to predict the prefix, the LM itself is used as the HyperNetwork 13], with only a tiny change the attention masks needed for prompt compression.
Compression and memory in transformers. The idea of "compressing" prompts is closely related to previous attempts at storing past representations to improve memory and long-range sequence In particular, the Compressive Transformer [27] compresses transformer activations into a smaller compressed memory using a learned convolutional operator. Gisting can be seen as a variant of the Compressive Transformer with 3 key differences. First, the compression function is not a separately learned function, but the LM's own self-attention mechanism, controlled by an input-dependent gist token. Second, the compression function is learned jointly with instruction finetuning via the standard language modeling loss, not a specialized auxiliary reconstruction loss as in (27.. Finally, our task of interest is not long-range sequence modeling, but caching and reusing instruction following prompts for efficiency reasons.
modeling in Transformers 9 21 27
42 4
Sparse attention mechanisms. By restricting attention masks, gisting draws inspiration from efficient/sparse attention methods in Transformers (see 32 for review). For example, some sliding window attention mechanisms (2 may remove the need to keep the entire KV cache around during decoding, but these more general methods are not optimized for caching arbitrary parts of the input sequence of varying length, which prompt compression demands. In light of this, gisting can be viewed an input-dependent sparse attention mechanism specifically aimed at improving efficiency of the prompting workflow now commonly used in LMs.
Discussion and Limitations
In this paper we presented gisting, framework for prompt compression in LMs, and a simple way of implementing gist models by modifying Transformer attention masks that incurs no additional cost over standard instruction finetuning. Gisting can be seen either as a modified form of instruction finetuning or a method for (meta- ccontext distillation of an LMM. Gist models can compress unseen OOD prompts up to 26x while maintaining output quality, resulting in up 40% FLOPs reduction and 4.2% wall clock speedups over unoptimized models, and enabling new methods for prompt caching in encoder-decoder models. While wall-time improvements for decoder-only LMs are smaller, gisting nevertheless enables caching 1 order of magnitude (26x) more prompts relative to full instructions.
Gisting is a promising method for improving LM efficiency, but carries some limitations. While gisting seems to succeed in capturing the "gist" of instructions (hence the name), achieving such com- pression necessarily results in some loss of nuance of the original instruction; Secton 5.1 illustrattes a concrete failure case we observed. Since the behavior of LMs on edge cases is already not well under- stood, it is especially important for practitioners to carefully evaluate whether the compute/accuracy tradcoff of gisting is sufficiently safe and robust for their use case, before deployment.
Nevertheless, we believe gisting opens several interesting directions for future work. First, the masking method presented here can be easily integrated into existing instruction finetuning workflows, but another exciting approach is to retrofit an existing, frozen LM by training smaller model to compress prompts, if finetuning the larger LM inconvenient. Second, the largest efficiency gains from gisting will result from compressing very long prompts, for example k--soot prompts for large k that may even exceed a single context window. Finally, compression performance can likely be improved through "gist pretraining" first learning to compress arbitrary spans of natural language, before then learning prompt compression. Such objectives could be devised by inserting gist tokens into other pretraining objectives, perhaps during language modeling or T5's span corruption objective.

---

Acknowledgments and Disclosure of Funding
We thank the Stanford Alpaca team, especially Xuechen Li, with assistance with Alpaca data and finetuning, and Gabriel Poesia for the codebase used to collect human evaluations. Additional thanks to Xiuyu for help debugging and ensuring reproducibility of the open-source codebase. JM supported by the Open Philanthropy AI Fellowship and XLL is supported by the Stanford Graduate Fellowship.
References
[1] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, al. general language assistant as laboratory for alignment. arXiv preprint 12.00221 2021.
[2] Beltagy, M. E. Peters, and Cohan. Longformer: The long-document transformer. arXiv:2004.055 22220044.5550 150, 2020.
[3] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
[4] C. Chen.
Transformer inference arithmetic.
https://kipp.ly/blog
transformer inw.rnnnneeennnee ari arithhetic/////m 2022.
[5] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuangg. J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality, https://vicunn: 1msys.. org/ 2023.
[6] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 2019.
[7] E. Choi, Y. Jo, J. Jang, and M. Seo. Prompt injection: Parameterization of fixed inputs, arXiv preprint arXiv:2206. 1149,, 2022.
[8] H. W. Chung. L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.11416, 2022.
[9] Z. Dai, Z. Yang. Y. Yang. J. G. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL. Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2177-2988, 2019.
[10] W. J. Dixon and F. J. Massey Jr. Introduction statistical analysis. McGraw-Hii Hill, 1951.
[11] X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URLLhhtts://bbbr.bbeke.. eetu/blog/222//04/03//aaa
[12] Gilardi, M. Alizadeh, and M. Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303. 15056, 2023.
[13] D. Ha, A. Dai, and Q V. Le. HyperNetworks. In International Conference Learning Representations, 2017.
[14] Hoffmann, Borgeaud, Mensch, E. Buchatskaya, T. Cai, Rutherford, D. L. Casas,
L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimas large language models. arXiv preprint arXiv:2203. 15556, 2022.
[15] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efficien transfer learning for NLP. International Conference on Machine Learning, pages 2790-2799, PMLR, 2019.
10

---

[16] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[17] Huang, H. Kwak, and An. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. arXiv preprint arXiv:2302.07736, 2023.
[18] B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[19] X. L. Li and P. Liang. trefi-tuuiin: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4582 4597, 2021.
[20] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[21] Liu, Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and Shazeer. Generat- ing Wikipedia by summarizing long sequences. In International Conference Learning Representations, 2018.
[22] OpenAL. Introducing ChatGPT.
2022.
[23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730-27744, 27744, 2022.
[24] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[25] J. Phang, Y. Mao, P. He, and W. Chen. HyperTuning: Toward adapting large language models without back-propagattin. arXiv preprint 1. 2022.
[26] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, Bradbury, A. Levskaya, J. Heek, K. Xiao,
S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022.
[27] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020.
(28) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified Text-to-Text Transformer. The Journal of Machine Learning Research, 248555555,, 2020.
[29] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
[30] C. Snell, D. Klein, and R. Zhong. Learning by distilling context. 22X.::2009.1188,, 2022.
arXiv preprint
[31] R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. httpp:://githhiie wwm/coooo///////eeee ttt//attssuulaa lab/ Stanford_alpacee 2023.
[32] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: survey. ACM Computing Surveys, 55(6):1-28, 2022.

---

[33] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[35] J. Wang, Y. Liang, F. Meng. H. Shi, Z. J. Xu, J. Qu, and Zhou. Is ChatGPT good NLG evaluator? A preliminary study. arXiv preprint arXiv:2303.0444 04048, 2023.
[36] Y. Wang. Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[37] Y. Wang. S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, al. Super-Nauurlllnstuuttoon Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 5085-5109, 2022.
[38] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models zero-shot learners. In International Conference Learning Representations, 2022.
[39] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. In Advances Neural Information Processing Systems, 2022.
[40] D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621-56344 Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL httpsst/c//aaaaatoooooo org/ 2022. wwwdinnnsnnnnl.....m 412
[41] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the- art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. doi: 10. http://///00emmmmdmmmmm..... URL https://aclanthologg wwwpthhooog..ooo/22000 2020. wttp:///.2....mmmdeemm..ss
[42] Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022.
[43] H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv, N. Duan, and W. Chen. Poolingformer: Long document modeling with pooling attention. In International Conference on Machine Learning, pages 12437-12446. PMLR, 2021.
12

---

A Example PyTorch Implementation of Gist Masking
See Listing A.1 for sample annotated implementation of gist masking. This Py Torch implementation relies on basic NumPy-style tensor operations and can thus be adapted easily to framework like JAX.
B Data, Training, Evaluation, and Compute Details
Code, data, and model checkpoints are available http:://gtttttu..momm///eee//m/t
Data. For LLaMA-7B, we used maximum sequence length of 512 tokens during training and evaluation, except with the Human validation split, where the maximum length was increased to 768 (the Human instructions are longer). Examples longer than this length are truncated from the end. For FLAN-T5-XXLL we set a maximum input length (task + input 2)) of 128 and a maximum output length of 256, except again for the Human split, where the maximum input and output lengths were both set to 384. For both models, we set maximum generation length of 512 tokens. These lengths were chosen such that 1 % of examples across the board were truncated during training and evaluation for both models.
Training. Full hyperparameters for training runs are located in Table A. These parameters were adapted from previous published work finetuning LLAMA/FLAN-TS For LLaMA-7B, parameters are identical those used training Alpaca Taori et al. [3I] For FLAN-T5-XXIL parameters are - TK-INSTRUCT GitHub repository rather than the le-5 learning rate in the paper.
identical to those used in training INSTUCTT [37], except with 5e-5 learning rate, used in the
LLaMA-7B was trained for 3000 steps, while FLAN-T5-XXI. was trained for 16000 steps. Since there are about 130k examples Alpacat, given the batch sizes in Table T.. th.Th corresponds to about ~3 epochs and -2 epochs of training, respsectively. These numbers, again, are identical to Taori al. [31] and Wang al. [36]. We note that the training time relatively flexible; for example, we did not see substantial gains training beyond epoch for FLAN-T5-XXL.
During evaluation and benchmarking, we simply greedily decoded the most likely sequence. We saw limited gains from beam search with beam size B 4.
Evaluation.
NVIDIA GPUs, 480GB RAM, and 16 CPUs, using PyTorch 2.0 24), Hugging Face Transformers 44 and DeepSpeed Training runs take about ~7 hours to complete for LLaMA-7B and ~25 hours for FLAN-T5-
Compute. Experiments were run cluster machine with 4xA
29
XXL. Benchmarking results were obtained on the same machine, but using just 1 of the A100
GPUs.
Table A.I: Hyperparameters for training runs.
LLaMA-7B
FLAN-T5-XXI.
num steps
3000
16000
num train epochs
2
R: 2
batch size
128
16
learning rate
2c-5
5c-5
warmup ratio
0.03
0
precision
bfl6
bf16
optimizer
AdamW
AdamW
Deepspeed
GPUs (A100 80GB)
4
ZeRO stage subgroup size max live params
3
3
le9
le9
le9
1e9
distance
le9
1e9
13

---

inport torch
def ewverse_cunsunxx Tensor)
Tensor;
"""Cumulative sun from right to left.
See https::/gtthub.ccmm// http:/twww..oooooooooorrrrse/sss
return Corch.sun(s dime-1, kerpdins-True) torch. cunsun(x, dim=-1)
10
12
def nake_nask_pras wirstttisstttttete.......sst.stt girst_gsst torch. Tensor, gist_token; int, dtypentorch, int64) +> torch.Tensor: Tensor: "R"Reuunss a mask where all tokens prior to the first gist token are maskee out.
Args:
Tensor of input tokens where the last dinension is the sequence length. gist_token: the integer 1d of the gist token. dtype: dtype nask, default int64.
17
19
Returns:
The requested mask.
return ((irputs = gist_token) (ont_ttoen).cusuu(-1) 1) >e 1) (dtype)
24
Tensor, gist_ token: int, dtypentorch. int64) Tensor: "Returs mask where all takens after the last gist token are nasked out.
nake_nask_aee post t_ llstt_ss_tss Lastttgisttt_ppppeel (ist(ittts::
Computes the
meak__rr_.irst_iist.ce
but reverses the sequence before and after the
Args:
31
inputs: a Tensor of input tokens where the last dinension is the sequence length. gist_ taken; integer of gist token. dtype: the dtype of the mask, default int84.
Returns:
The requested mask.
cutur gist_token) 1) (ypeedyy
38
def nake_gist_ (istiuutsss torch. Tensor, tist_token:: int, pad_token: int, dtype=torch. int84) torch. Tensor: "Crrrettes a gist mask from supplied inputs and gist/pad tokens.
43
Tokens after the last gist cannot attend to tokens prior to the first gist. Additionally, tokens *before* #after*
gist cannot
gist.
gist is broadcasted to (with singleton (where din 1 is the head dimension).
corpatibility with multi-headed
Args:
50
inputs: a Tensor of shape (batch_size, seq_len) input tokens.
gist_token: the integer of the gist token.
pad_ tok_et the intager id of the pad token. inputs == pad, taken are nasked out.
dtype: dtype
nask, default
Returns:
The requested mask of shape (batch_size, 1, seq,len, seq_len)
a*e
57
Attention mask for tokens before the last gist token. pre gist_missst
mask_nnkssppss taakk_oot__ast__iitte gisp_gss gist_token, dtypentorch. bool) None, None] Attention mask for takens after the last gist taken.
post_gist_nast
gist_token, dtype=torch, bool) None, None] Construct time nasks by pernuting to time dinension.
wask.pre_.iirs..iistte
pre_eist_tine time__s_tte_eekk pre_gist_nask wwt.kk.eeeeeeCCCCC 1, 3, 2))
64 mask torch, where(pre_gist tine tine_nask mask, pre_gist_nask post_gist_ nask) (inputs I= pad mad_token)(:; [;, None, None] out pod
return nask. typeed)
Listing A.l: Sample PyTorch implementation of gist masking.
14

---

C Details of ChatGPT Evaluation
Given user instruction and two AI assistant responses. Jo0 is select the response that is nore helpful, and give reasons why. Judge responses holistically, paying special attention to whether response correctly follows provided instruction responses are equal, you may call tie, but please use this response sparingly.
is factually
If
Example
Instruction: Given the of city, tell the name of the state it is in. Input: Los Angeles
Assistant California
Assistant B: Wyoning .**
Your output should be:
("reason"; *Los Angeles is in California, Only Assistant A is correct.", "choice": "A")
Example
Instruction: Give ne some examples of fun things to do with the kids on a weekend. Assistant For example, what should I with kids weekend? What if I'm in city? What if I'm on the beach?
Assistant
try going hoard games or bake cookies.
or museun. you stay hone, you play
Your output should be:
("reason": "Assistant A doesn't seen to be fol following the instruction, Assistant 8 provides helpful information with examples." "choice":
Example
Write python function that prints "Hello, World". Assistant A: def hello_world(); print("Hello!") "Hello, World"
Instruction:
Your output should be:
("reason": "Assistant just 'Hello, World', Assistant actually Python function, if it doesn't exactly print the right thing, so overall Assistant is better.". ", "choice": "A")
Example
Instruction:
Translate the fol lowing sentence fron English to French.
Input:
like
Assistant A: Me gustan los gatos.
Assistant S
output should
("reason": assistants got language wrong.", "choice": "tie")
Your response should only be in the ISON format above: THERE SHOULD BE OTHER CONTENT INCLUDED IN YOUR RESPONSE, write the "reason" key before writing the "choice" key, so that you think step-by-step before making your
YOUR REASONING BRIEF.
Listing A.2: Full prompt given ChatGPT for evaluation. This prompt populates system field in ChatGPT API; the actual example to be evaluated is formatted like the examples the prompt above, then given the sole input in the user field.
We used the ChatGPT API, specifically the chatgpt-3. t-turbbo engine, to ChatGPT evalua- tion experiments over a period of weeks between March 27 and April 7, 2023.
The full prompt given to ChatGPT is located in Listing A.2 and contains 4 examples of desired output from ChatGPT, including preferring factually accurate responses (Example 1), preferring responses that follow the instruction, even if imperfect (Examples 2 and 3), and examples of models
15

---

Table A.2: Pairwise Cohen's between human annotators (HI, H2, H3) and ChatGPT.
(a) LLaMA-7B
(b) FLAN-T5-XXL
H1
H2
H3
Average
H1
H2
H3
Average
I11
.21
.33
.27
I11
.33
.34
.34
H12
.11
.19
-20
112
.33
.33
.33
.33
.26
H3
.34
.33
ChatGPT
.38
.22
26
.99
ChatGPT .35
.18
.34
.29
being equally wrong (Examples 4). For the models under comparison, randomized the order of presentation of each model as either Assistant A or Assistant B, to avoid order effects.
ChatGPT was instructed to only respond in JSON format, outputting first a reason key followed by choice key, to encourage chain-of-thought reasoning [39]. On rare occasions (< 0.25% the time), ChatGPT would output response that did not conform to the requested JSON format (e.g. it would just give an unstructured paragraph). In these cases we manually went through and converted these responses to JSON, without altering ChatGPT's reasoning.
In total, we collected -22.5k judgments from ChatGPT an estimated cost $29.28. The full outputs for each model across the Alpaca+ validation splits, as well as ChatGPT's responses and choices, are available in the code link above.
D Additional Human Evaluation Details and Results
D.1 Experimental Details
For each of the 100 examples randomly selected from the Human validation split, we recruited US or UK-based, English-fluen annotators from Prolific, an online crowdsourcing platform. Experiments were IRB approved under a generic human experiments IRB given to the authors.
The annotation interface given to Prolific crowdworkers is located in Figure A.1 To verify task comprehension, participants were shown two simple examples before the main body of the task (Figure A.2), and were required to answer correctly before proceeding. We compensated participants USD 44.35/hour for an estimated cost (including Prolific fees) of USD $141.64.
D.2 Additional Results
See Table A.2 for a breakdown of Cohen's 's & between human annotators and ChatGPT. We used a weighted version of Cohen's 's with linear weights, since the response scale is ordinal (e.g. "tie" is closer judgment to "pos control win" than "gist win").
E
Exact Match Results
See Figure A.3 for plot of exact match rates for the gist and positive control models (as measured by exact string match).
Additional FLOPs details
The FLOPP required for Transformer formward pass with varying KV cache lengths can be estimated by modifying existing equations to account for self-attention back to the KV cache. As an example, we modify the FLOPs equations used for computing FLOPs in the Chinchilla paper (Appendix F in [4].. Let seq len_weilll with past seq_len + se__acce__eee Then the modified Transformer FLOPs equations are:
Embeddings
seq_len len x vocab size d model
16

---

Judging Al Assistants
Progress: 12/50
User Instruction: Write good Twitter Bio. Try cover all the provided main points in your generated bio.
User Input: Name: Jane Main points: Architect Interested Tech Written in first person Use
Jane is architect who is interested in tech. She loves to solve problems and create beautiful things.
Assistant A:
Assistant I'm Jane, an architect who loves explore latest tech trends. I'm always looking for ways make lives easier and more convenient, Follow me for the latest tech news, tips, and inspiration!
Which Assistant more helpful?
TIE
DON'T KNOW
Remember:
When making your judgment, please consider whether the response attempts to follow the provided instruction, and factually accurate.
you can't decide, e.g. because responses are the same or equally good bad, may call but please this response sparingly.
Finally, you do not understand the question being asked: you may still able make
an educated d guess about which Assistant is better, for example one assistant is clearly
not following the directions. If so, please do! Otherwise, click the don't know" button. There is no need look up additional information
spend huge amount of time example; just click "I don't know..
Figure A.I: Annotation interface given to Prolific crowdworkers.
User Instruction: Giva some examp things weekend.
Assistant A: For exampic, what should de with kids weekend? What 'I in city What I'm on the beaci??
could going zoo games
you
Instruction: synonym
lewing
User Imput: speecly
Assistant Bu lazy
Figure A.2: Example items given to humans before the start of the task.
17

---

Attention (Single Layer)
Key, query, and value projections: 2 3 seq_len d model (key_size x num heads)
Key and query logits: x seq_len seq_len with_past x (key_size x num_headyy
Softmax: 3 num_heade seq_len req_len_with_pee ppstt
Softmax query reductions: x seq_len seq len with past x (key_siee size x num heads)
Final linear: x seq_len (key_size num_heads) d d_model
Dense Block
seq_len x (d_ml model x ffw size + d_ model x ffw_size)
Final Logits
2 x seg len x d model x vocab size
Total Forward Pass FLOPs
embeddings num layers (attention single layer dense_ block) + final logits
can be seen that only operations in each attention layer depend on the KV cache size, and they take up relatively insignificant amount of FLOPs. As illustrative example, Figure A..4 shows the relative FLOPs contributions within a single layer of attention for LLaMA-7B, assuming 2000-length KV cache and single input token. Operations dependent the KV cache constitute at -10t of the total attention layer FLOPs; the rest are used KQV projections and dense layers for processing the single new input token.
Given KV cache compression rate of 26, as observed our Human validation split, the Chinchilla equations predict relative improvement of Gist caching of 1.%%. This is extremely close to the 0.11% improvement actually observed in Table These results show that optimizing the KV cache size does not actually lead to huge compute speedups during Transformer inference, at least for relatively small prompt lengths. Nevertheless, there are clear memory and storage benefits to be gained from prompt compression, as discussed in Section 6
G Additional Examples of Model Outputs
See Listing A.3 for additional examples of instructions, inputs, model completions (for LLaMA-7B and
across Pos, Gist, TF-IDF, and Neg conditions), human judgments, and ChatGPT judgments (treasons). Examples are not cherry-pickeds we enumerate the first 10 examples of the Alpaca+ Human validation split.
18

---

Pos Control Exact Match
50%
LLaMA-78 -78 FLAN-TS-X0
40%
30%
20%
10%
0%
Seen
Unseen
Human
Figure A.3: Exact match rates. Rate which the token gist models give the output the positive control (exact string match). Error are exact binomial confidence intervals.
softmax 192 (0.1%)
kq logits 16 (4.7%)
softmax query reductions (4.7%6
final linear 34 M (9.7%)
dense block 180 M (51.9%)
káv projections 101 M (29%)
depends cache procassing for single input taken
Figure A.4: FLOPs for each operation involved in single layer of self attention with 2000-1ength KV cache, according the Chinchilla estimates for LLaMA-7B. At most 9.6% of FLOPPs he optimized away by reducing the size of the KV cache.
19

---

Example 1
Instruction: Write good Twitter Blo, Try to cover all the provided main points in your generated blo, 6012
Input: Name: Jane
Kain points:
Architect
Interested in Tech
Written in first person
lseo casual tone
Reference Output:
think of mysall' kind architect.
I's Interested in tech
things, and Tove comiing up ideas, for how mace things better,
TF-IDF Nord: bic
LLaMA-7B
Pos: I'n Jane, architect and tech enthusiast.
in I's here to share ny thoughts and experiences with the world.
always looking
Gist: U' Jane, an architect and tech enchusiast.
I Tove to explore new places and learn about different oul tures.
I's alsoys looking
things
new experiences.
I's an avid reader and enjoy learning about the world around ne.
I's also big fan of tech and love to stay up to cate on the latest trends.
I's looking forward to
is architect what the world has to offer!
interested
technology.
She likes to stay up to date on the latest trends and 15 always looking for new ways to apply thei in her work.
Jane is creative probles solver who apgreaches each project with an innovative eye.
Neg: Jane is archi fect who interested in technology.
She
to stay up
the latest trends and always Looking new ways insorporate their into work.
she's I an avid reader of tech news and loves to explore the latest gadgets.
Jane is also big fan of the arts and enjoys visiting nuseurs and galleries when she gets the chance.
Human choice (Pos Gist): Pos Tie Gist
ChatGPT choice (Pos Gist): Pos
ChatsPT reason (Pos vs Gist): Assistant covers all the rain points provided in the instruction, while Assistant B goes off-tapic
and doesn't t. nention the interest in tech ruth. Assistant elso has a more professional tone, which is nore appropriate for Twicter bio. (Assistant Pos, Assistant B: Gist)
FLAN-T5-XXL
an architect who loves explore
trends.
I's always looking for now ways to nake our lives easter and more conventent.
Fallow ne for the latest tach news, tips, and inspiration!
Gist:
architect who interastad in tach.
She loves to solve probless and create beautiful things.
TF-IDF: Architect, Tech enthusiast
Nag: Tane is an architact who is interested in tech. Pos
choice (Pos
ChatGPT choice (Pos Gist): Pos
ChacGPT reason (Pos va Gist): Assistant covers all the nain points and uses casusl tone. Assistart is too beief and doesn't "t
provide any specific infornation about Jane's interests or personality. (Assistant A: Pos, Assistant B: Gist)
Example 2
Instruction: hare top
that not missed, Include aspects of each place well.
Reference Output: 1- Paris, Best for: Historic sights and romantic vibes 2- Nice, Dest for: Architecture and cining
Lyon, Best for: Dining and live music Bordeaux, Best
kine and antioues
5- Marseille, Best for: Yuseums and nightlife
6- Strasbourg, Best for: Historic architecture and dining Word: nissed
LLaMA-7B
top cities France should nissed include Paris, which is known for its Iconde tecture and bustling nightil'e, re and Nice, which is beautirul city on the Hediterrancan known for its colorful culture and delicious cuisine.
Gist: The top cities in France that should not be nissed are Parts, Nice, tyon, Taulouse, Marseille, and carnes.
incluse Eiffel The attractions Toulouse, the Port Marzeille in Marseille, and the Croisette in Cannes. Canal
Paris, Pronenade des Anglais in Kice, 0ld Lyan,
TF-IDF: aissed you.
Nog: The tost popular social redia platforn in Europe is the Androtd-based Ovi fros Nokia. Gist
Human choice (Pos
Pos
ChatGPT choice (Pos Gist): Gist
ChatGPT reason (Pos vs Gist): Assistant provides a wore comprahansive 1ist of top cities in France and includes the best aspects of each place. Assistant only mintions two cities and their general charactaristties without providing specifii antractionn details. (Assistant Gist, Assistant Pos)
FLAN-T5-XXL
Pos: The top cities in France that should not be missed include Paris, Herseille, Lyon, Toulouse, Nice, and Monaco.
Paris is the capital of France and is hove to the Eiffel Tower, Netre-Done de Paris, and the Louvre Museun. place visit for food and sine.
Marseille is besutiful
vibrant nightlife, and nighelife,
charning city Gist: Paris is one of the nost besutiful cities in France.
Toulouse
beautiful place
and Toxurious hotels.
It is great place to visit for its architecture, culture, and history.
The city is also know for its besutiful gardens and parks, well as rost iconic Landwarks the world, it is nust-see for aryone visiting Paris.
museams and galleries.
The Eiffel Tower is one
The Louvre is another rust see for aryone visiting Paris.
It is stunning museun that houses sore of the nost iconic art pieces in the world.
The Notre-Dane Paris another nust-see for
visiting Paris.
very disopointed.
Neg: The Shawshank Redenstion Forrest Gurp The Godfather Pulp Fiction The Dark Knight
Rusan choice (Pos vs Gist): Pas Pos Pos
ChatGPT choice (Pos Gist): 6ist
20

---

ChatePT (Pos Assistant only several cities doean't provide nuch detail about any then. Therefore, Assistant nore helpful. Assistant A: Gise, Assistant B: Pos)
provides information
Example 3
Instruction:
Give the title category, Pick Fron the list of News & Buzz, Trevel, Style, Arts Culture, Polities, Tech,
Input: The Wankky Eshibit Canbridge, HV is absolutely terrific. Reference Output: Arts & Culture
TF-IDF Nord: bu
LLaMA-7B
Culture
Gist: Arts Culture
TF-IDF: No such bias detected.
Exhibit Carbridge, absolutely fantastic.
choice (Pos Gist):
Chat6PT choice (Pos vs Gist): Tie
ChatePT reason (Pos vs (ist): Both assistants correctly icentified the news title related to Arts Culture. (Assistant A: Gist,
Assistant
FLAN-T5-XXL
Gist: Oulture
TF-IDF: The Bankay Fahibit Canbriage, HA is absolutely resarkable.
#aansy Eshsiitt Cambridge,
resarkable.
Husan choice (Pos vs Gist): Pos
Pos Pos
ChatePT choice (Pos va Gist): Pos
ChatePT reason (Pos Gist): The title is about art exhibit, Assistant 0's of "Arts Oulture' accurate
specific
(Assistant Gist, Assistant Pos)
Example 4
Instruction: You will be tested on your knowledge of classic witticions and aphorisns by completing the given aphorisr. Write the original quote the ansver. <g1> infinite
Reference Output: universe
stupidity; not universe.
(ateributo) to Albert Einstein)
TF-IDF Nord:
LLaMA-7B
nimber integers and things infinite, the universe huran stupidity."
wittlcisus.
Albert Einstein
TF-IDF: The number of integers and the number of real nabers.
things infinite:
of intagers
nubbr prives..
Nusan choice (Pos vs Gist):
Gist Gist
ChatGPT choice (Pos va Gist): Gist
ChatePT reason (Pos Gist): Assistant correctly completes the with the original author, while Assistant 6's is Pos)
factually
(Assistant
FLAN-T5-XXL
Pos: 'Tire and space are infinite."
Gist: "Actions and consequences are infinite."
TF-I0F: things
are statement, while space.
Neg:
Husan choice (Pos vs Gist): Pos Pos Pos
ChatGPT choice (Pos vs Eist): Tie
ChatePT reason (Pos Gist):
assistants did provide the correct completion the aphorisn. The original quote is things are infinite: universe and stupidity; not universe." Therefore, assistant is helpfal. (Assistant A: Gist, Assistant B: Pos)
Example 5
Instruction: Write the HTM syntax For the required action. <01> adding
the
Reference Output: <ing sro="ur1" "tee"tttertttteet"
TF-IDF Word: syntax
LLaMA-7B
Pos: <ing sren"inage. jpE'>
<nouttous>
TF-10r: <ing sre= denee..ngeeeppp jpz">
Neg: <irg srce sree"inage jog" alt="An image of cat">
Husan choice (Pos Gist): Pos IDK
choice (Pos Gist):
ChatePT reason (Pos vs Gist): Assistant provides the correct HTYL syntax for adding an image to the page, while Assistant doesn'
t provvide any sutout. (Assistant N: Pos, Assistant B: Gist)
FLAN-T5-XXL
src= Jp8">
(ic=einage... Jpe">
TF-IDF: cing sree" srea"inagn, "ng">
Neg: Adding an isago to the
Ruzan choice (Pos
Tie Gist
ChatGPT choice (Pos Gist)::
ChatePT reason (Pos vs Gist): Both assistants provided the correct HTN. syntax for addingg an image to the page. (Assistant A: Pos,
Assistant Gist)
21

---

Example 6
inforation.
Input: Daily
pharsacy
Reference Output: Police sivenalk and entrance for litter
Assess window displays rake they outdated Faded
unnecessary signs
rill bags at each checkout area
Clean and straighten checkout counters
seating/waiting
(pull product
departsents
Check one depar tment day for short and outdated product
Enpty all trash receptacles
Vacuun (when the store closed only if custorer traffic allows for it during hours)
LLaMA-7B
Pos: Check inventory of redications
Reorder medications
profiles
patsont Information
computer systen
Versfy patsent inforration
Nonitor expiration dates of redications
shelves
Answor telaphone provide assistance
Process prescriptions
Bafill prescriptions
inventory supolles
Stock
Clean and restock shelves
Answer telophone and provide assistance as neaded
inventory
supplies
prescriptions
Answer custoner questions
Process Insurance paperwork
pharnacy
Stock
Count pills
Take custoner phone calls
redications
Uipdate patient profiles
Porfors adrintstrative tasks
Review patient charts
prescriptions
Answer questions from custorers
Process paytents
Waintain Inventory
controlled
quality control tests
Roviow patient profiles
Contact doctors about prescriptions
Process payments
Maintain inventory
Monitor expiration dates iodications
Update
Perfore adninistrative tasks
Review patient charta
Contact doctors about prescriptions
questions
Process payments
Stock shelves
Count pills
phone
Monitor expiration dates redications
lipdate patient profiles
Perfors administrative tasks
Contact doctors about prescriptionss
Answer questions fron customers
inventory
Monitor expiration dates redications
Update patient profiles
Perfors administrative tasks
Contact dectors about prescriptions
Answer questions from customers
inventory
Nonitor expiration dates on redications
Update patient profiles
Perfors administrative
Contact dectors about prescriptions
Answer questions from custoners
inventory
Nonstor expiration dates
TF-IDF: 1.
22

---

Checking medications
Re-stocking shalves
3.
Entering prescription information corputer systen
Answering custorer questions
5.
Preparing labeling prescriptions
Scaming barcodes to check for potential drug. Interactions
7.
Processing insurance
Paintaining inventory levels of medications
9.
Cleaning organizing phanaacy
18.
Undatinn patient profiles with new infomattion
Nog: The pharnacist is daily tasks include checking and filling prescriptions, naintaining Inventory, and counseling patients on the proper
choice (Pos Gist): Pas
Gist
ChacePT choice (Pos vs Gist): Pos
ChesePT reason (Pos Gist): Assistant provides concise and corprehensive 1ist of daily tasks in the pharmacy, while Assistant
repecitiive Insludes irrel
1ike counting pills perforning quality control
more helpful. (Assistant Pos, Assistant Gist)
FLAN-TS-XXL
Pos: Check inventory Fill prescriptions Prepare medication Label and package nedications Check expiration cates Ensure Check for packaging Check for any problens inventory Check for problems with the redication Check for expiration
redications
any problems Check for any probless with the packaging Check for any problens with the labeling Check for any probless with the packaging Check for problems with the packaging Check for any problens with the
any prublems
Gist: To-do list for
inventory
shelves
Prepare prescriptions
Check inventory and stock sheives 4.
Check inventory stock shelves inventory
Check inventory and stock sheives 7.
Check inventory and stock shelves 5.
Check inventory stock shelves 9. inventory stock shelves Check inventory and stock shelves
TF-IDF: 1.
Stocking
papenwok
Greeting custonars
Taking orders 5.
Performing adninistrative tasks
Nag: daily
the pharnacy proparing medications, filling prescriptions, assisting customers, raintaining
inventory. Husan choice (Pos vs Gist): Gist
Pos
Gist
ChatGPT choice
reason (Pos Gist): Assistant provides conprchensive of daily tasks the pharnacy, including preparing labeling redications, checking for errors and problaxs, and ensuring all redications are in stock. Assistant only lists txo tasks and repeats one of thei nuitiple times. Assistant is nore helpful. (Assistant A: Pos, Assistant B: Gist)
Example 7
Instruction: given paper citation,
style.
Input: Chicagoo: Vasuant, Ashlssh, Shapeer, Noas, Parzar, NIkE, Uerkorwit, Jakob, Jones, 11n,, Coser, N., Kalisr, Likas,, Illia Polosukhin. "Attention Is All You Need. arkiv, (2817). https: /tdoi.org/11 http///....5///// 465Se/arkiv.121 83762.
MEA:
Output: Varwani, Ashish, "Attention Is All You heed.
arkiv, 2817, https://dci, ong/18, 4859//rrxiv 1786. 63762.
TE-10F Word:
LBBMA-7B
Vassani. Ashish, Shazeer, Noan, Farmar, Niki, Uszkoreit, Jakob, Jones, Lison, Gonez, Aidan N., Kaiser, Lukasz, Polosukhin.
"Attantion Is All You heed.
arXiv (ttbol..rg/1...4335//rrr 13762.
httos://dol.
Gist:
Vaswani, Ashish, Shazser, Noan, Parnar, Niki, Uszkorett, Takob, Jones, Lllon, Gonez Aldian N., Kaisor, Lukasz, and Ellia Polosukhin.
Witeetiin 1s You Need.
arXiv, (2817).
https://dol.org/1..
(ttbo..org/1B.4898///rrr 48958//rrtv. 1706.63762.
TF-IDF: Atcention 15 All You keed. arXiv.
2817. httpa:///dl..rrg1..e555333
Nag:
Chicago: Vaswant, Ashish, Shazeer, Noan, Parnar, Niki, liskkoriit, Jakah, Jones, L11on, Gora7, Aldon N., Kaiser, Lukasz, and rilsa Polasukhin.
*Attention
Need."
arXiv, (2817).
ittos://dol.org//1 (tt//8..485580 48558/arxiv. 1706. 63762.
Rusan choice (Pos vs Gist): IOK Tie Tic
ChatGPT choice
23

---

(Assistant Gist, ChatePT (Pos
citation Assistant includes after "arkiy" not necessary HI style, (Assiatant As Gist, Asslitaatt Pos)
FLAN-T5-XXL
Pos: Vaskani, nshish, Shazeer, Noam, Parsar, Niki, Uszkoreit, Jakob, Jones, Llion, Gonezz Aiden N., Kaiser, Lukase, and Illia Polowkkin.
"Atcention Is head.
arXiv, (2817).
Gist: Chicago: Vassani, Ashish, Shazser, Noam, Parnar, Niki, Usrkorait, Jakab, Jones, LTion, Gosez, Aldan N. Kaiser, Lukasz, and T11sa Polasukhin.
arxiv, (2817)
https://doi.org http://dl.og//.1555/0203 1et.///.55////.l...oo 1786.83762.
TF-IDF: Attention is all you need.
Nog:
Husan choice (Pos vs Gist): I0K
IDK
ChacePT choice (Pos va Bist): Gist
ChatePT reason (Pos Gist): The instruction asks for H cleatiion style, but neither assistant provided the cieatiin in M Fornat.
However, Assistant provided more information Assistant Pos)
bOt, Assistant slightly helpful. (Assistant Gist,
Example 8
Instruction: Provide pleasant compliment based on the quality you have been given. <61>
Input: Trushfulness
Reference Output: Everything sould better more people you.
TF-IDF Nord: pleasant
LLaMA-7B
Pos: You resorkably truthful and honest.
Trathfulness pleasant because leads trust, essential for any healthy relationship.
It also helps people build trust in their relationships with others, which can lead to betcer commuiccation and understanding. additionally, it help people avoid getting cought lies and deception, which danaging to relationships. leud, proud, unafrald
Nog: Truthfulness
Nusan choice (Pos vs Gist): Pos Pos
Pos
ChatePT choice (Pos va Gist): Pos
ChacePT reason (Pos Gist): Both responses are good, but Assistant R'A response is nore Metalled and specl fic, naking St. more
pleasent compliment
FLAN-T5-XXL
Pos: You are incredibly honest.
Gist: Thank you for being honest.
TF-I0F: Truthfuiness
that guides through
Neg: Truthruiness 15
Lightnouse, providing guidance
in
Husan choice (Pos vs Gist): 10K Pos
Pos
ChatGPT choice (Pos vs Gist): Pos
ChatSPT reason (Pos Gist):
and follow instruction, but Assistant enthusiastic personal, makiing nore pleasant corpliment. (Assistant Gist, Assistant Pos)
Example 9
Instruction: Make 1ist of acjectives that can be used to describe the given brand. <61> creative tech startup
Reference Output:
Innovative
2.
Creative
Disnuptive
4. cutting-edde
revolutionary
gani-changing
industry-leeding
8. trailblacing
groundbreaking world-cher corld-chorginn
TP-10r Nord: brand
LLaMA-7B
Pos: Innovative, creative, tech-savy,, fonerd-thinkingg dynatic, apile, fast-paced, outting-edge, irventive, pioneering, and disruptive.
Innovvatvee
Creative
Aubitious
Inventive
Innovative
Imaginative
Visionary
Entropreneurtal
Sptimistic
Deternined
Porsistent
Responaible
thoughtful
Respectful
Fepathetic
Relsable
24

---

Thustworthy
Agile
Sotimistic
in
Inaginative
Vistonary
Opt atic
Detormined
Porsistent
Responaible
Thoughtrul
Respectful
Fepathetic
Trustworthy
Adaptable
Agile
TechStartup
Neg: A creative tech startup 1s 15ke race car:: it needs great idea, well-tuned tean and the right resources to reach the
Finissh Tine.
choice (Pos
Gist
choice (Pos Gist)::
ChacePT reason (Pos va Gist): Both assistants provide a sinilar list of adjectives, but Assistant includes sase addi tdditional
adjectives that are vore specific to tech startup, such as "tach-swvy" "forward-thinkinng "cutting-adge", 'plonsering', helpful. (Assistant Gist,
Pos)
FLAN-T5-XXL
Pos:
Innovative Creative Tech Startup Innovative - Tech Startup Creative Tech Startup Tech Startup Creative Tech - Startup Tech Startup Tech Startup Tech Startup Tech Startup Tech Startup - Tech Startup tup Tech Tech Startup Tech Startup Tech Startup Tech Startup Tech Growing
Startup Tech Startup Tech Startup
Innovative
Startup
TF-10r: The Crestive Tech Startup.
Neg: "Creative Tech: Unlacking the Potencial of the Modern World"
Human choice (Pos
Pos Gist Gist
(Pos Bist):
ChatePT reason (Pos vs Gist):: Assistant 3's ressonse is repetitive and not helpful. Assistant A provides concise list of relevant adjectives. (Assistant A: Gist, Assistant B: Pos)
Example 10
Instruction:
Reference Output: Honey Line Fruit Salad Ingredients: (for servings)
1/2 Ib Fresh straxberry(225 8), quartered sangoes,
diced
2 banaras, slicad
1/2 1b fresh blueberry(225
honey
1ime, juiced
Pregaration:
sliced
large
a small bow,, mix honey and Tine juice. Four syrup over the Fruit and mix.
TF-I0F Nord:
LLaMA-7B
Chocolate-coffoe
Gist: Vegan Chocolate- Dippee Coconut Macaroons:
Ingredients:
flour
1/2 cup alnond milk
teaspoon baking powder
vanilla extract
1/2 cup vegan chocolate chips
Instructions:
Prehoat evan to 358F.
2.
boel, together the flour, melted
alnend milk, bakiing powder, and vanills extract.
Grease baking sheet with ron-stick cooking spray.
4.
bowl,
flakes and chocolate chips.
Stowly add the dry ingrediants to the wat Ingredients and nix until combined.
6.
batter prspared
Bake For 18 ninutes, or until golden brown.
8.
let cool bafore
Enjoy!
25

---

Tiranisu
Nag: eost popular redia platforn Europe Androld-aased
Human choice (Pos vs Gist): Gist
Gist
Gist
ChatePT choice (Pos vs (ist): Gist
ChatePT
(ist): Assistant provides recipe for vegan dessert, while Assistant suggests recipe may vegar-friencly as it contains coffee and nascarpone cheese. Assistant is more helofut, (Assistant At Gist, Assistant B: Pos)
FLAN-TS-XXL
Pos: dessert. recipe
conbination nelted chocolate cake.
milk.
topped
layer chocolate chips and Tayer coconut crean.
The cake is baked in preheated oven at 358 degrees for 28 minutes.
Gist: Vegan Chocolate Cake
IF-IDF:
Vanil1a
Neg: The Showshank Redemption Forrest Gurp The Godfather Pulp Fiction The Dark Knight
Husan choice (Pos ve Gist): Pos
Pos
Pos
ChasePT choice (Pos Gist): Pos
detailed recipe vegan chocolate cake, including ingredients bakinn instructions. (Assistant Gist, Assistant B: Pos)
(Pos
Listing A.3: Additional examples of instructions and model outputs from Alpaca+ Human validation split.
26


Reviews:
[{"review_id": "K35Qj13Xyc", "reviewer": "Reviewer_3Sa1", "summary": "This paper presents \"gisting\", which learns language models to compress instruction prompts into smaller sets of compressed context -- that includes special '<GIST>' tokens and the activation stacks above these tokens. Compressing instruction prompts allows saving context windows and saving compute for encoding instruction prompts. The task settings that this paper considers include:\n* [instruction prompt][input][output] --> [compressed instruction prompt][input][output]\n* [instruction prompt][output] --> [compressed instruction prompt][output]\nThe experimental results show that it is indeed possible to represent instruction prompts with much shorter compressed context while having only marginal performance loss. The authors evaluated the gisting model against uncompressed model on instruction following tasks comprehensively with ROUGE-L, preference of ChatGPT, and human evaluation.", "questions": "How is compression factor (in figure 3) estimated? It doesn't seem to be explained in the paper.", "limitations": "Limitations are properly discussed in the conclusion section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Novelty: compressing instruction prompts to save context windows and save compute for encoding instruction prompts is a novel and useful idea that can have good impact on real-world applications.\n* Very solid and well-designed experiments and evaluations: the authors present comprehensive evaluations by ROUGE-L (which is lexical), ChatGPT (which can be considered an automatic semantic evaluation), and human. Furthermore, when showing the human evaluation (Table 2), which is based on a subset of evaluation set, it is compared with ChatGPT results side-by-side and shows that outcomes are consistent and inter-annotator agreement level is similar.\n* Writing clarity: the paper is very easy to follow in most of the part. Figures, plots, and tables are well-polished. ", "weaknesses": "There are some descriptions in the paper such as **G(t) will be a set of soft gist tokens smaller than the number of tokens in t ...** (line 66-68), **Gisting compresses prompts into “gist tokens”** (Figure 1 caption), etc, suggesting that the gist **tokens** contains compressed information. However, if I understand the paper and code correctly, a gist token is really just a special additional token <GIST> in vocabulary and the gist token embedding does not change according to task t. What really contains the compressed information is those activations above the gist tokens. \n\nIn other words, gisting compresses prompts into gist \"activations\" instead of gist \"tokens\" (are inputs to the model). G(t) are activations, not \"soft gist tokens\". When I think of soft tokens, I would think of the type of soft tokens as in prompt tuning, and this seems clearly different. Please correct me if I'm wrong but I think these descriptions are incorrect and misleading and need to be fixed before the paper can be published.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer 3Sa1 for their detailed and helpful review! We are glad you find the paper “novel and useful”, with “very solid and well-designed experiments and evaluations”, as well as “very easy to follow”. Here are some responses to your comments and questions:\n\n> However, if I understand the paper and code correctly, a gist token is really just a special additional token <GIST> in vocabulary and the gist token embedding does not change according to task t. What really contains the compressed information is those activations above the gist tokens. \n\nThank you for pointing this out! Your understanding is correct. We agree that we should clarify in the paper that the goal is to compress prompts into gist activations, rather than gist tokens, of which there is a single gist token embedding that remains unchanged for each task. An analogous way to think about gisting is that it compresses prompts into a soft KV prefix (a la prefix-tuning) rather than soft tokens (a la prompt tuning). We will make the following changes in the camera-ready version of the paper, if accepted:\n\n- L6-7 (abstract): change “gist tokens” to “transformer activations on top of ‘gist’ tokens”\n- L31: Change “gist tokens” to “transformer activations on top of ‘gist’ tokens”\n- Figure 1 caption: change “compresses prompts into ‘gist tokens’” to “compresses prompts into transformer activations on top of ‘gist’ tokens”\n- L67: Change “will be a set of soft gist tokens” to “will the transformer activations on top of a set of gist tokens”\n- L100: Replace “forces the model to compress the information in the prompt into the gist tokens” to “forces the model to compress the information in the prompt into the gist prefix”\n- L142: change “failed to compress any information into the gist tokens” to “failed to compress any information into the gist prefix”\n- L185: change “compressing prompts into a single token” to “compressing prompts into a single token’s worth of activations”\n- L260: Replace with “Caching the compressed activations G(t) on top of the gist tokens”\n\nAlso see our response to reviewer nmKf, who has similar questions about the paper.\n\n> How is compression factor (in figure 3) estimated? It doesn't seem to be explained in the paper.\n\nThank you for pointing this out. The compression factor is calculated by computing the average length (in tokens) of the instruction in each validation split and dividing by the number of gist tokens in the model (since we are replacing the number of tokens in the prompt with N gist tokens). For a single gist token model, a 26x compression factor implies that the average length of the instructions in the human evaluation set is 26 tokens long, as stated in L130. We will make this clearer by adding a sentence after L189 in the paper."}, {"review_id": "VshQ3fUn8x", "reviewer": "Reviewer_W7yi", "summary": "The authors tackle the problem of wasted compute and waste context window space from repeatedly encoding a prompt.\n\nThe authors propose gisting, in which gist tokens are inserted after the prompt, and the attention mask is modified such that tokens after the gist tokens cannot attend to tokens before the gist tokens. This forces the gist tokens to encode and compress the prompt.\n\nExperiments are conducted with two models, LLaMA 7B and FLAN-T5-XXL 11B using a dataset the authors create from Self-Instruct and Stanford Alpaca. A hold-out set of 1000 seen, 1000 unseen, and 252 human prompts are used for evaluation.\n\nThe evaluation is conducted using a ROGUE-L score, ChatGPT3.5 model evaluation, and human evaluation score conducted on a subset of the human prompts.\n\nCompared to an “upper-bound” of training a model with a single gist token, but no modifications to the attention mask, gisting scores near this “upper-bound” using both ROGUE-L and ChatGPT3.5.\n\nHuman evaluation results show gisting winning 52.3% and 40.6% of the time over the “upper-bound” baseline for LLaMA and FLAN-T5-XXL respectively.\n\nCompression rates from gisting are material compared with no caching strategy, but more modest compared with caching the full instruction.\n", "questions": "Could you please clarify “having too many gist tokens hurts performance in some cases … the increased capacity enables overfitting to the training distribution”. In the extreme case, having enough gist tokens to copy the longest prompt t, we would expect equal performance between gisting and using original task instructions t.\n\nCould you please clarify the tie-breaking process referenced in this sentence: “Due to the subjectivity of breaking ties in a forced-ranking task…”. This is fairly important given that this is a proposed explanation for the low agreement numbers. Based on the prompt given to ChatGPT, instructions given to human evaluators, and evaluation examples found at the end of the Appendix, it appears no tie-breaking has occurred.", "limitations": "Gisting loses some nuance in the original instructions.\n\nThe compression level is not dynamic relative to the task.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Gisting is simple to implement, requiring only adding one or more gist tokens, and a few modifications in attention masks\n\nGisting learns prompt compression and instruction following at the same time, incurring no additional training\n\nGisting can be used even with unseen prompts and has some generalization capability\n\nExperiments use relatively large models and a relatively large dataset\n\nPaper is clearly written", "weaknesses": "The evaluation scheme is the primary weakness of the paper. Human evaluators are critical given the nature of the tasks in the dataset (which consist of many open-ended generation tasks) to check how reliable ChatGPT (along with the prompt the authors present in the Appendix) is as an evaluator. We see that human evaluators have low agreement (kappa of 0.24 and 0.33, Table 2). The overall experimental results would be more convincing if human agreement was high. For example, it would be interesting to see results on a high human agreement subset (say tasks with clear objective answers) to see if Gist’s Win % over Pos was still near 50%. Furthermore, Negative Control scores ~25-30% using ChatGPT on various validation splits, which is high considering that ~59% of the tasks in Alpaca+ have no input, and the task is the only information used to generate the output. It is plausible that human evaluators have very high agreement on those outputs generated with Negative Control, but with much lower scores than ChatGPT scores, which would indicate ChatGPT + author’s prompt as an unreliable evaluator. Presented as is, it is hard to tell whether 1) the gisting process is doing well at compressing the task t, resulting in a similar performance as using the original prompt or 2) there is a large amount of randomness inherent in the evaluation process (from factors such as differing human preferences), resulting in a near 50% score (as measured by ChatGPT). These observations, along with the odd fact that reducing the compression rate does not materially improve the scores as measured by ChatGPT casts into doubt the experimental results.  \n\nAs Section 5.1 shows, for some tasks, specific details need to be preserved to successfully accomplish the task. However, the authors claim that increasing the number of gisting tokens k does not help with performance. Furthermore, the choice of the number of gisting tokens is fixed for all tasks prior to training. Therefore, this method does not allow for dynamically trading off compression levels and model output quality.  It would be interesting to see evaluation results on tasks requiring differing levels of specificity.\n\nThe efficiency gains of Gisting is modest compared to instruction caching\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer W7yi for the detailed and careful review!\n\n## On low inter-annotator agreement\n\n> The overall experimental results would be more convincing if human agreement was high.\n\n**On the contrary, when comparing two models of equal quality, we do not expect high inter-annotator agreement!** We will explain by answering this question:\n\n> Could you please clarify the tie-breaking process: “Due to the subjectivity of breaking ties in a forced-ranking task…”.\n\nGood question; we will revise this line for clarity. What we mean is that due to the subjectivity of open-ended generation tasks, there will be noisy reasons why an annotator prefers one response to another, **even if the responses come from models that are equally good.** We expect this to be true even for tasks with objective answers, since in open-ended settings, an answer can be phrased in several ways and annotators might prefer one wording over another.\n\nIn our experiments, annotators have the option to call a tie, and our agreement metric takes ties into account, but neither humans nor ChatGPT call ties often (27% for humans and 13% for ChatGPT). Thus our evaluation is basically a forced choice task, where even if responses are equally good, annotators will often arbitrarily choose one response over another.\n\n**We ran a follow up experiment to verify this by asking 3 additional human annotators to rate *two different sets of samples from the LLaMA positive control***, using the same setup in the paper. As expected, the average win rate of one set of samples over the other is 50% according to humans (52%, 48%, 50%) and ChatGPT (54%). **Importantly, the average pairwise agreement of the humans in this experiment is low, and similar to the paper: a kappa of 0.30.** Meanwhile the agreement of ChatGPT with humans is 0.37. Please see the full Table in the general rebuttal PDF, and recall Table A.2 (a) in the paper shows an average agreement of **.24** for humans and **.29** for ChatGPT for LLaMA.\n\nThis means **we expect low inter-annotator agreement when models are hard to differentiate.** This supports the idea that ChatGPT is similar to human annotation, and that despite low annotator agreement, the win rates across human(s) and ChatGPT are similar, and converge on gisting being similar to the positive control in many cases.\n\nAs a final point, we stress that open-ended LM evaluation is challenging, and the field lacks common evaluation standards. Our forced-choice eval is similar to those done by the LMSys Chatbot Arena and recent LM projects such as Alpaca, Koala, and Vicuna. To mitigate these challenges, we used a wide spectrum of evaluations: first, lexical overlap compared to a gold response (ROUGE-L); second, AI annotation (via ChatGPT) across 1000s of outputs; third, human annotation. Together, we believe these signals provide converging evidence that gisting is competitive with positive controls, even if the inter-annotator agreement is low by design.\n\n## Should more gist tokens improve performance?\n\n> In the extreme case, having enough gist tokens to copy the longest prompt t, we would expect equal performance between gisting and using original task instructions t.\n\n**It is not the case that you would expect equal performance to the positive control when there are many gist tokens.** This is because gist compression is an ***entirely new model capability*** that the LM has to learn **almost completely from scratch**, since the new gist embedding is randomly initialized and the LM has never seen gist masking before.\n\nThe LM needs to learn a new, internal “model” $G(t)$ to compress prompts, and the number of gist tokens determines the “capacity” of this new internal model (more gist tokens = more gist activation parameters). **This means that the generalization performance of the compressor $G$ is subject to the same generalization tradeoffs we consider in standard machine learning:** more gist tokens (“a bigger model”) may result in lower training error (better compression of seen prompts) but worse test error (poorer OOD generalization).\n\nGiven a finite training set, in the extreme case, a model with enough gist tokens could learn to **memorize** each training prompt with a completely unique prefix, matching the positive control during training but failing to generalize.\n\nReviewer W7yi is correct that if the model learned to copy the prompt activations into the gist activations, and attend to the gist activations as if they were the original prompt activations, then this would result in equal performance as the positive control. **However, our results show that models do not learn this mechanism through optimization pressure alone.**\n\n## Other points\n\n> Negative Control scores ~25-30%...which is high considering that ~59% of the tasks in Alpaca+ have no input\n\nWe will clarify in the paper that the 59% figure refers to the **overall distribution** of Alpaca+. For evaluation, **we hold out prompts with non-empty inputs** (except for the Human split). **All prompts in the seen and unseen splits have inputs;** 83% of the prompts in the human split have inputs. See L126-128, though we will further clarify the unseen prompts also have inputs. This means that the negative control can usually make an educated guess about the task given the input. We believe that the fact the negative control is preferred sometimes demonstrates learnable biases in the distribution of Alpaca+, rather than issues with our evaluation.\n\n> does not allow for dynamically trading off compression levels and model output quality. \n\nWe believe the problem of dynamically estimating how much compression is possible for a prompt is a fascinating one, but we leave this for future work.\n\n> efficiency gains of Gisting is modest compared to instruction caching.\n\nPlease see our response to reviewer tZ74. Briefly:\n\n- Gist caching is **1 OoM more memory efficient** than instruction caching.\n- Instruction caching is not possible for encoder-decoder models.\n"}, {"review_id": "0fT6eTkW4n", "reviewer": "Reviewer_tZ74", "summary": "Prompting is the current way of using LLMs, but it occupies the context spaces. Instead of training the LLMs (e.g. fine-tuning), the paper presents a way to compress the prompt into gist tokens, which can be efficiently cached and reused. The method shows 26x compression rate, 40% FLOP reduction and wall clock time speed up. ", "questions": "Please address the weakness above. In particular,  when does prompt has longer sequence length? What would be the corresponding speedup?", "limitations": "The compression is lossy. The reviewer thinks this is significant because the benefits are not clear yet.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The idea is well motivated and straight-forward.\n(2) The implementation through masking is well derived, simple, and well illustrated. \n(3) The baselines for experiments are well proposed.\n(4) The experiment results support the proposed method well (Neg < TF-IDF < Gist <Pos) and is intuitive.\n(5) The failure cases are shown and well explained.", "weaknesses": "(1) The method achieve 4% wall-clock time reduction, which is not significant, especially when it compromises the accuracy. The claim that \"longer sequence length and larger batch sizes can lead to higher speedup\" (Line 267) is not shown quantatively. \n(2) Compressing from 26 tokens to 1 token is not significant, compared to usually 2K context length of the model. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer tZ74 for the detailed and thoughtful review! We appreciate that you find our method well motivated, straight-forward, and intuitive. The main concern of reviewer tZ74 is that the efficiency gains reported in the paper are “not significant”, regarding (1) the wall clock time reductions, and (2) the 26x prompt compression rate.\n\nWe will respond to both claims below, but also keep in mind that **efficiency speedups are not the only contributions claimed by our paper**.\n\n## On wall clock times\n\nReviewer tZ74 claims that a “4% wall-clock time reduction…is not significant.” Note that Table 3 has 95% confidence intervals which show a *statistically significant* improvement in wall times, so we assume the reviewer is using the phrase “not significant” colloquially as “not important”. Respectfully, we believe that the threshold for significance is highly dependent on the use case, and that for large models, a 4% wall-clock time reduction can add up to **significant cost savings over time,** especially when LMs are deployed and used thousands of times daily.\n\n### A highly optimized implementation of our method is out of scope for this paper\n\nFor ease of experimentation and reproducibility, we implemented gisting framework in PyTorch in the popular Huggingface Transformers library (see supplement), with substantial Python logic that likely diminishes our reported wall-time improvements. **We believe an optimized implementation of gisting** would increase wall-time improvements, **but is also out of scope for a proof-of-concept paper** introducing our technique, especially given the numerous non-wall-time benefits of gisting.\n\n## On compression ratios\n\nReviewer tZ74 also argues that a 26x compression rate is not significant. Note there is some debate here, with reviewer nmKf describing the compression ratio as “really promising” and “interesting to some practitioners”. We have two responses:\n\n### 1. 26x compression is an *average* compression rate and is the *maximum possible compression rate for this dataset*\n\nFirst, 26x is an **average compression rate** for the Alpaca+ human validation split. The prompts range from a minimum of 9 to a maximum of **117 tokens** long, i.e. **117x compression** in some cases. Although reviewer tZ74 correctly notes that these prompts tend to be shorter than an LM context window, we expect that this level of compression will allow LM users to avoid “context window exceeded” errors in many cases.\n\nAdditionally, by using just a single gist token, we have achieved the **maximum possible compression rate** explorable with Alpaca+, which to our knowledge was created from the largest instruction following datasets available to us at the time of our experiments (Self-Instruct, Alpaca). Gisting may be competitive at compressing even longer prompts, but exploring this requires a larger and richer dataset that was unavailable to us at the time. We are happy to hear suggestions from reviewers for additional datasets to try.\n\n### 2. 26x memory improvement is significant, *especially for prompt caching workflows*\n\nWe'd like to highlight two clearly significant improvements of gisting for **prompt caching**, where the KV caches of common prompts are stored to speed up inference. This is one of the primary methods for speeding up LLM inference in production (see paper for citations). Within this framework, gisting offers:\n\n- **A**: **An *order of magnitude* decrease in storage and memory costs.** As stated in L284, gisting allows developers to cache up to **26x more prompts from users** than full instruction caching, using the same amount of storage. Even though the relative memory requirement of a *single prompt* is insignificant compared to the memory required for an LLM, if a developer wishes to have hundreds or thousands of prompts cached in GPU VRAM for fast decoding, the prompt storage requirements quickly dominate. For example, the average memory required to store the KV cache of a prompt in the human validation split for LLaMA-7B is 27.3 MB (see L281 in the paper), so caching 100 or 1000 prompts requires **2.73 or 27.3 GB VRAM**, respectively. In such cases, gisting *greatly* increases the number of possible prompts that can be simultaneously cached. We believe this is an important contribution, even if one believes the 26x reduction in prompt length is insignificant for a single prompt.\n- **B**: **New options for prompt caching in encoder-decoder models.** As stated in L258, gisting enables a form of **prompt caching** that is not possible in ordinary encoder-decoder models, since the encoder normally expects to perform bidirectional attention between the full instruction and the input. We believe this is a valuable contribution which opens up new workflows for encoder-decoder LMs.\n\n**In fact, we believe A and B above stand by themselves as valuable contributions to the community, even if there were no other reported efficiency benefits (e.g. wall-time).** We will make it clearer in the paper that these memory improvements are just as important as wall-time, expanding on L284-285.\n\n**Overall,** while it is accurate to describe gisting as lossy, our empirical results show an often negligible impact on downstream task performance. Given the numerous improvements discussed above, we believe gisting will be a useful option to developers as they consider various tradeoffs between accuracy and efficiency in LM inference.\n\n## Other paper contributions besides efficiency speedups\n\nFinally, please keep in mind that efficiency reductions are not the only reported contributions of our paper. We also provide:\n\n- A mathematical framework for “meta-context distillation” of a language model (Section 2.1);\n- A novel way to learn compression in transformers via token-dependent attention masking (Section 3).\n\nWe believe that these aspects of the paper are also stand-alone, useful contributions for those interested in memory, compression, and efficiency in transformers.\n"}, {"review_id": "wImphxMA91", "reviewer": "Reviewer_nmKf", "summary": "Due to the computationally inefficient nature of long prompts for language models (LMs) today, this work proposes one framework to compress prompts into a smaller set of \"gist\" virtual tokens. Different from existing work that distills the context for a single NLP task, this work focuses on a distribution of tasks by predicting the gist tokens (via LMs learned embedding) for each task, seamlessly built upon instruction tuning. The design of their generator G would be the key to their success of the generality to gist tokens. The experiments show the effectiveness of their designs. ", "questions": "See my weaknesses part. Clarifying these in your paper clearly would at least be useful to some readers. ", "limitations": "In addition to what they claimed in their paper,\n\nThe starting point of their proposal is significant to the community, as deploying LLMs with structured, templated prompts are becoming the common practice to a wide range of NLP, or general AI tasks, reducing the computational budgets would be beneficial to a lot of practitioners. However, their framework still require the access to the model logits in their in-context distillation loss, restricting the impacts to white-box LMs. And more precisely, how to reduce the costs for black-box LLMs is more interesting to others.   ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. As soft prompts limit themselves in the computational efficiency, and fine-tuning is limited by the bulk nature by retraining to adapt LMs to each specific task, this work combines the strengths of these two, by learning some virtual tokens seamlessly upon the supervised instruction tuning (a common practice in current LLM deployment and serving). To me, the simple design working well for both training-time and inference-time is elegant and interesting;\n2. The compression ratio looks really promising, making their technical contributions solid and interesting to some practitioners;\n3. The paper is nearly very structured, and the experiments on some instruction tuning datasets look convincing. \n ", "weaknesses": "I still have some confusions about its working mechanisms, regarding unseen prompts, and # of gist tokens (newly added vocabulary to the LM). \n\nFirstly, is it learning one gist token (concatenate k of them in training) per task, and for new task, we do need to add more vocabularies? So it would be like one task one gist token?\n\nSecondly, for unseen and seen prompts, is the training of gist tokens like \"average of the context distillation loss\" over all seen prompts for one particular task, and for unseen prompt, but still the same task type, you simply re-use existing gist tokens (the gist token embeddings would be updated by the cross-attention by the unseen prompts and learned embeddings in inference, though no updates)? Hope my understandings are correct. \n\nI am happy to raise my scores if you can help me address them, and make me feel more interested. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to reviewer nmKf for their detailed and thoughtful review! We appreciate you find the paper “elegant and interesting”, with “really promising” compression ratios and “solid and interesting” technical contributions.\n\nThe main concern of reviewer nmKf seems to be confusion about the gist compression mechanism, which we agree should be explained more clearly in the paper. Below we’ll answer reviewer nmKf’s questions and explain how we’ll revise the paper for clarity. Please also see our response to reviewer 3Sa1, who had some similar questions regarding the explanation of gist tokens in our paper.\n\n> Firstly, is it learning one gist token (concatenate k of them in training) per task, and for new task, we do need to add more vocabularies? So it would be like one task one gist token?\n\n**Only one additional gist token,** ***total,*** **is needed to enable gist caching, regardless of the number of tasks.** In other words, we increase the size of the LM embedding matrix by 1. **For each task, we concatenate k copies of the same gist token.** We will make this clearer by clarifying in L96 that the “k successive gist tokens in between” are **k copies of the same gist token,** and that the gist tokens are the same for each task at both train and test time.\n\nThe reason why the gist token remains unchanged across tasks is because, as reviewer 3Sa1 also points out, **what changes is not the gist token or gist token embedding for each task, but rather the transformer activations computed *on top* of the gist tokens.** Providing the gist token as input to the model serves as a sort of “control signal” that encourages the model to learn to compress the prompt into the activations on top of the gist tokens. The model ideally learns to compress arbitrary prompts into the gist activations, so that it can generalize zero-shot to new prompts at test time.\n\n> Secondly, for unseen and seen prompts, is the training of gist tokens like \"average of the context distillation loss\" over all seen prompts for one particular task, …\n\n\nThis description of the training process is mostly correct, though the loss is not “for one particular task”; the loss is the average distillation loss across all tasks, where you can think of a prompt as being equivalent to a task.\n\n> … and for unseen prompt, but still the same task type, you simply re-use existing gist tokens (the gist token embeddings would be updated by the cross-attention by the unseen prompts and learned embeddings in inference, though no updates)? Hope my understandings are correct.\n\nHere as well, we are not sure what you mean by “task type”: the unseen prompts define unseen tasks that the model needs to compress at test time.\n\nHopefully some of the confusion here is cleared up by our answer to your first question. **At inference time, for unseen prompts, we add the exact same gist token seen during training after the new prompt.** The embedding of the gist token is the same as what was learned during training, and no updates to this embedding happen during test time. **What *does* change is the transformer activations on top of this gist embedding:** via self-attention, the Transformer computes a new unique set of activations on top of the gist tokens, that hopefully contain a compressed version of the new prompt, even though the prompt was never seen during training time.\n\nTo summarize, by training to compress a wide distribution of tasks (=prompts) into the activations on top of the fixed gist tokens appended after the prompt (by averaging the context distillation loss across all training tasks), we hope that the transformer learns a compression mechanism that generalizes zero-shot to unseen tasks (=prompts) at test time.\n\nAgain, please see our response to Reviewer 3Sa1 for additional ways in which we will make the description of our methods clearer in the paper, and let us know if you have further points of confusion—we are happy to continue answering any follow up questions.\n\n> However, their framework still require the access to the model logits in their in-context distillation loss, restricting the impacts to white-box LMs. And more precisely, how to reduce the costs for black-box LLMs is more interesting to others.\n\nThis is a good point! To facilitate sharing and reproducibility with the scientific community, we focus our current paper on white-box, open source LMs for now, though optimizing inference costs of black-box LLM APIs is a fascinating problem.\n"}]

Author rebuttal:
Thank you to reviewers 3Sa1, W7yi, tZ74, and nmKf for their uniformly detailed and constructive reviews, and to the area chair for overseeing this process!

We are glad that a majority of the reviewers are currently positive on the paper, and that reviewers found our ideas “novel and useful” (3Sa1), “well motivated and straight-forward” (tZ74), “simple to implement” (W7yi), and “elegant and interesting” (nmKf), and our experiments “very solid and well-designed” (3Sa1), “convincing” (nmKf) and “support[ing] the proposed method well” (tZ74). We are especially appreciative that reviewers find the paper “easy to follow” (3Sa1), clearly written (W7yi), and “very structured” (nmKf).

We have responded individually to each reviewer’s concerns in direct replies.

**Our attached rebuttal PDF contains a table of inter-annotator agreement statistics we observed from an experiment run in response to reviewer W7yi;** please see that review for additional context. For convenience, we have also reprinted the inter-annotator agreement statistics reported in the original paper as well.


Meta-review: This paper introduces a novel concept of compressing instruction prompts to improve real-world applications. The experiments and evaluations are well-designed, including lexical, automatic semantic, and human evaluations. The writing is clear and easy to follow. However, one reviewer pointed out that the claimed 4% time reduction may not justify the compromise in accuracy (+ instruction caching seems to be a strong baseline already). Also, low human evaluator agreement and uncertainties about ChatGPT's reliability.

Several points need to be addressed before publication:
- The paper contains misleading descriptions regarding "gist tokens" and their role in compressing prompts.
- Better to show generalize ability of prompt compression in a longer prompt context, for example 2k content length.
- Make the technical details more clear, especially about the unseen prompts.
Decision: Accept (poster)