$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput
Authors: Yunho Jin
Chun-Feng Wu
David Brooks
Gu-Yeon Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: Throughput
GPU utilization
Sequence length prediction
TL;DR: We increase the text generation throughput and GPU utilization by predicting the output sequence length and scheduling queries based on the prediction.

Abstract: Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem. To this end, we propose $S^3$, which predicts the output sequence length, schedules generation queries based on the prediction to increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49× throughput over those systems that assume the worst case for the output sequence length.

S Increasing GPU Utilization during Generative Inference for Higher Throughput
Yunho Jin Harvard University
Chun-Feng Wu National Yang Ming Chiao Tung University
David Brooks Harvard University
Gu-Yeon Wei Harvard University
Abstract
Generating texts with large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to even larger than the model itself. This problem exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating complete sequence as they do not know the output sequence length. This restricts to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing system with priori knowledge of the output sequence can mitigate this problem. To this end, we propose S which predicts the output sequence length, schedules generation queries based on the prediction increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49x throughput over those systems that assume the worst case for the output sequence length.
Introduction
Text generation has become increasingly popular in various services, leading to surge in the use of large language models (LLMs) known for their high accuracy. LLMs have distinct features that dif- ferentiate them from non- Transformer-basee models, including a requirement for massive amounts of compute and memory resources. Specifically, we observe that Transformer-bar based LLMs are often limited by memory capacity and bandwidth, resulting in significant underutilization of compute re- sources. When serving GPT-J JJ on an NVIDIA A100 GPU, the utilization of GPU compute resources can as low as 0.4%. This highlights the memory-bounh nature of the L.MMs and the need for efficient memory utilization to increase the utilization of GPU compute resources.
common approach to boost GPU utilization and enhance throughput is to increase batch size. This is due to the fact that inputs within a batch share the same model weights, thus the GPU only needs load the model weight from its high bandwidth memory (HBM) to the on-chip SRAM once and reuse it for all inputs within the batch. The GPU uses more of its compute resources when processing the same model weights. Increasing batch size simple optimization technique and is, therefore, commonly used in serving convolutional and fully-connected neural network models [[2].
However, the self-attention layer in Transformer-based text generation LLMS presents challenge to this simple optimization due to its autoregressive nature. Specifically, when generating a new token in sequence, the model needs to attend to all previous tokens in the sequence, requiring the model to retain all information from previous tokens and store them in HBM. We call this region in the HBM holding the information key/value cache (KV cache). The size of the KV cache grows with larger batch sizes and longer sequences, which limits the maximum batch size, thereby low- ering GPU utilization and ultimately reducing throughput. To support the growing KV cache size, Huggingface's 's Transformers library constantly allocates new memory at each token generation
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GPT-J 6B
GPT-NEOX 20B
LLAMA-13B
LLAMA-33B
6 GPUS
8 GPUS
10 GPUS
2
256
2
256
128
32
64
Throughout (seq
Througheul (sea
16
32
2
16
2
21
44
2
Online: SLO
20
2-1
2-1
2-2
2
2- 22
2
24
2
Latency (sec)
Latency (sec)
Figure 1: Latency versus throughput trade-off among different models (left, online scenario with single GPU running models within the size confines of the GPU) and the number of GPUs (right, offline scenario distributing GPT-3 175B to 6, 8, and 10 GPUs) when generating 60 tokens, inspired by FlexGen (6). The markers in the lines represent batch sizes, from the maximum batch size that can be loaded on an A 100 GPU, incrementing by the power of two. Allocating the exact amount of memory for each sequence expands the curve higher throughput at the cost of higher latency. The solid lines show the trade-off in vanilla systems and dotted lines show how much S can expand the trade-off. The numbers represent the maximum batch sizes for S and vanilla systems. The vertical line in the left figure denotes the latency SLO for reading a 60-token-long sequence.
and incurs latency overhead associated with memory allocation. This improves usability since users do not have to know the output sequence length but suffers long inference latency. Alternatively, NVIDIA's Fasterrrranfoommr library [4] pre-allocates memory for the maximum sequence length, which ends up wasting memory by allocating more than necessary. This approach limits batch size and prioritizes latency over throughput. The trend towards long maximum sequence lengths in recent models (e.g.. 8K or 32K) (5) amplifies the KV cache overhead, demanding more efficient text generation models.
memory utilization in serving Transformer-bart
In light of these challenges, we propose S scheduling sequences with speculation, framework that maximizes throughput via predicting the output sequence length and reducing memory waste. The frequent memory allocations Huggingface's Transformers and the limited batch size in Faster- Transformer stem from the fact that we lack prior knowledge of the output sequence length. S addresses these issues by predicting the expected output sequence length and allocating the corre- sponding amount of memory for each query. It also schedules sequences increase the GPU utiliza- tion. Finally, 3 runs a supervisor the background that detects mispredictions and adjusts the size of allocated memory be more accurate. By integrating these components together, S optimizes memory usage and scheduling to maximize throughput during deployment of Transformer-bar based LLMs for text generation GPUs.
There are two types of LLM deployment scenarios: online and offline. Online scenarios such as chatbots 8 require service providers to generate a sequence within tight latency service level objective (SLO) constraint. Offline scenarios include applications such as scoring [9] or data wran- gling [10). and have loose latency SLOs, emphasizing throughput over end-to-end latency. In con- trast, PaterTransformer
(4) and xFormers (TT) prioritize reducing latency. We argue that ensuring the latency remains below the SL.O constraint renders it unnecessary to prioritize further latency re- duction. To this end, we design S² to achieve higher throughput under those latency SLOs. Figure highlights how much S can improve throughput when trading off latency. For online scenarios, assume latency SLO set to the average reading speed of English readers, 4 words per second [12) and 0.1875 second per token [33 The models the left figure that are smaller than 100 billion pa- rameters satisfy this SLO for all possible batch sizes. This SLO offers service providers to improve throughput over all models. In fact, for LLAMA-33B (14), there are opportunities get throughput benefits with no latency penalties. The right figure, sweeping over different number of GPUs, shows opportunities maximize throughput in offline scenarios that have loose latency SLO.
We evaluate $3 assessing both its throughput and cost-efficiency. Our analysis includes both offline and online scenarios. In online scenarios under the average reading speed latency SLO constraint,

---

we find that S can generate up to 6.49x more sequences while adhering to the same SLO constraint. In offline scenarios, we observe that S achieves a speedup up to 6.49x for different models. S does not affect the models' perplexity as it does not change the models' architectures. Furthermore, we evaluate the cost-efficiency of S and find that using GPUs, S provides almost identical throughput compared to a vanilla system with 10 GPUs.
To summarize, we make the following contributions:
We increase the achievable batch size under longer latency SLO and allow service providers to serve with higher throughput in both online and offline scenarios by using larger batch sizes.
We fine-tune a Distillbert model to predict output sequence lengths given an input prompt.
We provide mechanism to recover from mispredictions. S preempts sequences that ex- ceed their allocated memory and retrain the predictor learn from its mistakes.
Background and Motivation
2.1 Generative AI Models
Transformer-based generative model is autoregressive. It predicts the most probable token based on past tokens. Since the model generates one token at time, has iterate over itself times to generate a sequence that is n-tokens long. One iteration involves an input token traversing through the model which is stack of transformer layers containing one attention, two feed-forward layers. Especially, the self-attention layer uses information on the past tokens to generate the next token.
layer norm, and
For example, the model at the ith iteration attends the current token (t,) with every token it already generated (fo,--.4411 in the self-attention layer. We can express the self-attention layer as:
KT din
Nout =
where d, is hidden dimension of the model, hout, E Rdd are output hidden vector, current query vector, respectively, and K, V E Rexxh are key and value matrices. The jus rows in the K and V matrices represent key and value vectors of tj. respectively. The two dot products attend t, to all the key and value vectors in the current sequence.
The model stores K and V matrices as the key/value (KV) cache to avoid having to generate key and value vectors each iteration. Otherwise, it has store hidden states of every previous token and multiply it with a weight matrix WoKV E Rd4, 2d,, at every transformer layer. This would require 2(i - 111d, Id, FLOPS per layer almost identical to 24d,, FLOPs the self-attention layer at 4, The size of the KV cache is 4ld, bytes per token when using half-precision numbers. The cache uses 2 bytes for every number for both the key and value cache where / the number of transformer layers in model. For example, GPT-NEOX 15 20 billion parameter model, has 44 layers and 6144 hidden dimensions and thus uses IMB per KV cache per token.
2.2 KV Cache Management on GPUs
The KV cache is relatively small (e.g., several MBs) and can be easily stored in the GPU HBM (high-bandwidth memory) when the sequence short as the cache stores information about the previous tokens in the sequence. It grows the model generates more tokens. Using this dynamic nature, Huggingface's Transformers (3) (IF=-rannformers) constantly allocates more memory to the KV cache and stalls the computation until it is complete. This approach allows the library to allocate the exact amount of memory for each cache at the cost of frequent memory accesses.
To mitigate this, NVIDIA's memory for every sequence 16). removes redundant memory accesses by simply filling in the reserved memory in an append-only fashion. However, this approach comes with its own drawback as reserves more than strietly- memory for sequences. For GPT-NEOX with a maximum sequence length of 2048 tokens, Faster Transformer reserves 2048 tokens even for sequences that
library reserves the maximum sequence length of

---

100%
8
2.00
1.88923
69:36 70.56
7.95
0.52
0.41 0.92
3:58 11.5424.0
02
0.2
1.38
2.03
4.02
7.01
16.13
28.01
49.74
77450.25
70.25
NA
32. 0.41 0.92 1.00 2.14 6.12
11.79
24.04
39.13
00.33 (9.22 70.05
32
0.52
1.03
2.02
4.00
7.91
16.27
28.92
50.51
75
04.04
14
NA
Sequence length
04-041 0.81
9.0
1.33
7,98
N4
128 0:41 0,90 1.55 1.17 6.41 17
17.0822
NA
1MA
128
0.52
1.02
2.01
4.05
5,00
15
29.56
NA
AA
144
NA
250
0.78
2.24
NA
N4
250
2.03
8.10
NA
NA
044 0.80 3.20 12.11
14N
1.33
2.04
5.10
NA
14
1024 0.41 0.02 1.04 2.27 0.40 12.50 NA
NA.
NA
NA
164
1024
0.52
1.03 2.04 4.05 NA
NA
NA
NA
KA
NA
NA
NA
256
512
1024
128
512
0%
Batch size
(a)
(b)
Figure 2: (a) GPT-J's and (b)GPT-NEOX's NVIDIA A100 compute utilization.
end up being 50 tokens long. The maximum batch size that model with 80GB A100 GPU is less than 20 with the 40GB model size and 2.2GB per sequence. The small batch size underutilizes the massive compute resources in the GPU. The rationale for reserving the maximum sequence length amount of memory with the underutilization problem is to ensure that generates full sequences and enhances user experience.
can use for this
2.3 Observation
Language models memory bound We demonstrate the extent of GPU resource underutiliza- tion when we run GPT-J with 6B parameters on an A100 GPU. The relatively smaller model shows wider spectrum of different batch sizes and sequence lengths since we can fit larger batches with longer sequences in the GPU. Fig. 2 (a) shows the GPU utilization swept over different batch sizes and sequence lengths. As the figure denotes, increasing the batch size achieves higher utilization but eventually faces a memory cliff, where an out-of-memory (OOM) error kills the process. Take the batches with 1024 sequence length for example. 32 sequences thus 12.56% is the maximum utilization that we can achieve with this sequence length. Fig. (b) shows similar underutilization in the larger GPT-NEOX model due to the memory cliff problem. This model faces the memory cliff with smaller batch sizes and shorter sequences since the model consumes more of the GPU memory. Please note that HF-Transformers still needs to know the output sequence length before batching imputs to avoid the memory cliff.
the maximum batch size and
As the figures illustrate, increasing the batch size can enhance throughput in neural networks. This approach does not require intricate optimizations and enables the GPU to load the model weight from its HBM to the on-chip SRAM only once and share it among larger number of inputs. By doing so, the GPU can activate its idle compute resources and concurrently handle multiple inputs. Nevertheless, the memory cliff poses challenge, limiting the utilization of additional resources. However, as we elaborate, this issue can be resolved.
Reasons behind the inefficiencies Both the frequent memory allocations in HF-Trrasformers and the limited batch size in TrsterTransfrrmer come from the fact that we are not aware of the gener- ated sequence length. If we know the precise length of the generated sequence, we can allocate exact memory to each sequence and resolve the repetitive memory reservation and the unnecessary memory allocation problems Transfosfsrr and Fraterrransformer, respectively.
3 3 Design
S is a system-algeee algorithm co-design framework that maximizes GPU utilization with sequence length prediction to achieve higher throughput. S has three components as shown in Fig. 1) predictor, 2) scheduler, and 3) supervisor. A text generation query arrives in a request pool in the host ARAM. The predictor then predicts its output sequence length which the scheduler uses to batch requests. The scheduler dispatches the batch to the GPU and the text generator model generates texts in the

---

Host
GPU
Predicted length
Scheduler
Predictor
Feedback
Current utilization
Allocate
Text Generator
Supervisor
Preempt
Figure 3: Overview of S², The boxes in yellow denote new components proposed by $
batch. The supervisor oversees the GPU utilizzaiion and handles mispredictions. We describe each component in detail and how they interact with each other in this section.
Output sequence length predictor We use a predictor to predict the output sequence length and resolve the frequent and redundant memory allocation problems. Specifically, we fine-tune a Distil- bert [17 model that was trained for sequence classification to classify which length bucket the output sequence length falls into. We bucketize the sequence lengthes since known that machine learn- ing models not capable of the "last mile" search compared to narrowing the candidates down to the last mile. Each bucket is allocated the range of mez sequencr length and we use 10 buckets.
number of buckets
To this end, we fine-tune the model on the Alpaca dataset (18). one of the representative question- and-answering datasets, and use the questions as inputs and the lengthes of the answers as labels. We observe that this predictor predicts the correct bucket with 98.61 % accuracy. The average distance between the wrong predictions and the correct bucket 1.03 meaning that the error converges to when we double the bucket size. We also evaluate the predictor a model fine-tuned with Google Natural- quurra-nuuetionn dataset [19] and observe an accuracy 77. 13%. makes smaller mistakes more often than larger ones. For completeness, we fine-tune model on the Pile dataset (20), non- question- and-answering dataset, and see 65.6% accuracy. The predictor shows surprisingly high accuracy compared randomly guessing the bins as the latter is correct only 10% of the time.
We choose Distilbert, 66 million parameter model for its small size and fast inference. The model size is negligible since it is smaller than even single transformer layer in the billion-scale models (e.g.. 214 million for 6 billion GPT-J [2T] model). The latency is also negligible since the predictor model runs only once when request arrives at the server while the text generation model runs times to generate an n-token long output sequence. We measure that Distilbert takes 3.7ms to run compared to several seconds for the text generation models on an NVIDIA A100 GPU.
Length-aware sequence scheduler The scheduler batches and schedules sequences based on the predicted results to maximize the GPU utilization without exceeding the GPU HBM capacity. We can formulate this problem bin is the HBM size and the item weight is the size of the KV cache for each sequence.
variant of the bin packing problem with single bin. The capacity
We use the decreasing first fit algorithm 22 as the solution to the bin packing problem for its simplicity. The scheduler queues the lengthiest sequences first, reserving room for shorter sequences within the GPU's available HBM. It orders sequences in the request pool by length in decreasing order and iterates through the pool check if the KV cache of the current sequence does not exceed the available HBM. If so, it includes the sequence in the current batch and reduces the available HBM by the size of the KV cache. The scheduler continues this process until either there is no available HBM or it has iterated through the entire request pool. This approach has been consciously adopted due to its minimal associated overhead, while still maintaining an ppproximately-optine resolution for the problem hand.
The resulting batch is irregularly shaped where some sequences are longer than others. Unfortu- nately, current frameworks either do not support irregularly shaped batch 3 or support it with limited performance (23). Those that do not support this functionality pad the short sequences with padding tokens to match the length of sequences in the same batch. The padding tokens waste both computation and memory since they do not hold any useful information. ORCA (16) introduces an interesting solution to this problem termed selective batching. The authors of the work identify
5

---

Table 1: Model architecture used in the evaluations
Model
Num Params
Num layers
Model dim
Num heads
GPT-J [21
6B
28
5120
16
LLAMA 13B
14
13B
40
4096
40
GPT-NEOX
15
20B
44
6144
64
LLAMA 33B
14
30B
60
6656
52
GPT3 175B
26
175B
96
12288
96
that inputs to certain layers (c.g., feed-forward) share identical weights, in contrast to inputs to other layers (i.e., coff-attention) which do not share weights. Consequently, this enables streamlined batch processing flow, wherein layers with shared weights are batch-processed and the batch is mo- mentarily unpacked process each input serially through the attention layers. ORCA shows that this has a negligible impact on the latency since the inputs to the self-attention layers do not share weights and do not benefit from batching. As such, we follow ORCA and use its selective batching technique.
Also borrowing from ORCA the iteration-level scheduling technique, S does not wait until all sequences the current batch finish generation. Instead, it checks if any sequence the batch has finished generation at every iteration. This grants S higher scheduling flexibility and removes any redundant waiting time. Finally, if model cannot fit in one GPU, S uses pipeline parallelism and shard the models in Transformer layer granularity.
Supervisor The supervisor is charge of supervising the GPU utilization and handling mispre- dictions. The supervisor runs in the background to check for the available space in the HBM and passes the information to the scheduler. The scheduler then appends a sequence to the running batch if the available memory is large enough for the sequence.
The supervisor is also responsible for handling mispredictions. In the case of short predictions, the supervisor preempts those sequences that exceed their reserved memory. It monitors the length of the current output sequences and evicts them if they are not finished but used up its reserved memory. It asynchronously moves the current state of those sequences including the KV cache and the generated tokens to the request pool and frees up the GPU memory. Now the K and matrices are fragmented with blank rows where the evicted KV cache was originally stored in. The supervisor shifts the rows below the blank one so that all rows are stored contiguously. This memory format required by current libaries [24 25) and also resolves the fragmentation issue. Finally, the supervisor doubles the assigned memory for the evicted sequences to fix the short misprediction.
Finally, the supervisor constantly trains the predictor in the background. It uses the sequences that the predictor mistook to train the predictor so that it learn from its mistakes. This training time is relatively short and our measurement shows that each training iteration takes 1 Ims on average while sequence generation takes several seconds or more. This implies that the retraining overhead is less than 10% even if we train the predictor for 10 epochs.
Putting it all together We summarize this section with explanation of how S uses each com- ponent to serve a pool. The predictor predicts the output sequence length of the sequences in the pool. The supervisor runs in the background and checks the current HBM usage. Next, the scheduler uses both the pre- dictions and the available HBM to batch requests for maximum GPU utilization. finishes its job by scheduling that batch to the GPU which generates the scheduled sequences.
request. First, text-generating requests arrive at the request
Evaluation
We show that S achieves higher throughput by predicting the output sequence length. does so by using larger batch sizes hence smaller numbers of iterations. One iteration refers to processing Transformer model once to generate token. We also show that $ can reduce the cost of serving models by using fewer GPUs.

---

ORCA
Oracle
Baseline
S3
Oracle
3.50
3.00
40
2.50
Sequences
Sequeces/
2,00
1.50
1.00
0.50
0.00
GPT-J
LLAMA 138 GPT LLAMA 338
GPT-J
LLAMA-138 GPT-NEOX LLAMA-308
(a) Maximum throughput (Alpaca [18)
Maximum throughput (Google-NQ 19
Baseline
53
Oracle
ORCA
Oracle
1.00
600
500
0.80
Sequences
0.60
200
0.20
100
0.00
GPT-J
LLAMA-138 GPT-NEOX LLAMA-30B
GPT-J
LLAMA 13B GPT NEOX LLAMA 338
(c) Maximum throughput (The Pile (20)
(d) Generated sequences under latency SLO
Figure 4: Latency and throughput of different models and datasets.
Table Maximum throughput of the three systems measured in tokens/s
Baseline
S
Oracle
GPT-J LLAMA- 13B
2061.16
2349.67
2569.09
1018. 15
1641.55
1907.09
GPT-NEOX
490.15
1344.94
1530.05
LLAMA-30B
91.46
593.73
834.30
Environment We our evaluation NVIDIA 80GB A100 GPU connected to the host DRAM via PCle x.0x8 8 in a Lenovo ThinkSystem SD650-N V2 Server 27
Baselines We compare with ORCA [16), Transformer serving system that increases throughput by iteration level scheduling and selective batching- ORCA to allocate the maximum sequence length of memory when it is not aware of the output sequence length to guarantee full sequence generation. 5 predicts the output sequence length and allocates memory based on the prediction. We implement the systems on top of FasterTransformer (4) since this library is faster than HF- Transformers (3) due to more optimizations. also compare S with an ideal system with perfect predictor which we term Oracle.
Models We use models ranging from billion parameters 175 billion parameters for the evalu- ation. The specifics of these models are explained in table
4.1 Throughput Analysis
We evaluate ' throughput using Alpaca (18). Google Natural Questions (Google-NQ) [19 and The Pile (20) datasets. Specifically, we query 5 with questions and ask it to generate the answers.
Maximum throughput Fig. (a) (c) reports the maximum throughput in sequences per second for different models and different datasets. We measure the throughput using the maximum batch size of each configuration. It shows that S outperforms ORCA by 1.13x and up to 6.49 x and closely matches Oracle, differing from 9.34% and up to 40.52%. The difference magnified with larger models because the batch size is limited even for $ since most of the HBM
Offline scenario:

---

Table Average batch size and number of iterations for different models
ORCA
S
Oracle
Model
Batch size
Num iter
Batch size
Num iter
Batch size
Num iter
GPT-J
69.94
7988
530
1054
564.88
989
LLAMA 13B
31.61
17675
274.66
2034
527.04
1060
GPT-NEOX
16.89
33069
157.59
3545
292.19
1912
LLAMA 33B
4
139790
42.47
13155
77.35
7223
ORCA S Grade
80
Generation
Penalty
Overhead
70
100%
60
30
80%
Sequenos
33.44
60%
88
8
20%
Number of GPUS
GPT-J
LLAMA
GPT NEOX LLAMA
GPT3
Figure 5: Maximum throughput of GPT3 run- ning on different numbers of GPUs.
Figure 6: Latency breakdown of $
used to hold model weights. The batch size in S gets cut off before saturating the throughput as shown in Fig. We also report the throughput in tokens/s in table eevaluated on the Alpaca dataset.
We can notice that the maximum throughput increases by 6.49x while the batch size increases by nearly 10x for every model. This comes from the unique behavior of Transformers. Feed-forware layers in the models benefit from batching layers while self-attention layers do not. This is because inputs in feed-forwards share the same weight while inputs in self-attentions attend to their own sequences. S performance benefit is increased when the parallelized portion is larger than the serialized portion. However, increasing the batch size entails adding more serialized inputs thereby growing the serialized portion. GPT-J is a good example of this characteristic which shows a similar throughput jump of 1.1 from ORCA to S and 1.09 from S and Oracle while the batch size differs by 8.69 and 1.97x, respectively.
Online scenario: SLO-aware throughput We now consider scenario with a latency SLO con- straint. We set the SLO as 0.1875 seconds to generate one token given that average English readers read at words per second [12] or 0.25 second per word, and 0.75 words per token [13]. We next calculate the latency SLO for each sequence by multiplying 0815 by its sequence length (c.g., 11.25s for sequence with 60 tokens).
Fig. 4d reports the number of sequences that each model generates using ORCA, S and Oracle. Oracle exceeds the SLO for GPT-J, LLAMA 13B, and GPT-NEOX when it chooses the maximum batch size. So we limit the batch size for these models during this evaluation. S generates a similar number of sequences with the ideal case and 1.13x to 6.49 x more sequences compared to ORCA. The throughput increase similar to offline scenario since S meets the SLO in most cases with its maximum batch size. However, the difference between S and Oracle reduces by 10% compared to the offline scenarios because we limit the batch size hence the throughput Oracle.
4.2 Cost Analysis
We evaluate with different numbers of GPUs. We partition GPT-3 into and 8 GPUs in pipeline- parallel manner, allocating 16 and 12 transformer layers per GPU for each configuration, respec- tively. We also evaluate on GPU setting where we allocate 10 layers remaining GPU. $ pipelines each batch and schedules the batch whenever the GPU processing the first partition passes its result to the next batch. This reduces GPU idle time by having every GPU processing batches concurrently.
GPUs and 6 to the
8

---

Table 4: Average batch size and number of iterations for different system configurationss on Alpaca
ORCA
S
Oracle
Num GPUs
Batch size
Num iter
Batch size
Num iter
Batch size
Num iter
11.95
46734
114.22
4891
209.47
2667
8
28.68
19482
247.85
2254
470.65
1187
10
48.99
11403
399.62
1398
564.88
989
Table 5: Accuracies and runtime of different predictors on different datasets
Datasets
Runtime
Model accuracy (%)
Model size
Alpaca 18
Google [19]
The Pile [20]
(ms)
MS-minibert
22M
98.06
77.99
60.1
2.3
Distilbert-base
66M
98.6
82.68
65.6
4.1
Bert-base
110M
99.54
85.08
68.2
7.6
Bert-large
340M
99.6
89.25
71.9
14.5
that S achieves similar throughput using 6 GPUs compared to ORCA with GPUs. ORCA shows 0.92% higher throughput than S be specific. More GPUs shard the model into finer pieces and leave more space for storing the KV cache, allowing us to increase the batch size. Table 4 supports this claim by reporting larger batch sizes with more GPUs. S can achieve similar effect with fewer GPUs by optimizing memory allocation strategy.
Fig. 5 reports the maximum throughput using the different numbers of GPUs. First,
Naturally, it leads to similar question with the one in the throughput evaluation why S with GPUs shows similar throughput with ORCA with 10 GPUs even with 2.33x the batch size. The answer is in the pipelined execution and the sequential nature of self-attention layers in Transformer- based models as explained in 4.1 The systems complete batch at every kumber of GPUS instead of at every layers. For example, S with 6 GPUs completes a batch at every 16 layers instead of 96 for GPT3. Similarly, ORCA with 10 GPUs completes every 10 layers and thus processes more quickly. The increase in the latency negates the throughput benefit from $ such that the two systems show almost identical throughput even with the 2.33x larger batch size when using $3
4.3 Overhead: Latency Breakdown
We evaluate runtime latency of each component in S². We classify the runtime latency into three categories:
generation, penalty, and overhead. Generation is the time S spent processing the model to generate tokens. Penalty denotes the time it took for S to preempt the KV cache and hidden states from the GPU and load it back to the GPU. Overhead includes the time it took for predictor, scheduler, and supervisor, combined. Fig. [ show that penalty and overhead combined negligible (i.e., 11% average) compared increase if the predictor is less accurate and triggers more data traffic between the GPU and the host. In contrast, the overhead will increase if we employ a more accurate but heavier predictor, thus introducing a new trade-off.
the generation. Of course, the penalty would
4.4 Predictor Ablation Study
We vary the predictor size from 22M 340M parameters and report their accuracies different datasets in table 5 We can observe a similar trend in all three datasets where a larger predictor generates more accurate predictions. The accuracies differ among different datasets since the length distribution differs. Specifically, Alpaca [18) showed the smallest variance among the length distri- butions in the three datasets compared to The Pile 20 showing the greatest.
Related Works
The high interest in machine learning has sparked numer- ous research in service platforms 4 16 28 38). Especially, the surprising performance of Transformer-based language models has directed many researchers to develop Transformer-specffee
Machine learning serving systems

---

serving systems 6 (7 39) Most of the systems focus on reducing the latency without much concern for throughput with the exceptions of [6,]]. The throughput-oriente systems use memory hierarchy to store parts of the model in slower memory and to increase the batch size. However, they all allocate the same memory to every sequence and do not consider preempting a sequence based on prediction. S can improve these systems by reducing the required memory size, removing the need for larger but slower memories such as SSDs, and reducing the fiscal cost of memory overall.
FastServe 39 tackles the head-of-line blocking problem caused by query-level scheduling while S addresses the inefficient memory allocation issue. It proactively manages KV cache similar to S supervisor by migrating sequences between the host memory and the GPU HBM. However, FastServe does not have an output sequence length predictor and thus uses a skip-join Multi-Level Feedback Queue since it is unaware of the job execution time. We expect Sè to work with FastServe so that S predictor delivers more information to FastServe for better scheduling.
Reducing the KV cache overhead The issue of attention layers in Transformers requiring quadratic computation and memory with respect to the sequence length has been extensively studied. Various approaches have been proposed to address this problem. Low-rank approximation [40[4]] and exploiting sparsity 442 44) are among the methods that aim to mitigate the issue. Another ap- proach, known as multi-query attention [5), suggests reducing the cache size by utilizing one attention head per key-value (KV) cache instead of employing multiple attention heads, as illus- trated in Table 1. Additionally, works focusing model size reduction through compression [46) can also contribute to reducing the size of the KV cache. These ap- proaches are complementary to work and can be employed to reduce the penalty caused by mispredictions, allowing for the use of a less accurate but lighter predictor.
and quantization 66 47 (77488) 48)
Sequence length prediction While there are limited works that predict output sequence length based on an input sequence, Yan et al. (49) propose convolution- small network with embed- ding layers to forecast output sequence length in machine translation. In our approach, we employ similar strategy but utilize a small Transformer-basse predictor to estimate the output sequence lengths in text generation tasks. This predictor allows us accurately predict the output sequence length for text generation workloads.
6 Limitations and Conclusion
Limitations We make the following assumptions in this work. We assume that text generation request traces mimic the publicly available question-and-ansserinn datasets since there are no publicly-available traces. Analyzing the actual trace would facilitate deploying S in commercial workloads. Similarly, text generation task does not have any standardized latency SLO constraint as in other machine learning workloads (50) since it is a relatively new service. So we assume average reading speed of an English reader our SLO. We will be able to evaluate S² in broader scenarios if organizations publicly release different SLOS for different text generation applications.
Conclusion summary, we introduce $3, a framework designed to achieve high throughput in serving Transformer-based generative models. S leverages predictor to estimate the output length of generated sequences and schedules them accordingly to maximize throughput. Additionally, $ handles potential prediction errors guarantee reliability. By allocating varying memory sizes to different inputs, SZ acknowledges that not all sequences should be treated equally. This approach expands the conventional trade-off between latency and throughput frontier, paving the way for new possibilities in optimizing the latency-throughput trade-off.
7 Acknowledgement
We thank the anonymous reviewers for their thoughtful comments and suggestions. This material is based upon work supported by the National Science Foundation under Grant No. 1704834 and No. 211 18985 and supported part by the Application Driving Architectures (ADA) Research Center and the National Science and Technology Council, Taiwan, under Grant No. 112-2222-B-A49 MY2. Chun-Feng Wu acknowledges the support from the Yushan Young Scholar Program by the Ministry of Education (MOE) in Taiwan.
10

---

References
[1] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, "Clipper: A Low-Latency online prediction serving system," in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). Boston, MA: USENIX Association, Mar. 2017, pp. 613-627. [Online]. Available:
[2] R. Chard, Z. Li, K. Chard, L. Ward, Y. Babuji, A. Woodard, S. Tuecke, B. Blaiszik, M. J. Franklin, and I. Foster, "Dihub: Model and data serving for science," " in 2019 IEEE Interna- lional Parallel and Distributed Processing Symposium (IPDPS), 2019, pp. 283-292.
[3] H. Face, "Transformers,"
[4]
NVIDIA, "FasterTransformeet" http://hhhhooomm///////////aaaaa
[5] OpenAI, "OpenAI research GPT4," https///ooenaa..com///sss//////
[6] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y. Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez, P. Liang, C. Ré, Stoica, and C. Zhang, throgh-t Generative Inference of
Large Language Models with a Single GPU," arXiv e-prints, p. arXiv:2303.0686. Mar. 2023.
[7]
Google, "Bard," httt:::s/barr..goglcccoo
[8] OpenAI, "ChatGPT," hntpss//chaat.openai.coom
[9] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan,
Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao,
J. Wang, K. Santhanam, L. Orr, L. Zheng. M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli,
T. Hashimoto, T. Ieard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Mai, Y. Zhang, and
Y. Koreeda, "Holistic Evaluation of Language Models," arXiv e-prints, p. arXiv:221 1.222...000 1,09110,, Nov. 2022.
[10] A. Narayan, I. Chami, L. Orr, S. Arora, and C. Ré, "Can Foundation Models Wrangle Your Data?" arXiv e-prints, p. arXiv:2205.099 009911, May 2022.
[11] B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, Hu, M. Tin- tore, S. Zhang, P. Labatut, and D. Haziza, "xformers: A modular and hackable transformer modelling library."
2022.
[12] M. Brysbaert, "How many words do we read per minute? a review and meta-analysis of reading rate," Journal of Memory and Language, vol. 109, 104047, 2019. [Online]. Available:
http://ccecciettccooo//eeeee/eee https:/heelpappeaaacccm/mmmaaata
[13] OpenAI, "OpenAl API Documentation."
4936856 what are- tokens and- hww.-towoouutttteeeeeeeeot count-them
[14] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.. 1122.11971 13971, 2023.
[15] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leaby, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, "GPT-NeoX-2D" 20B: An open-source autoregressive language model," in Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models. Dublin:: Association for Computational Linguistics, May 2022, pp. 95-136. [Online]. Available:
[16) G.-I. Yu, J. S. Jeong, G.-W. Kim, Kim, and B.-G. Chun, "Orea: A distributed serving system for Transformer-Bant Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp- 521-538. [Online]. Available: http://wwwwwwexoooooooeoeeoeoode
generative models," in 16th USENIX Symposium on Operating Systems
[17] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT,, a distilled version of BERT: smaller, faster, cheaper and lighter," arXiv e-prints, p. arXiv:1910.00 108, Oct. 2019.
[18] R. Taori, I. Guirajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Stanford alpaca: instruction-following Ilama model," https://github.com/
tatsu-
2023.

---

[19] Т. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, Dai, J. Uszkoreit, Q. Le, and S. Petrov, "Natural questions: benchmark for question answering research," 2019.
[20] L. Gao, S. Biderman, S. Black, L. Golding. T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," p. arXiv:2101.00000 000027, Dec. 2020.
[21] B. Wang and A. Komatsuzaki, "GPT-J-6B: Billion Parameter Autoregressive Language Model," https///gthub..oomknngoooooooool
transoootttttttttmer-eeee May 2021.
[22] D. S. Johnson, "Near-optimal bin packing algorithms," Ph.D. dissertation, Massachusetts In- stitute of Technology, 1973.
[23] PyTorch, "Pytorch Nested Tensor," hhtps://yyurcaagggdossssaaeeeeee
[24] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang.
De-
Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32. Curran Asso- ciates, Inc., 2019, pp. 8024-8035. [Online]. Available: hhtp::/papers.nnett www.rss.nuuuuuuucccc cccuris.cc/ppper// haip://oroooaammmppaaaaaeesyh--- peaprorrmannceddeeellainnnngg library.pbrr
[25] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis. J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., "Tensorflow: A system for large-scale machine learning." " in 12th {USENIX] Symposium Operating Systems Design and Implementation ((OSDI) 16), 2016, pp. 265-
283.
[26] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan,
R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler,
M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and
D. Amodei, "Language Models are Few-Shot Learners"" arXiv e-prints, p. arXiv:2005.1466.
May 2020.
[27] Lenovo, "Lenovo ThinkSystem SD650-N V2 Server," 1p1396-thnnsssttem-
https//lenovopresslennvvcco
Soasttem-d6500-ie sd650-n- V---seeer
[28] Romero, Q. Li, N. Yadwadkar, and C. Kozyrakis, "INFaaS: Automated model- less inference serving," in 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, Jul. 2021, pp. 397-411. [Online]. Available: https:
[29] A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and Mace, "Serving DNNs like clockwork: Performance predictability from the bottom up." in 14th USENIX Symposium Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 443-462. [Online]. Available:
[30] S. Choi, S. Lee, Y. Kim, J. Park, Y. Kwon, and J. Huh, "Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing." " in 2022 USENIX Annual Technical Conference (USENIX ATC 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 196-216. [Online]. Available: https:/wwww.uuennxx.oooooorreeee
presentatiom/choo-seunnggg
[31] R. Yazdani Aminabadi, S. Rajbhandari, M. Zhang, A. A. Awan, C. Li, D. Li. E. Zheng, J. Rasley, S. Smith, O. Ruwase, and Y. He, "DeepSpeed Inference: Enabling Efficient In- ference of Transformer Models Unprecedented Scale," arXiv e-prints, p. arXIv:2207.00033. Jun. 2022.
[32] U. Gupta, S. Hsia, V. Saraph, X. Wang, B. Reagen, G.-Y. Wei, H.-H. S. Lee, D. Brooks, and C.- J. Wu, "DeepRecSys: A System for Optimizing End-To-Ent At-scale Neural Recommendation Inference," arXiv e-prints, p. ArXiv:2001.02772 Jan. 2020.
[33] S. Hsia, U. Gupta, B. Acun, N. Ardalani, Zhong, G.-Y. Wei, D. Brooks, and C.-J. Wu, "Mp-rec: Hardware-software co-design to enable multi-path recommendation," in
12

---

Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS 2023. New York,
NY, USA: Association for Computing Machinery, 2023, p. 449-465. [Online]. Available: http:///...rrrro.11///5/5//...//
[34] J. Fang, Y. Yu. C. Zhao, and J. Zhou, "TurboTransformert An Efficient GPU Serving System For Transformer Models," arXiv e-prints, p. arXiv.2010.05680, Oct. 2020.
[35] C.-F. Wu, C.-J. Wu, G.-Y. Wei, and D. Brooks, "A joint management middleware improve training performance of deep recommendation systems with ssds," in Proceedings of the 59th ACM/IEEE Design Automation Conference (DAC 22), 2022, pp. 157-162.
[36] C. Olston,, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V. Rajashekhar, S. Ramesh, and
J. Soyke, SeensorFlow-Serinng Flexible, PerfoPeeroe ML Serving," arXiv e-prints, P.
arXiv:1712.06133 Dec. 2017.
[37] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek,
K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," arXiv preprint
arXiv:2211.06 1.00100, 2022.
[38] X. Wang. Y. Xiong. Y. Wei, M. Wang, and L. Li, "LightSeq: A High Performance Inference Library for Transformers," arXiv e-prints. p. arXiv:2010.1388
Oct. 2020.
[39] B. Wu, Y. Zhong, Z. Zhang, G. Huang, X. Liu, and X. Jin, "Fast Distributed Inference Serving for Large Language Models," arXiv e-prints, p. MrXiv2335.005920 May 2023.
[40] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, Sarlos, P. Hawkins,
J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller, "Rethinking At- tention with Performers," " arXiv e-prints, p. arXiv:2009. 14794, Sep. 2020.
[41] V. Likhosherstov, K. Choromanski, J. Davis, X. Song, and A. Weller, "Sub-Linear Memory: How to Make Performers SLiM," arXiv e-prints, p. arXiv:2012. 11346, Dec. 2020.
[42] I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The Long-Document Transformer," arXiv e-prints, p. arXiv:2004.05150, Apr. 2020.
[43] M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula,
Q. Wang, L. Yang, and A. Ahmed, "Big Bird: Transformers for Longer Sequences," arXiv e-prints, p. arXiv:2007.1400 14062, Jul. 2020.
[44] H. Wang, 7. Zhang, and S. Han, "SpAtten: Efficient Sparse Attention Architecture with Cas- cade Token and Head Pruning," arXiv e-prints, p. arXiv:2012.08 2.09522, Dec. 2020.
[45] N. Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need," arXiv e-prints,
p. arXiv:1911.02155: Nov. 2019.
[46] J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap, "Compressive Transformers for
Long-Range Sequence Modelling," arXiv e-prints, arXiv: 1911: 1915500777, Nov. 2019.
[47] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, "I-BERT: Integer-only BERT
Quantization," arXiv e-prints, arXiv:2101.0132 Jan. 2021.
[48] Z. Liu, Y. Wang, K. Han, W. Zhang. Ma, and W. Gao, "Post-training quantization for vision transformer," in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 28 092-28 28 103. [Online]. Available: http://2///leecc85566676777 77999997776777777777ll 77eadddd777ac777cce5555 pateeedd//aceSS--pppepppppp
files/
[49] Z. Yang. Y. Gao, W. Wang, and H. Ney, "Predicting and using target length in neural machine translation," in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. Suzhou, China: Association for Computational I. Linguistics, http:///aaattttoooooo/////////aa
Dec. 2020, pp. 389-395. [Online]. Available:
[50] V. Janapa Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. St. John, P. Kan- war, D. Lee, J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhi- menko, A. Tejusve Raghunath Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, Wei, Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou, "MLPerf Inference Benchmark." arXiv e-prints, p. arXiv: 1911::9911.225... 1.02549, Nov. 2019.
13


Reviews:
[{"review_id": "PaTf48dEBk", "reviewer": "Reviewer_CH64", "summary": "The paper proposes a simple yet effective systematic solution to increase GPU usage and throughput during inference. The authors first make an interesting observation that the existing large language models are bounded by memory and the inefficiency is caused by the lack of awareness of the sequence length. Accordingly, the authors first fine-tune a small model DistillBert to predict the sequence length. Based on it, the authors accordingly schedule the generation of queries and deal with mispredictions. The authors conduct experiments with models from 6 billion to 175 billion parameters to evaluate the effectiveness of the proposed system.", "questions": "See weakness.", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ The paper solves a practical problem, personally, I think improving the GPU utilization can benefit the community and the deployment of different applications.\n\n++ The proposed solution is simple yet effective. Predicting the output sequence length first is a simple and intuitive way to schedule the network and better utilize the GPU. Although incorporating a new model introduces additional lags, the authors show that such overhead is negligible in Figure 6.\n\n++ The results look promising; the proposed method allows for larger inference batch size and during the inference, e.g., 32 -> 256 for LLaMA-13B and 4 -> 32 for LLaMA-32B, and hence achieves higher throughput.\n", "weaknesses": "-- Subsection Length-aware sequence scheduler in Section 3 is not very intuitive to the readers, especially the ones that are not directly working on the same topic. It would be better to illustrate more on how the bin packing problem is solved and the batching technique in ORCA (with an Algorithm or diagram or more descriptions).\n\n-- The authors only evaluate the proposed method on NVIDIA A100 GPU. However, it seems the design is not dependent on the architecture of A100 GPU. It would be better to also evaluate the proposed method on other GPU architectures.\n\n-- Minor: Distilbert in L169 and Distil-bert in L155", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **More details of $S^3$ in Section 3 Subsection Length-aware sequence scheduler**\n\nIn addressing the bin-packing problem, we leveraged the decreasing first-fit algorithm, which orchestrates bin-packing by prioritizing the largest to smallest load sequence. Notably, $S^3$ orchestrates its packing strategy by queuing the lengthiest sequences first, reserving room for shorter sequences within the GPU's available HBM. This approach has been consciously adopted due to its minimal associated overhead, while still maintaining an approximately-optimal resolution for the problem at hand.\nIn a distinct vein, ORCA introduces a novel technique termed selective batching, explained upon in the range of lines 190 to 196. Within the layers of a Transformer model, inputs to feed-forward layers share identical weights. However, attention layers prohibit weight sharing among inputs. In a shrewd maneuver, ORCA discerns this challenge within Transformer-based models and strategically batch processes layers amenable to weight-sharing inputs. Consequently, this enables a streamlined batch-wise processing flow, wherein the batch is momentarily unpacked to process each input serially through the attention layers. This output is subsequently packed into a batch configuration, facilitating collective processing within non-attention layers.\nAs we endeavor to refine and enhance the comprehensibility of our submission, we will incorporate the elucidated insights into the camera-ready version, ensuring a more intuitive and accessible presentation for the esteemed readership upon its potential acceptance.\n- **$S^3$’s performance on different GPUs**\n\nWe evaluated $S^3$ on NVIDIA A100 GPUs since they were the accessible state-of-the-art GPUs. We could not get access to H100 due to its global hardware shortage. Similarly, we could not evalaute on AMD GPUs due to their limited programmability. \n- **Minor: Distilbert in L169 and Distil-bert in L155**\n\nWe will use the term Distilbert throughout the revised version of the manuscript.\n"}, {"review_id": "62L9VGeAhV", "reviewer": "Reviewer_945f", "summary": "This paper presents a new solution to the challenges of GPU underutilization and increasing batch size in the generation task of Large Language Models (LLMs), rooted in the memory-intensive requirement to retain the K and V values of prior tokens. To tackle these issues, the paper proposes an efficient framework named S^3, which uses a fine-tuned Distillbert model as a predictor to forecast the output sequence length based on an input prompt. This predictive model guides query generation scheduling and manages mispredictions by halting sequences exceeding allocated memory and doubling their assigned memory. The proposed approach increases the maximum configurable batch size in both online and offline scenarios and improves throughput, reporting a throughput increase of up to 6.49 times. Although this paper offers improvements in handling memory allocation for LLM inference, there are some concerns about evaluation and ablation of the proposed techniques.", "questions": "- The figure captions could use more detail. For instance, is Figure 1 (left) showing an online scenario, and how many GPUs were used? Similarly, does Figure 1 (right) represent an offline scenario, and how large is the model size?\n\n- The term \"number of iterations\" requires clarification. It's evaluated in the sections assessing performance, but isn't clearly defined. \n\n- In Section 4.1, it's mentioned that \"Oracle exceeds the SLO for GPT-J, LLAMA 13B, and GPT-NEOX when it chooses the maximum batch size\". It's unclear why Oracle would select a batch size that violates the SLO.\n\n- Both Figures 4 (a) and (b) claim a maximum performance improvement of 6.49x. How can this be, especially when Figure 4 (b) pertains to \"Generated sequences under latency SLO\"? Is the data in Figure 4 (b) based on a scenario of maximum throughput?\n\n- The application of S^3 seems to yield less throughput improvement as the model size decreases. Is there still a significant performance improvement when applying S^3 to smaller models like GPT-3 small, which has around 125M parameters?\n\n- Regarding throughput-related performance metrics, it's unclear how Oracle's (ideal predictor and scheduler) performance was measured. The scheduling might have been done under an entirely ideal scenario, or it might be based on the assumption that the predictor makes no errors in predicting input sequences.", "limitations": "The authors properly stated the limitations of the proposed methods.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper tackles the vital challenge of predicting output sequence length through a unique DistillBERT-based co-optimization of the system and algorithm. This approach effectively addresses the limitations of existing frameworks, namely HF-Transformer and FasterTransformer, leading to improved throughput while meeting SLO latency.\n\n- While larger models typically yield better performance, as the space for K, V caches reduce, effective memory allocation management becomes increasingly essential. The S^3 framework notably elevates the maximum throughput, particularly in larger models.\n\n- The S^3 framework optimizes the latency-throughput trade-off by adjusting memory allocation, leaving the model architecture intact and thereby maintaining model perplexity. Operating independently and focusing on performance enhancement, the framework doesn't impact the accuracy of existing LLM models.\n\n", "weaknesses": "- While a major contribution of the paper is the prediction of output sequence length, the effectiveness of this proposed predictor isn't thoroughly evaluated. Tables 2 and 3 show a significant difference between the batch size predictions of S^3 and Oracle. The reasons behind this discrepancy need to be explained.\n\n- The effectiveness of the predictor appears to vary greatly with the fine-tuning dataset, with accuracies ranging from 65.6% to 98.6%. The paper proposes online learning, but this isn't evaluated. Additionally, the paper lacks an ablation study on the choice of predictor models and sizes.\n\n- The paper only compares the latency-throughput trade-off between vanilla systems and S^3 and doesn't compare average throughput and latency in specific tasks. This is crucial, as output sequence length prediction accuracy, which seems to be task-dependent, would likely impact this.\n\n- While S^3 meets the SLO limit, its implementation generally results in an increase in end-to-end latency. The paper doesn't adequately address how much each proposed technique contributes to this overhead.\n\n- The paper introduces a supervisor component to handle mispredictions, but the description of this technique is unclear. The supervisor appears to allocate double the memory when a previous allocation isn't sufficient. However, in typical scenarios with scarce memory, this could lead to problems if the predictor misestimates the batch size. More detailed explanations of various scenarios would be useful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **The discrepancy between the batch sizes in $S^3$ and Oracle**\n\nThe predictive mechanism of $S^3$, elaborated upon in Section 3 under the Predictor paragraph, revolves around forecasting the length bucket. $i^{th}$ bucket contains sequences with lengths in [max_seq_len/num_buckets * i, max_seq_len/num_buckets*(i+1)). Consequently, even when the predictor attains accurate answers, a potential variance of up to max_seq_len / num_bucket remains. Nevertheless, our throughput assessments validate $S^3$'s efficacy in capitalizing on a substantial batch size that saturates utilization of available compute resources. Fig. 1 demonstrates the saturation and throughput evaluation in Section 4 reports the small throughput difference.\n\n- **Ablation study on the predictor**\n\nWe agree with the reviewer that the ablation study will strengthen the paper. The accuracy results are shown in the table below. Larger models result in higher accuracy but longer latency. We chose Distilbert by eliminating other models. Specifically, MS-minibert had low accuracy while Bert models were too large compared to their accuracy gain. \n\n|Model accuracy|Model size|Alpaca|Google|Pile|\n|-|-|-|-|-|\nMS-minibert|22M|98.06|77.99|60.1\nDistilbert-base|66M|98.6|82.68|65.6\nBert-base|110M|99.54|85.08|68.2\nBert-large|340M|99.6|89.25|71.9\n\n- **Evaluation on different tasks**\n\nWe report the throughput of $S^3$ on different tasks including text retrieval [19] and web scrape [20]. The benefit of $S^3$ decreases when tested on web scraping. There are two main reasons: 1) lower predictor accuracy and 2) longer output sequence length. 1) Lower predictor accuracy generates larger data traffic between the GPU and the host, increasing the latency. 2) Longer output sequence length results in smaller memory waste when allocating maximum sequence length of memory to each sequence. Therefore, memory savings by $S^3$ are reduced.\n \n|||Alpaca|Google|Pile|\n|-|-|-|-|-|\nGPT-J|ORCA|35.73|2.78|0.75\n||$S^3$|40.73|2.85|0.80|\n||Oracle|44.54|2.90|0.85\nLLAMA-13B|ORCA|17.65|2.77|0.62\n||$S^3$|28.46|2.97|0.64\n||Oracle|33.06|3.11|0.64\nGPT-NEOX|ORCA|8.50|1.33|0.39\n||$S^3$|23.32|1.49|0.43\n||Oracle|26.53|1.58|0.54\nLLAMA-30B|ORCA|1.59|0.25|0.09\n||$S^3$|10.29|0.54|0.12\n||Oracle|14.46|0.59|0.15\n\n- **Overhead breakdown of $S^3$**\n\nWe merged the latency overhead attributed to each constituent element within the $S^3$ in Fig. 6 due to two reasons. Firstly, the observed overhead across the components was found to be inconsequential; secondly, it's imperative to emphasize that the effectiveness of $S^3$ hinges upon the collective utilization of all its components. To elucidate, using solely the predictor, without the scheduler or the supervisor, or the exclusive deployment of the supervisor in isolation, fails to yield any discernible advantages or enhancements.\n- **More detailed role of the supervisor**\n\nThe reviewer is correct that the supervisor allocates double the previously allocated memory when the previous allocation is insufficient. However, we found out that doubling the initial prediction resolved most of the mispredictions in the evaluated task. \n- **Details in Figure 1 caption**\n\nThe left figure portrays an online scenario, where a single GPU is allocated for each model. This setup assesses the impact of $S^3$ on language models that remain within the size confines of a single GPU. Conversely, the right figure illustrates an offline scenario, wherein the GPT-3 175 billion parameter model is partitioned across a varying number of GPUs. This configuration aims to show the performance of $S^3$ within a distributed computational environment. We will augment the caption in the revised version for clarity.\n- **Clarify the term “number of iterations”**\n\nOne iteration refers to processing a Transformer model once to generate a token as explained in Section 2 L87. We will reiterate this at the start of Section 4.\n- **Why does Oracle choose a batch size that violates the SLO**\n\nWe designed the system with Oracle to maximize throughput. We implemented the system so that Oracle knows the output sequence length prior to the actual generation, without additional difference with $S^3$. Therefore, it will maximize its batch size in order to maximize the throughput, hence violating the SLO.\n- **How does throughput enhancement under latency SLO match the one without the SLO constraint?**\n\nGenerated sequences under the latency SLO in Fig. 4 (b) were counted when ORCA, $S^3$, and Oracle systems maximized their throughput. Sequences that exceeded the latency SLO were ignored during the count. Both Fig. 4 (a) and Fig. 4 (b) show the same maximum throughput increases. This is because the largest batch size that $S^3$ could utilize for LLAMA 33B model was small enough such that the latency of processing requests fell below the latency SLO. We direct the reviewer to Fig. 1 (a) where the SLO is longer than the execution latency of the maximum batch size when using $S^3$.\n- **How does $S^3$ apply to smaller models?**\n\nThe reviewer is correct in that $S^3$’s benefit is diminished when using smaller models. This is because smaller models are able to utilize large enough batch sizes to saturate the GPU computation resources without efficient memory management due to the small model and KV cache sizes. However, using GPUs with smaller HBM will cause the same issue of memory scarcity and $S^3$ will help in a similar manner. As the trend toward ML models is enlarging the model size, we believe that efficient memory utilization will become more important.\n- **How was Oracle’s performance measured?**\n\nWe evaluated Oracle’s performance by only switching the predictor in $S^3$ with a predictor that always makes the correct prediction. We first ran a profile process that measured the output sequence lengths. Then we ran the actual evaluation where we measured the latency and throughput assuming that the generation models generate identical output as the models in the first profiling process.\n"}, {"review_id": "j3Qu7YO3Uy", "reviewer": "Reviewer_1WaJ", "summary": "The paper tackles the problem of predicting the number of generated tokens for transformers in text generation tasks. This will help with better memory allocation and batch size management. The previous systems either used dynamic memory allocation (Hugging Face) which incurs inference overhead, or preallocated memory for over-estimated output length, limiting the batch size. This paper's prediction of output length achieves a larger batch size, thus increasing the throughput. Experiments show a 6.49x improvement in throughput. The predictor is get by training a Distilbert model.", "questions": "1. Figure 1: is it get from generating with 60 token context length? This is not a long context.\n2. line 71: \"6.49× more sequences\" is vague, because sequences can have different lengths.\n3. line 212: \"The supervisor shifts the rows below the blank one so that all rows are stored contiguously.\". How much time overhead does this step cause?\n4. Why are you measuring the throughput as sequence/s, not token/s? Sequences have different lengths, which makes sequence/s hard to interpret. I can understand that you want to count how many requests can be served, but maybe also add plots for token/s.\n5. When you are using the Alpaca dataset, how did you setup the request distribution (coming time, etc)? \n6. How does your method compare to this one \"Fast Distributed Inference Serving for Large Language Models\" that was released recently?\n\nminor:\nline 47: typo before \"since most of the HBM is used to...\"\nline 307: [31, 39] are not focus on throughput.", "limitations": "To accurately predict the output length, the distribution of the test trace should be similar to the data used to train the predictor. Also, different models will generate answers with different lengths, which requires the training data to reflect a similar behavior to the model used in the generation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea is simple but effective.\n2. The paper is clear, well-written, and easy to read.\n3. The evaluation is clear and convincing. It contains various settings, including different models, different hardware setups, online/offline with clear information presented (latency breakdown, batch size).", "weaknesses": "1. It has not been discussed that different models can generate outputs with different lengths.\n2. The evaluation does not contain different request patterns.\n3. Missing the evaluation of throughput in term of token/s.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Do different models generate sequences with different lengths?**\n\nDifferent models showed similar output sequence length distributions that were skewed toward short sequences.\n- **The rationale behind using sequences/s.**\n\nWe wanted to provide the intuition of throughput in terms of how many more requests can $S^3$ process compared to other systems. This metric is similar to job completion time used in “Fast Distributed Inference Serving for Large Language Models” [51]. However, we will add throughput evaluation in terms of tokens/s on the additional page. Please find the result in tokens/s below.\n\n \n|             | Baseline | $S^3$ | Oracle |\n|---------|------------|----|---------|\nGPT-J| 2061.16  |2349.67|2569.09|\nLLAMA-13B|1018.15|1641.55|1907.09|\nGPT-NEOX|490.15|1344.94|1530.05|\nLLAMA-30B|91.46|593.73|834.30|\n\n\n- **Are the results in Fig.1 from generating 60 tokens?**\n\nYes, the results in Fig.1 are from generating 60 tokens. We used this sequence length as it is the sequence length used by Google in their paper [37] when measuring the latency and throughput of different transformer-based language models.\nWhat is the request distribution?\nDue to the unavailability of real-world request traces, we assumed that every request is queued in the request pool. The objective of the experiments was to evaluate the performance of $S^3$ during high pressure and to observe the maximum batch size that $S^3$ can generate during such a scenario. \n- **How does $S^3$ compare to “Fast Distributed Inference Serving for Large Language Models”?**\n\nFastServe, the system proposed in the paper “Fast Distributed Inference Serving for Large Language Models” and $S^3$ share similar approaches but with different motivations. $S^3$ and FastServe both use iteration-level scheduling and host-memory offloading. While $S^3$ addresses the inefficient memory allocation issue, FastServe tackles the head-of-line blocking problem caused by query-level scheduling in current LLM serving systems with focus on the scheduler.\nFastServe introduces two contributions, a skip-join Multi-Level Feedback Queue (MLFQ) and proactive KV cache management systems. The proactive KV cache management system is similar with $S^3$’s supervisor in that both of them preempts sequences that have low priority to the host memory and loads them back on the GPU HBM when there are enough available memory. The skip-join MLFQ is used since FastServe does not have an output sequence length predictor and thus does not know the job execution time. It only uses the input sequence length as a guide to set the priority in its skip-join MLFQ. We expect $S^3$ can work with FastServe so that the predictor can give more information to FastServe’s skip-join MLFQ to further accelerate language model serving systems.\nWe will add this in the related works section in the camera-ready version if accepted.\n- **Minor typos**\n\nWe will fix the typos in the revised version.\n\n**New references**\n\n51. Fast Distributed Inference Serving for Large Language Models"}, {"review_id": "mUr9lxhl07", "reviewer": "Reviewer_s4gr", "summary": "The paper proposes a scheme that increases the throughput during inference on Transformer large language models (LLM). Typically, LLMs require large amounts of memory, for model parameters and for the KV (key/value) cache. The KV cache size depends of the output sequence length, which is not known when inference starts. Some implementations allocate memory for the KV cache in small increments, causing large latency, while others preallocate for up to the maximum output sequence length, causing potential memory waste. The authors propose a method, called S3 (scheduling sequences with speculation), that predicts the output length, and allocates the KV cache accordingly. The predictor is a fine-tuned Distilbert model (66M parameters) with small size and fast prediction time. A scheduler batches requests according to a greedy strategy. A supervisor is in charge of checking GPU utilization and handling mispredictions, while at the same time training the predictor in the background. An experimental analysis shows the flexibility of S3 in offering a trade-off between latency and throughput.", "questions": "1. Could you add to the experimental section results for the case when no KV caching is used?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Transformers are memory and compute intensive. Designing schemes that make better use of memory, thus increasing the latency and/or throughput is an important topic. The proposed solution is simple, and the predictor ran as part of S3 adds negligible overhead. The experimental results show improvements, generating by up to 6.49 times more sequences compared to other existing system.", "weaknesses": "The system will need more analysis in the future, based on real traces (which are not available for research community now), and considering realistic service level objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Real traces and SLO**\n\nWe agree with the reviewer and believe that the paper could be strengthened by using real-world request traces and realistic latency SLOs. However, as mentioned both by the reviewer and in Section 6 Limitations, we acknowledge that the traces are not available to the research community at the time of writing. Additionally, we set the latency SLO in our paper is set as the reading speed of an average human [12]. But this will be strengthened if the research community can agree upon a standard benchmark as in MLPerf [50].\n- **How does $S^3$ perform when no KV caching is used?**\n\nNumber of FLOPs for recomputating the key (K) and value (V) matrices increases linearly with respect to the previous sequence length. Additionally, the K and V matrices should be stored in the GPU HBM to process self-attention layers as explained in Section 2. So the regenerated K and V matrices will use the identical amount of memory capacity as KV caches and the memory capacity problem will persist. We will add the throughput evaluation of this configuration to Section 4 in the revised version.\n"}, {"review_id": "vcHPTyNlQ9", "reviewer": "Reviewer_RWv5", "summary": "This work builds an LLM inference platform, called S^3, around a sequence length predictor.\nThe sequence length predictor is used to\n1. batch LLM generations\n2. pre-allocate kv cache (where all seq have similar predicted seq len)\nS^3 also has a method for handling seq len prediction errors. They do pipelined generation. They unpad shorter seq to ignore wasting compute.", "questions": "The authors note: \"More GPUs shard the model into finer pieces and leave more space for storing the KV cache, allowing us to increase the batch size. \" - line 280\nThe method also pipelines generations; kv caches for all generations must be stored in mem, this in turn lowers the possible batch size. Is it possible to quantify the trade off?\n\nWill the code be made public? People can easily benchmark the implementation helping the authors identify shortcomings and improve the overall performance over time.\n", "limitations": "Requires training a seq len predictor which hopefully generalizes to different data distributions.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "LLM inference is becoming a larger and larger part of the total compute used within large organizations. Decrease inference costs is an extremely impactful line of work.\n\nThe analysis of overheads is good.", "weaknesses": "\nFor kv cache pre-allocation, the authors note that pre-allocation limits seq len, not pre-allocating kv cache slows generation. An obvious baseline is pre-allocating for S tokens at a time (eg S=64) and freeing kv cache mem of generations which have finished (this frees mem for the really long generations). This simple baseline would decrease the kv cache concat overhead by a factor of S and eliminates the overhead of the S^3 algo.\n\nThe work uses a sequence length predictor to build an LLM inference platform. This is great systems research, I'm not sure it's positioned well for NeurIPS.\nThe work doesn't show how much of the benefit comes from pipelining vs kv cache preallocation vs seq len batching vs unpading. I'm not sure how much improvement each method is or if we're introducing other overheads by including methods such as seq len prediction.\n\nThere are great works on improving LLM inference. Why only compare to ORCA?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **How does $S^3$ perform against pre-allocating S tokens at a time?**\n\nTable below shows the throughput compared to the reviewer’s recommended system with S=64. Compared to $S^3$, pre-allocation of S tokens allows larger batch size but at the cost of more frequent evictions. That is the scheduler batches assuming that every sequence has input_sequence_length + S tokens at the start. Thus it is able to batch more sequences when S is smaller than the actual generated sequence. However, in such a case, the GPU HBM becomes full more frequently and more sequences are evicted to the host memory and loaded back to the HBM, generating more data traffic. \n\n||GPT-J|LLAMA-13B|GPT-NEOX|LLAMA-30B\n|-|-|-|-|-|\nPre-allocate|26.57| 15.32 | 12.1 | 8.39\n$S^3$|40.73|28.46|23.32|10.29\n\n- **Positioning for NeurIPS?**\n\nThe emergence of expansive models characterized by extended processing periods and substantial energy consumption has triggered a surge of interest among NeurIPS researchers in the optimization of machine learning (ML) systems. In substantiation of this trend, NeurIPS convened a series of ML systems workshops in 2022 [55, 56], reflecting the growing emphasis on system-level enhancements within the field. Furthermore, a slew of ML systems papers have graced the discourse in recent times. Notable examples include Nimble [57], MCUNet [58], and FlashAttention [59], which garnered attention by securing slots in the main conference track. The examples, among others, exemplifies the burgeoning focus on crafting and refining ML systems to align with the evolving landscape of large-scale models.\n- **Benefit from each technique in $S^3$ and latency overhead**\n\nThe main contribution of $S^3$ is the predictor that predicts the output sequence length and pre-allocating the KV cache. Other techniques such as pipelining, batching, and unpadding are proposed in other works. Specifically, pipeline-parallel execution is used by many works [4, 16, 31, 39] while sequence length batching and unpadding are introduced in ORCA [16]. We compare $S^3$ to ORCA to pinpoint the benefit of our main contribution.\nAs the reviewer might already know, we want to re-iterate that the overhead introduced by $S^3$’s predictor, scheduler, and supervisor are 11% on average, combined.\n- **Why only compare to ORCA?**\n\nWe compared $S^3$ to ORCA since it was the state-of-the-art deployment system at the time of writing. However, ORCA is not opens-sourced so we implemented ORCA on top of FasterTransformer [4], an accelerated language model processing library by NVIDIA. Thus we are comparing $S^3$ to both ORCA and FasterTransformer by comparing to our implementation of ORCA. \n- **Tradeoff of pipelining and memory consumption**\n\nThe reviewer is correct about a large KV cache is one of the main culprits of using smaller batch size. However, please note that not all KV caches for the entire generations should be stored in GPU memory. Rather, the scheduler in $S^3$ ensures that the currently running batch size does not exceed the GPU memory capacity and stores KV cache for those sequences that are not currently running in the host memory as described in Section 3 subsection Length-aware sequence scheduler.\n- **Opensourcing the code**\n\nThe code of $S^3$ will be open-sourced if accepted. We look forward to the research community making full use of our work and developing $S^3$ over time.\n\n**New references**\n\n52. Challenges in Deploying and Monitoring Machine Learning Systems (https://nips.cc/virtual/2022/workshop/49982)\n53. Machine Learning for Systems (https://nips.cc/virtual/2022/workshop/49961) \n54. Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning\n55. MCUNet: Tiny Deep Learning on IoT Devices\n56. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\n"}]

Author rebuttal:
We extend our sincere gratitude to the reviewers for their thoughtful and insightful feedback on our NeurIPS submission, titled “$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput.” Their meticulous evaluation and constructive comments have undeniably enriched the quality of our work. We highly appreciate the time and effort invested by the reviewers in meticulously assessing our manuscript. In this rebuttal, we address each of the reviewers' comments and suggestions comprehensively, outlining the revisions and enhancements we will make to our paper in response. Through this iterative exchange of ideas, we aspire to contribute to the advancement of knowledge in the field while maintaining the rigor and integrity of our research. Thank you once again for your invaluable contributions to our work.

Meta-review: Overview: The paper proposes a simple yet effective systematic solution to increase GPU usage and throughput during inference. The authors first make an interesting observation that the existing large language models are bounded by memory and the inefficiency is caused by the lack of awareness of the sequence length. Accordingly, the authors first fine-tune a small model DistillBert to predict the sequence length. Based on it, the authors accordingly schedule the generation of queries and deal with mispredictions. The authors conduct experiments with models from 6 billion to 175 billion parameters to evaluate the effectiveness of the proposed system.

Pros:

- Instead of listing, the positive points are mostly within all the reviews included in the reviewing process. Interesting Problem Setting, Simplicity of approach, Thorough literature review, Interesting results.

Cons: 

- Only disadvantage the lack of exhaustive results, that was handled during the discussion.

Overall: Based on the initial reviews + discussion with the authors, the paper has only positive comments, and all the comments raised by the reviewers were adequately tackled by the authors.

The only requirement for the authors is to include (if possible and if space allows) any additional discussion that is useful during the rebuttal phase, in order to improve the paper. Please do not include material that is not presented during the rebuttal and material that cannot be checked, unless another round of review is required.
Decision: Accept (poster)