Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models
Authors: Deepak Narayanan
Keshav Santhanam
Peter Henderson
Rishi Bommasani
Tony Lee
Percy Liang
Venue: NeurIPS 2023 poster (2023)
Keywords: Systems for Machine Learning
Inference efficiency
Transformer models
Text generation APIs
Capability-efficiency tradeoffs
TL;DR: We propose a new metric and methodology to compare inference efficiency across different autoregressive Transformer-based model APIs.

Abstract: Large language models (LLMs) are highly capable but also computationally expensive. 
Characterizing the _fundamental tradeoff_ between inference efficiency and model capabilities is thus important, but requires an efficiency metric that is comparable across models from different providers.
Unfortunately, raw runtimes measured through black-box APIs do not satisfy this property: model providers can implement software and hardware optimizations orthogonal to the model, and shared infrastructure introduces performance contention.
We propose a new metric for inference efficiency called _idealized runtime_, that puts models on equal footing as though they were served on uniform hardware and software without performance contention, and a cost model to efficiently estimate this metric for autoregressive Transformer models.
We also propose variants of the idealized runtime that incorporate the number and type of accelerators needed to serve the model.
Using these metrics, we compare ten LLMs developed in 2022 to provide the first analysis of inference efficiency-capability tradeoffs; we make several observations from this analysis, including the fact that the superior inference runtime performance of certain APIs is often a byproduct of optimizations within the API rather than the underlying model.
Our code is open sourced at https://github.com/stanford-crfm/helm-efficiency.

Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models
Deepak Narayanan'
Keshav Santhanam Stanford University
Peter Henderson Stanford University phend@cs stanford.
NVIDIA dnarayanan@nvidia.com
Rishi Bommasani
Tony Lee Stanford University tonythle@@stanford..o
Percy Liang
Stanford University n mlprish@@stanford edu
Stanford University pliang@cs irtteecsssttan stanford.edu
Abstract
Large language models (LLMs) are highly capable but also computationally ex- pensive. Characterizing the fundamental tradeoff between inference efficiency and model capabilities is thus important, but requires an efficiency metric that is comparable across models from different providers. Unfortunately, raw runtimes measured through black-box APIs do not satisfy this property: model providers implement software and hardware optimizations orthogonal to the model, and shared infrastructure introduces performance contention. We propose a new met- equal
ric for inference efficiency called idealized runtine, that puts models footing as though they were served on uniform hardware and software without performance contention, and cost model to efficiently estimate this metric for autoregressive Transformer models. We also propose variants of the idealized runtime that incorporate the number and type of accelerators needed serve the model. Using these metrics, we compare ten LLMs developed in 2022 to provide the first analysis of inference efficiency-capabilii tradeoff's; we make several observations from this analysis, including the fact that the superior infer- ence runtime performance of certain APIs is often byproduct of optimizations within the API rather than the underlying model. Our code is open sourced at https///gggtuuu..coomsttnnrrrddd
httpp::/gittub.. www.stanooodoooooo/oooooooooooo wwwranffrrdcffm/eeee Lm- efficiency
1
Introduction
Large language models (LLMs; Devlin et al. 2019 Brown et al. 2020
Lieber et al. have grown in size by almost four orders of magnitude in recent years, achieving state-of-the the-arr accuracy on traditional tasks like question answering and summarization (Zellers al. 2019 Hendrycks et al.
Rae et al.
2021
2021 Smith al., 2022 Black et al.) 2022; Chowdhery al. 2022 OpenAl 2023
2020; Zhang et al.) (202).. LLMs display many exciting new capabilities as well. like reasoning
about the physical world (isk et al. (020), solving math problems (Cobbe al. [021 Li al. 2023), and generating code (Chen et al. 202T). to name few. To capitalize on these capabilities, several organizations offer access LLMs through black-box APIs (OpenA]) A12T Cohere) and many companies are deploying LLM powered products at scale like ChatGPT, Bing, jasper ai and Github Copilot (Reuters) Microsoftt Scale VP).
When building models, both users and developers must balance the benefits of new capabilities against the costs of scale. Recent efforts have begun to systematically evaluate and compare the downstream
Work done while author Microsoft Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Raw runtime denoised runtime noise)
Prompt
Black-boo API
Prompt
Idealized runtime
Chosen hardware and software (e.g., A100 GPUs and Megatron)
Prompt has num_prompt___ gom_prompt_tokene tomett, output has num nun_output_tokee tookens
Figure 1: Comparison of raw runtime the two runtime metrics proposed in this work.
task accuracies LLMs (Brown al.
2020; Rac et al. 2021 Liang al., 2023 Srivastava et al.
2023). while others have examined the massive energy, financial, and computational costs of model
2020 Strubell et al. 2019, Bender et al. 2021 Patterson et al. 20 Bommasani et al. |021 $5.3). However, few have considered the trade-offs of inference
training (Cao al. 2020 Henderson et al.
efficiency vs. capability improvements (Bommasani et al.) 2023). This is important since model
inference costs might far outweigh training costs for certain applications (e.g., ChatGPT had about 180 million unique visitors in August 2023 (Reuters 2023
Our main contribution in this paper is a cost model for the end-to-end runtime of autoregressive gener- ation using Transformer models with multi-head attention (Vaswani et al.) 2017). Since autoregressive inference is composed of two stages that have vastly different computational properties (Cursor 2023), the end-to-end runtime cannot be estimated directly from the total number of floating- ppint operations. We instead consider each of these two stages independently, and observe that inference runtime can be expressed as the sum of parameterized piecewise linear function of the number of prompt tokens and a linear function of the number of output tokens under certain assumption These parameters are specific to the target model and the software hardware stack used. Our cost model can be efficiently fit to runtime profiles collected on either dedicated hardware or through black-box APIs, and allows us to estimate the runtime of a workload without running it in its entirety.
Raw runtimes of inference queries to black-box APIs are not inherently comparable across model providers since the API can include optimizations orthogonal to the model (c.g. caching, customized hardware, etc.) and be susceptible to performance variance (e.g., in our experiments, we found that heavy load can worsen raw runtime by up to 2x for certain model providers). This makes it hard to gauge the inference efficiency of models on level playing field, which can be important for model creators and researchers to understand the full long-termy costs of various training decisions (e.g.. model architecture/ / number of parameters). Raw runtime is still good metric for end users who are directly impacted by slow (or fast) predictions.
We propose inference efficiency metrics that facilitate apples-to-apples comparisons across models. The main metric we propose is the idealized runtime, which is the runtime of an inference query run on a specified common software and hardware stack. The idcalized runtime can also be used to compute idealized energy and dollar costs to take into account the cost of the hardware used to serve the model. All of these idealized metrics can be estimated efficiently with our cost model.
Using these metrics, conduct novel analysis of inference efficiency-capabile various Transformer models available through black-box APIS ($5.)). Our analysis reveals several insights about the relative performance of these models. For example, the vanilla OpenAI/davinng model is often on the Pareto frontier of the afficiency-capabilii trade-off landscape when using raw runtime the efficiency metric on 4 NLP scenarios covering sentiment analysis, question answering and classification. However, this efficiency appears to come from optimizations within the API rather than the model; /davnci consistently not on the idcalized runtime Pareto frontier. Overall, we found the idcalized metries to be a useful tool for model creators and researchers to understand the true inference runtime, energy, and dollar costs that result from a particular model architecture and training process.
tradeoffs for
"Our cost model focuses on dense Transformer models for now. We believe this a reasonable compromise since modern text generation APIs are powered almost exclusively by such models. The cost model needs to be slightly modified to cover popular variants like mixture-of of expert (MoE) models. We also mainly concentrate on small to medium context windows in this paper, see $3.1.2 for full discussion.

---

2 Autoregressive Inference of Language Models
The input to language model is sequence of tokens (e.g., words). Autoregressive language models like GPT-3 (Brown et al. 2020) estimate the conditional probability Pr(2,/21,-11 of token æ; given prefix tokens ........ -i During training, where we know all tokens in the training input a priori, the conditional probabilities Pr(x1 (0), Pr( Pr(zg (T::2).... Pr(x.//a1s-- can be estimated in parallel, and thus only single forward pass needs to be executed in every iteration before the backward pass. However, at inference time, outputs of the model need to be fed back in as imputs to generate subsequent outputs. In particular, a token 2; is sampled from the conditional probability distribution obtained by running a forward pass through the model. Different sampling approaches can be used to obtain the token z; from the conditional probability distribution (1: common approaches include greedy sampling, random sampling with temperature annealing, nucleus sampling, and beam search. The process then needs to be repeated for the next token 1;+1 and so on. Consequently, inference through an autoregressive language model needs to perform multiple forward passes. This entire procedure differs from traditional inference for other models which require just single forward pass.
1.
Inference queries to language models are seeded with prompt, which is of initial tokens T1, 22..... Zp (we assume that the prompt has p tokens). The conditional distribution Pr(xp+1|11 can then be computed through forward pass. We call this the "prompt encoding" phase. Each subsequent generated token (sampled from Pr(#++1 (1:s) where i p) needs its own forward pass through the model, which we term the "token generation" phase.
3 Cost Model for Autoregressive Inference Runtime
We now seek derive cost model T(o, p; 0) for the runtime of autoregressive inference of Trans- former models (Vaswani et al. 2017). is the number of tokens in the input prompt, o is the number of generated output tokens, and 0 is of (learnt) parameters specific to particular model, software, and hardware deployment.
In this section, first specify the functional form of T based the number of floating boiting-pointt operations in the prompt encoding and token generation phases. We do not use the total number of foating-poin operations to directly estimate runtime since every floating -pinnt operation does not have the same "cost": in particular, the prompt-encoding phase is compute-boun -ound while the token-generating phase is memory- hanerr-bnndwith--bbe bound for small batch ( 2023 Yu et al. 2022). We then validate the derived expression of T with experiments. Finally, we describe a procedure estimate the 0 parameters from profiled runtimes; the profiled runtimes can be from dedicated server deployment, or from black-box APIs.
3.1 Derivation of Cost Model
To generate o tokens, 1 additional forward passes are needed (the first token is generated in the prompt encoding phase). The runtime of generating tokens given prompt with p tokens is the sum prompt_encoding encoding (ime and outputtggeeerattet time
3.1.1 Number of Floating-Point Operations
To derive expression for the end-to-end runtime of autoregressive inference, we first derive expressions for the number of floating-poin operations required for each of the two steps.
In this paper, we are interested in the smallest possible inference runtime for given query and assume inputs to the model are grouped into batches of size 1, but the batch size can be made larger in general to improve accelerator utilization and throughput at the cost of latency. Our cost model T can be adapted to accommodate other batch sizes
3 DDeeending on number of output tokens generated and the prompt size, end-to-end autoregressive inference becomes largely memory- memory-bandwiddee bandwidth bound as well; the larger the number of generated output tokens, the lower the effective throughput since computation is in the memory-bandwidee bandwidth-bouw portion of the computation for longer.
3

---

Prompt encoding, Transformer models consist of many Transformer layers, which themselves are composed of an attention layer (which measures the "importance" of input tokens to each other) and two-layer FFN in traditional formulations. As outlined in | the total number of operations that need be run the prompt encoding phase for a single prompt of size p 24ph?1 (1 + d.), where l is the number of Transformer layers in the model and h is the hidden size. p is usually & 6h, so the number of compute operations needed to encode prompts is 24ph"l. We assume that the costs of projecting into vocabulary space in the output layer of the model and sampling the next token given the distribution Pr(#:+1 (Tis)) are cheap compared to the other operators in the Transformer layer Narayanan et al. 2021
Output token generation. When using language models autoregressively to generate new text, all operators in the transformer model must be performed incrementally. Concretely, the key, query, and value transformations in the self-attention layer need to be performed for just the new token, and self-attention scores need to be computed between the new token and all previous tokens. We can compute the number of floating- point operations needed per Transformer layer to perform these computations. Let be the number of tokens generated so far (i.e., we are trying to generate the (i + 1)th token, including the prompt). The total number compute operations needed to generate the + 1)th token for single prompt is 24h2t + 4ihl 24h2/ (1 + 4 (see AA. for details). If i < 6h, which is largely true in practice (e.g., for OpenAI/davinci: the maximum context length is 12288), the number of floating-poing operations to generate new token is roughly independent of the number of tokens generated so far. When context windows become large (e.g., in excess of tens of thousands of tokens as seen in some newer models (Anthropic [023 MosaicML 2023)), the cost of generating each new token is no longer independent of the number of tokens seen so far. In this paper, we focus on smaller context windows (<10k tokens) where i < 6h.
2048 and
3.1.2 Final Parametric Form
Runtime can be expressed as the ratio of the number of floating-point operations and throughput:
(24h?l)p
prompt_encoding_tineee tiime(tttttpppe a)
app
(I)
24/²1
output output__eneration_time 8)
wwweeeratiiottrougggeeeooooeoeee -B(o-1) 1)
(2)
Here, Ap is the average runtime per prompt token (different for different prompt sizes p) and 3 is the runtime to generate each additional output token. prompt_encodig throughput initially increases as p and arithmetic intensity (Williams et al. 2009) increase; we approximate prompt encoding (pt as piecewise linear. output -eeerrenneeennneeee thrrrighttt constant independent of the token being generated; this means _errurttoneerennnn__ Litrrattiittttmeeeteeee is a linear function of o. prompt_encome throogghhhh and output_ Srrppuut_geerrtionnnoo get.erraiionthoouuhhuuuuuu can differ by an order of magnitude or more.
We break up the space of all possible prompt sizes into disjoint intervals (1), (p1,P2).-... (Pn-1, Pa]] Po is assumed be the smallest possible prompt size of O, and Pr. is assumed to be the largest possible prompt size (i.e., the maximum context window size supported by the model). For each interval (pj, P)+1], we have a corresponding parameter aj that needs to be estimated; ay is the average runtime per prompt token when the prompt size is in the interval P))))
Summing Equation I and E us expression for the end-to-end runtime
T(p,0; = (a,3) = prompt_emoooe crompt_encoding_timeee (im a) + (o- 1).3
n-1
Σ j=0
(p, < < SPpSitttoppp + (o- -1)3.
To support very large context windows, the above equation needs to be modified from the sum of a piecewise linear function of p and linear function of the sum of piecewise quadratic function of p and a quadratic function of o.
Forr simplicity. we will use prompt_ encoding time in all subsequent equations since piecewise linear functions are clunky write.

---

Runtime (seconds)
p=1024
Runtime (seconds)
p=1
p=1024
P=512
p=1536
p=512
p=1536
5
2-
4
8
16
32
48
64
Number of output tokens (o)
Number of output tokens (o)
(a) OpenAT/davinci.
(b) MS+NV/TNLG v2.
Figure 2: End-to-end runtimes for different prompt sizes (shown in legend in terms of number of tokens) and models the number of generated output tokens is varied using Megatron. We also show
a dotted best-fit line.
Runtime (seconds)
Runtime (seconds)
Runtime (seconds)
0.6
Runtime (seconds)
1.5
0.4
0.2-
Wew
1.0-
0.1
0.2
0.2
0.5
0.0
0.0
1024
1920
512 1024
1920
512 1024
1920
512 1024
1920
Number of
Number of
Number
Number of
prompt tokens (p)
prompt tokens (p)
prompt tokens (p)
prompt tokens (p)
(a) Anthropic/v4-s3.
(b) AI21/31-Jumbo v1.
OpenAI/davinci.
(d) MS+NV/TNLG v2.
Figure 3: Prompt encoding runtimes versus prompt sizes.
We also show dotted best-fit line.
Model (owner/name)
Provider
h
n
# Params (B)
# GPUsxGPU type
OpenAI/davinci
OpenAl
12288
96
96
175
8x80GB-A100
A121/31-Large v1
A121 Labs
4096
32
32
6.7
1x80GB-A100
A121/31-Grande v1
A121 Labs
5120
50
40
17
1x80GB-A100
A121/J1-Jumbo v1
A121 Labs
13824
76
96
178
8X80GB-A100
Cohere/XL v26220609
Cohere
8192
64
64
52
4x80GB-A100
Anthropic/v4-s3
Anthropic
8192
64
64
52
4x80GB-A100
MS+NV/TNLG v2
Microsoft
20480
105
128
530
24x80GB-A100
FleutherAI/GPT-J
Together
4096
28
16
6
1x80GB-A100
Yandex/YaLM
Together
10240
80
128
100
4x80GB-A100
BigScience/BLOOM
Together
14336
70
112
176
8x80GB-A100
Table 1: Models studied in this paper. also specify the number of GPUs GPU type used to estimate the default idealized runtimes. Different configurations are used with 32GB-V100 GPUs.
3.2 Empirical Validation
We can validate the above equations empirically on instantiations of state-of of-the-rrt LLMs.
Setup. We use Megatron (VIDIA)). high performance GPU implementation of Transformer models with support for autoregressive inference For given model, we used the minimum number of GPUs necessary minimize cost. For example, OpenAI/davinci cannot fit single 80-GB A 100 GPU; we use tensor model parallelism (Shoeybi et al. 2019) to ensure that the model parameters fit in GPU memory in such cases. Tensor model parallelism works well within multi-GPU server (Narayanan et al. 2021) since expensive all-to-all communication collectives like all-reduce are limited to fast bandidh NVLink. For even larger models like MS+NV/TNLG v2, we need other forms of parallelism like pipeline model parallelism in order to fit the model in GPU memory without poor scaling. We used NVIDIA HGX servers with 8 NVIDIA A SXM4 80GB GPUs; A100 GPUs were the fastest widely available GPU as of October 2022, when we did this work.
*CUDA version 11.5.0. Megatron commit hash e156d2f and fp16 precision.

---

We evaluate 10 models, ranging in size from 6 to 530 billion parameters. Evaluated models are available in different ways: some were public via a commercial API (e.g. OpenAIIdavinng AI21/J1-Jumbe v1), some were private but the model owner provided research access for this effort (Anthropic/v4-a s3, MS+NV/TNLG v2), and some were public and free (e.g., and were run using the Together API We do not evaluate models with withheld model architec- ture details (e.g., ChatGPT). Table I shows the full set of evaluated models, along with the key hyperparameters released by the respective model owner that determine their size.
Results. Figure 2 shows the end-to-end runtime measured using the above setup, versus number of generated output tokens for different prompt sizes and models. We instantiate models based on reported architectures, but with random (untrained) parameters, as we only care about estimating runtime, and runtime is independent of the model's 's parameters given a prompt size and number of output tokens. We randomly sampled 4 prompt sizes ((1,512, 512, 1024, 1536)) from the space of all possible prompt sizes and 7 different number of output tokens ((1, 8, 16, 32, 64)). Runtime was averaged over 100 prompts of the same size. For each p. we compute a best-fit line using linear regression. We observe that the coefficients of determination (JR²) for the resulting time estimates are very close to 1.0 (> 0.999) for all models and conclude runtime shows linear relationship with the number of output tokens for each prompt size (i.e., o generation time is a linear function of o).
Runtime also increases with prompt size. Figure 3 shows the prompt encoding time (i.e., runtime when the number of generated output tokens is set to 1) versus prompt size (p) for 4 models. We show prompt encoding runtimes for all p E 32, 64, 128, 192..... 1856, 1920]. Runtime and the prompt size have roughly linear relationship, especially at large prompt sizes (shown by the dotted best-fit line). At small prompt sizes, Op changes quickly; consequently, points deviate from the fest-fit line on the left side of each figure.
3.3 Estimation Procedure
Equation 3 provides expression for the end-to-end runtime of autoregressive Transformer LLMs for arbitrary prompt size p and number of generated tokens o that depends on (m, s, h) -speecif parameters (m is model, is software, h is hardware). This suggests efficient way of estimating 9 = (a, B) and consequently the runtime of a query with given prompt size and number of output tokens target system from small number of profiled runtimes.
For each model and target system, we follow two-step process. First, for each prompt size p E {po. (Po.PI.... .......PP............ }.). we profile the autoregressive Transformer L.L. with numbers of output tokens equal to to get the corresponding aj = runtime,/ /p.j. Equipped with these a, values, we leverage the fact that [aaal_eetimeepp (p o) prompt encoding_tinng __ime((tm))))) is linear function in to fit single linear regression model with y = runtime difference and æ = o - I to estimate slope 3. In practice, profile the full cross product of set of prompt sizes and number of output tokens (i.e., (p. o) E (Po, Pı.... Pn} } x for, .2...., , 0m } and use the resulting runtimes to estimate a and 33 for the target model and system.
4 Idealized and Denoised Metrics
We now propose two runtime metrics that can be efficiently estimated using the cost function T.
Idealized runtime. The idealized runtime is the runtime of an inference query assuming a particular model architecture, software and hardware implementation (e.g., NVIDIA A100 GPUs and Megatron respectively). It allows for the inference efficiency of models to be directly compared with each other.
idealized (((m)liz)) = Tisaliin (p. or (a (m..s))) ((mm)))))
(ime a (dolizz))) )) + (o- - t)sadnel )- (4)
Denoised runtime. We also propose runtime metric that factors out the noise from performance variation when using black-box, APIs. We call this the denoised runtime; it is the best-case runtime of a particular query when given access to the same software and hardware used by the API provider.
T Trdenoised
m on API a (p, og (a ou a Pm a moompt_encoding time O m on
denoised geeenoised)
demoised ) + (o - gdennised m on
(5)
https://together.aal

---

Denoised runtime (seconds)
102
20
Idealized runtime
(seconds)
10
15
10
10-
10
10"1
5
102
10
20
10
20
Raw runtime (seconds)
Denoised runtime (seconds)
(a) Denoised runtime vs. raw runtime.
(b) Idealized runtime vs. denoised runtime.
Figure 4: Denoised vs. raw runtime and idcalized vs. denoised runtime for various models across range of queries along with dotted y = x line in log scale on the left and linear scale on the right (for both Figures 4aj and 4b). OpenAI model points are shown in green, AI21 Labs points are shown in red, and points corresponding to other models are shown in black.
The runtime for generation using black-box API is similar Equation S but with an additional overhead term added to the prompt encoding and token generation runtimes. overhead captures the fixed costs of using an API to serve model predictions instead of using accelerators locally (e.g.. round-trip latency of communicating with a remote API server) and performance variability (c.g.. queuing delay or performance contention across requests). We quantify these overheads ŞB.I
Estimating runtimes models run in a black-box system introduces additional complexity: run- times using black-box APIs display higher variance relative to dedicated hardware ($B.1) due to performance contention, which can make it hard to estimate a and B. Since the variable performance overhead is random variable 7) 2 O, run multiple trials for each (p:o) pair in the profiling step and then perform linear regression using the minimum runtime (i.e., the runtime with minimum vari- able overhead across trials) for each p and o. We in BB.2 that this procedure is able to effectively factor out the performance variation observed in raw runtime measurements from black-box APIs.
Idealized dollar cost. Thus far, have only focused on runtime. However, another important consideration is cost: larger models often require more accelerators just to fit the model in accelerator memory and so comparing runtimes alone might not be fair. We propose take into account the number accelerators for serving. Both metrics are derived from the idealized runtime. Unfortunately,
metrics that explicitly
cannot similarly modify the denoised runtime since we do not know the number of chips used by the model provider. We compute the idealized dollar cost as (m.,.h) (secs) x
aaccelerator Cacselerator à ($/sec).
Vaccelerator is the number of accelerators used a time to serve a single request (1 if not using model parallelism, > 1 otherwise), and Caccelerator is the per-unit-time cost of the hardware h (e.g.. if is NVIDIA A 1100 GPUs, then Caccelerator could then the per-hour cost of renting an NVIDIA A100 GPU on AWS and the idealized dollar cost is the cost of serving the model on AWS A100 GPUs).
Idealized energy cost. Similar work that has examined the energy cost of training (Henderson et al. 2020 Strubell et al. 2019 Patterson et al. 202T), we can estimate the idealized energy cost pideitizdd (secs) X Pacceleratorr à X Paccelerator h (W). Paccelerator is the power draw of hardware h. We ((m,,,
can compare the idealized energy cost of running an inference query the energy cost of training a model to better understand the number of inference queries needed to amortize training costs.
5 Results
In this section, we seek to answer the following: (a) Is the proposed methodology to estimate inference runtime of autoregressive Transformer models accurate and efficient compared exhaustive profiling? (b) Can this method reveal interesting insights about models'
tradeoffs?
5.1 Accuracy of Runtime Estimation Procedure
We computed the coefficients of determination for runtimes of queries passed through black-box APIs for the models in Table Despite performance variance, we see that the estimated runtimes using the methodology based on linear regression are fairly accurate (R² 0.9), giving us further confidence that our cost model for runtime is accurate (full results in Table 2 in B.2).
7

---

Figure 4a compares denoised runtimes to raw runtimes for range of prompt sizes and number of generated output tokens. Experiments were conducted in September and October 2022. We observe that raw runtimes for the most part (96.6% of points) are greater than the estimated denoised runtimes (below the y = x dotted line). Figure 4b is similar, but shows idealized runtime with A A100 GPUs and NVIDIA's Megatron (Shoeybi et al. 2019) versus denoised runtime (same hardware and software setup as described in (3.2). In a number of cases, the idealized runtime is much lower than the denoised runtime, since the relevant API uses slower hardware and/or software implementations and other features like batching that sacrifice higher throughput for lower latency. For A121 Labs models, idealized runtimes are greater than denoised runtimes 15.7% of the time. For OpenAI models, idealized runtimes are greater than denoised runtimes 64.2% of the time suggesting that they are using better hardware and / or more optimized software stack. For all other model providers, idcalized runtimes are always lower than the denoised runtimes, indicating that our hardware and software stack assumptions fairly accurate.
5.2 Efficiently Evaluating Other Hardware or Software Optimizations
We can use the cost model proposed in this paper to efficiently evaluate the effectiveness of other hardware accelerators such as TPUs (Jouppi et al. 2017) or other generations of GPUs. As an example. Figure II in $B.4sshows how runtime and cost for various models is affected by using older NVIDIA VT00 GPUs instead of NVIDIA A100 GPUs (the default hardware accelerator used in this paper). While we expect these GPUs to be slower, we can also reasonably expect them to be cheaper (due to lower per-hour costs (Amazon)). In practice, we find that this is not the case, suggesting V100 GPUs are slower and more expensive. This is partially because we often have to use double the GPUs fit the model parameters GPU memory, since V100 GPUs only have 32GB of device memory compared to 80GB on the A100 GPUs. This analysis requires profiling on the order of hours (< 2 hours for most models, depending on the number of (p, o) values profiled) once, compared to hours per benchmark (depending on number of queries in the benchmark) for exhaustive profiling. As we show in |B.3 running a few calibration queries and then fitting the cost model to obtain (m, s, -speciici parameters is much more efficient than exhaustive profiling. Our cost model also allows us to evaluate different software implementations and optimizations such as FasterTransformee (NVIDIA) or Flash- Decoding (Dao et al. 2023).
5.3 Elficiency-Capabblly Tradeoff's
We can now use the metrics proposed in this paper to evaluate the efficiency-capabiiity tradeoffs of various language models accessible through black-box APIs. We focus the few-shot evaluation setting adopted by BIG-Bench (Srivastava et al.) 2023) and HELM (Liang et al. 2023). We consider four tasks in HELM: sentiment analysis task (IMDB), two question answering tasks (MMLU [college chemistry] (Hendrycks et al. 2020) and BoolQ CClark et al. 2019)), and classification task (RAFT [terms of service] (Alex et al. 2021)). ). Figure presents the results, with each row of graphs comparing average accuracy to a different efficiency metric. We highlight a few takeaways.
5
Effect of model scale. We observe that only a subset of the evaluated models fall Pareto frontier, with different models on the Pareto frontier for different tasks. This suggests that model scale alone does not predict model capabilities.
Heterogeneous software/ hardware. The OpenAI/davinci model appears the Pareto frontier for each benchmark when using raw runtimes but not the idealized metrics. This suggests that the OpenAl API implementation is more optimized than others: this could be due to a number of systems optimizations, such as query caching, even model compression techniques like distillation, quantization or sparsification (Polino et al. 2018). Comparing these models on level footing (same software and hardware) requires metrics that can factor out the effect of performance optimizations orthogonal to the model, such as idealized runtime.
Model architecture design. The relative positions of BigScience/BLLOM and Yandex/YaLM on the idealized cost and floating-point operations (+ model size) graphs are sometimes reversed: achieves cheaper idealized cost (which takes into account the lower
while SigScience/BLOO number of GPUs that Yandex/YaLM requires), Yandex/YaLM uses fewer floating-poii operations. BigScience//LLOM's improved performance can be at least partially attributed to more thorough
8

---

OpenA//davinii
A121/J1-Jumbo
Microsoft-NV/TNGG v2
Yandex/YALL BigScience/BL.Ol BLOMM
AILa/J1-Laage v1
Cohere/XL v20220609
EleutherAl/GPTJ
A121/J1-Gramm v1
Anthropic/v4-s3
0.8
Accuraty
0.8
Accuracy
0.3
Accuracy
0.6
Accuracy
0.7
0.4
0.6-
0.6
0.2
0.2
10
10
10
10"
10"
10'
10"
Model size
Model size
Model size
Model size
(billion parameters)
(billion parameters)
(billion parameters)
(billion parameters)
0.8-
0.8
Accuracy
0.8
Accuracy
0.3
Accuracy
0.6
Accuracy
0.7
0.6
0.2
0.2
0.5
10
10
10
10
10
1015
1015
10tH
1011
10
10'
10'5
Number of floating-
Number of floating-
Number of floating-
Number floating:
point operations
point operations
point operations
point operations
0.8
0.8
Accuracy
0.8
Accuracy
0.3
Acouracy
0.6
Accuracy
0.7
0.4
0.6
0.6
0.2
0.5
10
10
10
10
100
10'
10-1
10
10'
10"
100
10'
Raw runtime (secs)
Raw runtime (secs)
Raw runtime (secs)
Raw runtime (secs)
0.8-
0.8
Accuracy
0.8
Accuracy
0.3
Accuracy
0.6-
Accuracy
0.7
0.4
0.6
0.2
02
0.5
10'
10
10"
10
10
10
101
10"
10"
10"
Denaised runtime (secs)
Denoised runtime (secs)
Denoised runtime (secs)
Denoised runtime (secs)
0.8-
Accuracy
0.8
Accuracy
0.3
Accuracy
0.6-
Accuracy
0.7
0.4
0.6
0.6
0.2
0.2
10
10
10
10"
10
'00
10
Idealized runtime (secs)
Idealized runtime (secs)
Idealized runtime (secs)
Idealized runtime (secs)
0.8
0.8
Accuracy
0.8
Accuracy
0.3
Accuracy
0.6
Accuracy
0.7
0.4
0.6
0.6
0.5
10
10
10
10
10"1"
10"
10
10
10"
10
10t1
10
Idealized cost (centa)
Idealized cost (cents)
Idealized cost (cents)
Idealized (cents)
(a) IMDB.
(b) MMLU, chemistry.
(c) RAFT, terms service.
(d) BoolQ.
Figure 5: Capability vs. efficiency tradeoff graphs. Capability is shown as accuracy on the target task. Six efficiency metrics are shown: model size (billions parameters), per-query number of floating-point operations (FLOPs), raw runtime, denoised runtime, idealized runtime (all in seconds), and idealized cost (in cents). Idealized metries were estimated on the hardware and software setup described in $3.2 Metrics are averaged over all scenario instances. Models on the Pareto efficiency frontier are shown as squares with a black dotted line connecting the points.
search through model architectures for minimum runtime with given number of floating-pont operations in the forward pass (Scao et al. 2022).
Run-to-run variance. A121/31-Grande v1 often achieves worse raw runtime than AI21/J1 -umbo v1 despite having 10x fewer parameters, since the Grande model experiences higher performance variance (Figure 7) The idcalized metrics make it casier to the true efficiency-ccpabiliyy tradeoffs.

---

The denoised runtime metric can also help an end user reason through whether the observed inference performance an artifact of performance contention.
Cost comparison. We can also compare these estimated inference costs to the costs charged by the black-box API provider. We observe that they are up to order of magnitude lower than the charged actual costs, However, we note that these reported costs do not incorporate the significant cost of training models, which presumably gets amortized into the cost users pay with black-box APIs.
6 Related Work
A large body of work has looked studying the impact of model scale on model capabilities. Scaling laws and other benchmarking efforts. By fitting a curve to dozens of training runs, scaling laws (Kaplan et al. 2020) show how the size of a model in model family affects the training and validation loss of these models. While these scaling laws are instructive, we also care about the capabilities of models along other axes beyond validation loss (e.g., are models robust; do they exhibit stereotypes?). Moreover, large language models have been shown to exhibit emergent behavior that cannot easily be expressed as continuous function of scale (Wei et al. (022). Similarly, even though model size is used as proxy for training and inference runtime performance, it is not useful when trying to answer questions like "Can model X meet latency SLO of 100 milliseconds?". Consequently, empirical analysis is still needed to understand the capabilities of these models.
Floating-poin operations and other proxy metrics for efficiency. The number of floating- point operations (FLOPs) required for the forward pass of model has also often been used to estimate inference efficiency. While this is fine approximation, it is not ideal for couple different reasons. First, runtime does not correlate exactly with the number of FLOPs required (Scao et al., 2022). In particular, two operators with the same number of FLOPs could be executed with different efficiencies if one of the operators involves more memory accesses, preventing execution at peak device throughput. Second, as with model size, the number of FLOPs is hard to interpret. LLMs are often part of larger applications, and the performance requirements of these applications impose runtime constraints on LLM inference. It is hard to translate FLOPs to something actionable.
Inference runtime estimation for other types of models. Typically, inference for ML models is straightforward: an input of a particular size is passed through the model, in the process generating intermediate outputs and eventually a final prediction from a single forward pass. The sizes of intermediate outputs do not change from input to input, resulting in negligible runtime variance. This consequently makes inference runtime estimation easy. However, LLMs are different: while the hidden size does not change from input to input, the prompt size (in number of tokens) can be different for various inputs; runtime also depends on the number of output tokens generated.
Carbon costs of ML computation. Many papers (Canziani et al. 2016 Cao et al. 20200] Henderson al., 2020, Strubell et al.] 2019; Bender al., 2021 Patterson et al.) (021) have discussed the
importance of quantifying the cost of training models, both from an energy and emitted CO2 perspective. This is often possible because model providers are open about details on training necessary to compute these metrics (Black et al., [2022] Patterson et al. 2021). While recent work has emphasized the need for considering inference-tima efficiency (Henderson et al. 2020 [oomassnni et al. 2021 $5.3), information on inference-time costs of LLMs is more scant for multitude of reasons (e.g optimizations powering an API might be part of a company's 's competitive advantage).
7 Conclusion
This work presents a systematic study of inference efficiency for autoregressive Transformer models accessible through black-box APIs. We showed both analytically and empirically that the inference runtime for these models is the sum of a piecewise linear function of the prompt size and linear function of the number of output tokens, and designed an idealized runtime metric that can be estimated efficiently with minimal extra profiling- We are hopeful that our work helps model creators make better informed decisions about long-term model investments spanning training and serving.
10

---

Acknowledgements
We thank the anonymous reviewers, Trevor Gale, Siddharth Karamcheti and Matei Zaharia for their help and feedback that improved this work. This work was supported in part by the A12050 program at Schmidt Futures (Grant G-22-63429). This work builds on top of the Stanford HELM project; are grateful to its many contributors. We also thank the following individuals at their respective organizations for the access, support, and/or credits required to evaluate their models:
A121 Labs: Opher Lieber, Barak Lenz, Dan Padnos, Yoav Shoham.
Anthropic: Ben Mann, Jackson Kernion, Deep Ganguli, Jack Clark, Dario Amodei.
Cohere: Lewis Stott, Ellie Evans, Bill McCartney, Aidan Gomez.
Microsoft and the Turing Academic Program: Payal Bajaj. Barun Patra, Ahmed H. Awadal- lah, Saurabh Tiwary, Eric Horvitz.
OpenAl: Lama Ahmad, Miles Brundage.
References
A121. A121 Models API. https://www.aill.com/stttuu
Alex, N., Lifland, E., Tunstall, L.,, Thakur, A., Maham, P., Riedel, C. J., Hine, E., Ashurst, C., Sedille, P. Carlier, A., al. RAFT: A Real-World Few-Shot Text Classification Benchmark, arXiv preprint arXiv:2109.14076, 2021.
Amazon. AWS Pricing for GPU Instances. 2365. 0005445minngpu--cccopeeeee///e/
Anthropic. Claude httrs://www.attrrro...ccc/////er
2023.
Bender, E. M., Gebru, T., McMillan-Major A., and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et PiQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L.,, He, H., Leahy, C., McDonell, K., Phang, J., Michael, P., Prashanth, U, S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. Proceedings of BigScience Episode #5- Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosselut, A., Brunskill, E., Brynjolfsson, E., et al. On the Opportunities and Risks of Foundation Models. arXiv preprint arXiv: 108.::208..02588, 2021.
Bommasani, R., Klyman, K., K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., and Liang, P. The Foundation Model Transparency Index. arXiv preprint 10. 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam. P., Sastry, G., Askell, A., et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
Canziani, A., Paszke, A., and Culurciello, E. An Analysis of Deep Neural Network Models for Practical Applications. arXiv preprint 1605 1605.0076 00678, 2016.
Cao, Q., Balasubramanian, A., and Balasubramanian, N. Towards Accurate and Reliable Energy Measurement of NL.P Models. Proceedings of SustaiNI.P: Workshop on Simple and Efficient Natural Language Processing, 2020.

---

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint ar arXiv:2204.02. 02311,,,..,,, 2022.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems. arXiv preprint 1rXiv:2110.14168 2021.
Cohere. Cohere Models API. hntpss:/coherr..a//gene/
Cursor. Why GPT-3.5 is (Mostly) Cheaper than Llama 2.
https://cursor.s//blog
Ulama- inference 2023.
Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash Decoding for Long- CongCCtneex Inference. https html 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre- Tre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., and Pineau, Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning. Journal of Machine Learning Research, 21 21(248):1-43, 2020.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song. D., and Steinhardt, J. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009. 2000:000000000000, 2020.
Jouppi, N. P., Young, C.,, Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, 2017.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Rad- ford, A., Wu, J., and Amodei, D. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361 2020.
Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks Are All You Need II: 1.5 Technical Report. arXiv preprint S1ti::230.004665. 2023.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., et al. Holistic Evaluation of Language Models. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
Lieber, O., Sharir, O., Lentz, B., and Shoham, Y. Jurassic-1: Technical Details and Evaluation. 2021.
Microsoft. Confirmed the new Bing runs on OpenAl's GPT-4, httpps::/blogs..bnngg htwp:///...bbiii..ooooo//oeerere march 2023/Confirmed the- new- Bing- runs on www.ooonAAA22888999s----o- GPT-4
MosaicML. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs.
httts://www.mosaicch// htt.:/aaaa..aii.ooooooooooto///p 2023.
Narayanan, D.,, Shoeybi, M., Casper. LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D.. Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient Large-Scale Language Model Training on GPU Clusters using Mcgatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.
NVIDIA. NVIDIA Faster Transffoooo.r.rr httppp///dihhaacccoo///////t/ttt
12

---

NVIDIA. Megatron-LM Repository. https:///gthbb.occooo//w/aaaaatt Lm
OpenAl. OpenAl Models API. https://bata..eennaa..cooooooooo
OpenAI. GPT-4 Technical Report. arXiv preprint (rti:/m30rr.::223300077777 20233.
Patterson, D., Gonzalez, J., Le. Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., and J. Carbon Emissions and Large Neural Network Training. arXiv preprint arXiv:21060 10350, 2021.
Polino, A., Pascanu, R., and Alistarh, D. Model Compression via Distillation and Quantization. arXiv 2018.
preprint
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, Henderson, S., Ring, R., Young, S., et al. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv preprint arXiv:2112. 2021.
Reuters. ChatGPT sets Record for Fastest-Growing User Base. https://www.reuterrs.co rechnoooogyychaatt--eees----o-oo fastesor--fstttt--rrroooo hrg.s///.oooooo...ooo.////a//a/a
Reuters. ChatGPT Trafiic Slips Again for Third Month in Row. https::/wwwwrretterrsccom
"eechrology/chattttttrratt traftaatttttttttff--taaoa-atit slips aaais-agai--- third month row- 20w.
Scale VP.
Scale Generative AI Index.
http:://ww..scaeev.ccom///
introducing the scale-generative ai-ix
Scao, T. L., Wang. T., Hesslow, D., Saulnier, L., Bekman, S., Bari, M. S., Biderman, S., Elsahar,
H., Phang. J., Press, O., Raffel, C., Sanh, V., Shen, S., Sutawika, L.. Tae, J., Yong. Z. X., Launay, J.. and Beltagy, I. What Language Model Train if You Have One Million GPU Hours? In Challenges & Perspectives in Creating Large Language Models, 2022.
Shazeer, N. Fast Transformer Decoding: One Write-Head is All You Need. arXiv preprint arXiv:1911.02150 2019.
Shoeybi, M.. Patwary, M. Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM. Training Multi-Billion Parameter Language Models using Model Parallelism. arXiv preprini arXiv:1909.0808: 08050000000....... 2019.
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. Using DeepSpeed and Megatron to train Negatron-Turing NLG 530B, Large-Scale Generative Language Model. arXiv preprint arXiv:2201. 1990, 2022.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, R., Santoro, A., Gupta, A., Garriga-Alonso A., et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Rexeurch, 2023.
Strubell, E., Ganesh, A., and McCallum, A. Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is All You Need. Advances Neural Information Processing Systems, 30, 2017.
Wei. J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D.,, Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, Emergent Abilities of Large Language Models. Transactions Machine Learning Research, 2022.
Williams, S.. Waterman, A., and Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(())65-76, 2009.
Yu, G.-I,, Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A Distributed Serving System for Transformer- Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
13

---

Zellers, R., Holtzman, A.. Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking Large Language models for News Summarization, arXiv preprint arXiv:2301.13848, 2023.
14

---

Appendices
distribution over
Embedding (V .1)
The
Attention
FFN (4h A)
brown fox
FFN (
Output
jumps
Transformer layer xi
Figure 6: High-level schematic Transformer model with Transformer layers generating text at inference time given prompt "The brown fox jumps".
A Derivation for Number of Floating-PPoin Operations
This paper focuses on language applications for Transformer models, where the model input text. The input text is first preprocessed into sequence of tokens (c.g., words) through process called tokenization. Feature representations for each token (obtained by passing one-hot representations of the tokens through an embedding layer) are passed through multiple Transformer layers. Inputs to each Transformer layer are typically 3-dimensional tensors of shape (b, 8, h) where is the microbatch size (number of sequences), is the sequence length (number of tokens in each sequence), and is the hidden size (dimensionality of the model). For simplicity, we denote inputs as X.
Transformer layers in language models use self-attention to allow tokens to "interact" with each other. We assume multi-head attention; other forms of attention like multi-query attention (Shazeer 2019)
slightly change the analysis.
Self-attention is composed of the following operations:
Attention key (K), value (V), query (Q) transformations. Given input X, we perform matrix multiplications K X W X W and Q Wº WKK WV and W are learned parameters.
Attention score computation. Matrix multiplication Q x KT followed by application of the softmax function obtain score tensor Z. Each element Zij is an importance score between query token and key token j. This is the primary mechanism that allows interaction across tokens in sequence.
Attention over value computation. Matrix multiplication of scores Z by values V.
The subsequent two-layer feed forward network (FFN) consists of two linear layers (implemented as matrix multiplications). For most models, this involves multiplying the output the self-attention layer by matrix with dimension h 4h and then multiplying the resulting output (after other operators like layer norm) by matrix with dimension 4h x h. F how these operators connected in typical "decoder-only" Transformer model.
We can now derive an expression for the number of floating-point operations in typical forward pass through the model, using similar form of analysis as prior work (Narayanan et al. 2021
A.1 Training
We use the same notation as before: is the microbatch size (number of sequences) and h the hidden size of the model. In practice, the self-attention layer computation described in ŞA performed with different parameter matrices W.K. W.Y. and W.O. This is called running the self-attention layer with multiple attention heads. We assume that the Transformer model has n attention heads.
is the sequence length in terms of number of tokens. Inputs X to the Transformer layer have shape (b, s, h). The Transformer layer's computation during training can then reduced to the following matrix multiplication operations.
15

---

Attention key, value, query transformations: These can be expressed as a single matrix multiplication of size: (bs, h) x (h, 3h). Output is of size (bs, 3h).
Attention score computation: bn batched matrix multiplications (BMMs), each of size (s,) x (h/n, 8). Output is of size (bn, 8).
Attention over value computation: bn batched matrix multiplications of size (s, x h/n). Output is of size (bn, s, h/n).
Post-attenticn linear projection: single matrix multiplication of size (bs, h) x (h, h) to coalesce outputs of attention heads to a single per-sequence vector size h. Output is of total size (bs, h)-
Matrix multiplications in the MLP layer of size (bs, h)) x (h,, 4h) and (bs, 4h) x (4h, h.. Outputs are of size (bs, 4h) and (bs, ht).
Using the fact that (m. n) x (n, k) matrix multiplication needs 2mnk floating foating-point operations, the total number of compute operations is complete the forward pass through Transformer layer during training is 24bsh2 + 6h )- A Transformer model typically has / Transffrmer layers, resulting total of 24bsh?t 2 (1 ) floating-pon point operations in single forward pass through the model. For most LLMs, < 6h, meaning our expression for the number of floating-poin operations in single training forward pass can be simplified to 2.
A.2 Autoregressive Inference
We can similarly compute the number of floating- peiiin operations needed to generate a single output token during autoregressive inference. i is the number of tokens generated so far (i.e., the (i + 1)th token, including the prompt, needs to generated next). The operators to be run in each Transformer layer in this phase are:
Attention key (K), value (V), query (Q) transformations: These can be expressed as single matrix multiplication of size (b, h) (h, 3h.
Attention score computation: br batched matrix multiplications (BMMs), each of size (1, h/n) x (h/n,) i) (only Q value for the latest token used; and values accumulated over all tokens so far).
Attention over value computation: bn batched matrix multiplication of size (1, x (i, h/n).
Post-attention linear projection: a single matrix multiplication of size (b, h) x (h, h).
Matrix multiplications in the MLP layer of size (b, h) x (h, 4h) and (b, 4h) x (4h, h).
Consequently, the total number of compute operations needed to generate the (i + 1)th token is 24bb²l + 4bihl = 24b/2t (1 + 6k
16

---

B Additional Results
In this section, we show additional results. $B.1 we quantify the performance variation of using black-box APIs; in $B.2) provide additional evidence that the estimation procedure described in $3.3 is accurate; in $B.33 we quantitatively show the cost benefits of our procedure compared to exhaustive profiling; and in $B.4, show case study of the type of quick analysis that this procedure can enable.
B.1 Performance Variation in Black-Boo APIs
We first measure the performance variation seen while using black-box text generation APIs.
1
1
Number of
output butputttoeet tokens
2-
Number of
output butputtokee tokens
4-
16-
16
32-
32:
64
64
0.0
1.0
1.5
Runtime (seconds)
Runtime (seconds)
(a) A12g/31-Large v1.
(b) A121/31-Grande v1.
1
output tokens
Number of
16-
32
64
2
Runtime (seconds)
(c) AI21/J1-Jumbo vl.
Figure 7: Per-instance runtimes using black-box APIs to access L.MMs for multiple instances (prompt size, p = 512).
Runtime (seconds)
Runtime (seconds)
1
p=1024
3
p
1=1024
+
P1512
p=1536
p=512
p=1536
2
64
16
64
Number of output tokens (o)
Number output tokens (o)
(a) A12g/31-Large v1.
(b) A121/31-Grande vl.
Runtime (seconds)
10.0
Runtime (seconds)
p=1
p=10
7.5-
5=512
D= 1536
p=1
1024
5.0
515
p=1 p-1536 536
2
0.0
1
4
8
16
32
48
64
14
8
16
32
48
64
Number of output tokens (o)
Number of output tokens (o)
(c) Cohere/XL v20220609.
(d) Anthropic/v4-s3.
Figure 8: Median (dotted line) and min / max (lower and upper boundaries of shaded region) end-to- end runtimes across 5 trials for different prompt sizes (shown in legend in terms of number of tokens) as the number of generated output tokens is varied using black-box APIs. End-to-end runtimes show variation across the 5 trials for a given prompt size and number of output tokens.
17

---

Runtime (seconds)
7.5
+ 64
5.0
2.5
0.0
2
2
2 2
2
2
Number of parallel requests
Figure 9: Minimum runtime across 10 trials as number of parallel queries increases for the Anthropic/v4-s3 model. Prompt size is 512 tokens and the number of output tokens is varied (shown in legend). Experiment was run in 10/2022.
Variation of runtimes across trials. To better quantify performance variability when using black-box APIs, we run multiple trials of synthetic queries where we control the size of the prompt and the number of generated output tokens. Figure and Figure g shows per-trial runtimes for different model offerings; Figure 7 shows runtimes for models from the same model provider (A121). Unless otherwise noted, all experiments in this paper were run in September or October 2022 with the latest API versions available at the time.
We see discernible performance variance across multiple trials for different models, across prompt sizes and number of generated output tokens. Certain models experience higher performance variability: Figure 7 shows A12a/e1-Grande
has much higher performance variance than AI21/31-Large v1 or AI21/J1-Jumbo v1 (larger spread among points for a query of given size). A121/31-Grande has an average coefficient of variation of about 0.55 compared to much smaller coefficients of variation (~0.2) for the other A121 models. Even for models with lower spreads (e.g.. A121/31-Large v1), we see that outlier points can have as much as 3 x higher runtime.
Figure g also interestingly shows that this performance variance can obfuscate the linear relationship between model runtime and number of output tokens.
Variation runtimes with load. To understand the impact of load on performance contention and increase the number of queries sent in parallel to the various black-box APIs. Figure 9 shows runtime versus number of parallel queries for different numbers of output tokens and fixed prompt size of 512 tokens for the Anthropic cwtrroppccsssss///ee s3 model. We observe as much as a 2x increase runtime, indicating that load can lead to increased contention on API servers and consequently increased observed runtime.
end-to-end runtime, measured query runtime
18

---

B.2 Accuracy of Runtime Estimation Procedure
Model (owner/name)
R
OpenAI/davinci
0.976
A121/31-Large v1
0.987
A121/31-Grande v1
0.916
A121/31-Jumbo v1
0.995
Cohere/XL v20220609
0.995
Anthropic/v4-s3
0.997
Table 2: Models and coefficient of determination (R²) of time estimates for end-to-end text generation for various models using black-box APIs.
Runtime (seconds)
Runtime (seconds)
1.5
1.0
0.5
p-1
p-1024
p=1
p=1024
P-512
p- 1536
p=512
p=1 536
0.0
8
16
32
48
64
16
32
48
64
Number of output tokens (o)
Number of output tokens (o)
(a) AI21/J1-La v1.
(b) A121/31-Grande v1.
Runtime (seconds)
Runtime (seconds)
p=1
p=1024
?
p=512
p=1536
2
2-
p=1
p=1024
P=512
p=1536
16
48
64
8
32
Number output tokens
Number of output tokens (o)
(c) A121/31-Jumbo v1.
(d) Cohere/XL v28229609.
Figure 10: Minimum end-to-end runtimes for different prompt sizes (shown in legend in terms of number of tokens) as the number of generated output tokens is varied using black-box APIs, along with best line estimated using the linear regression method described in $3.3 of this paper.
Model (owner/name)
Metric
prompt encoding time (p = (012/0024/1530)
Per-output-token generation time (i)
OpenAI/davinci
idcalized
0.178 / 0.323 0.476
0.081
Megatron, A100)
jdenosedd
0.045 / 0.033 0.142
0.030
A121/31-Grande
idealized m. Megatron, A100) demoised
0.097 0.190 0.298
0.038
0.172 / 0.351 00511
0.021
A121/J1-Jumbo vl
idallized
0.164 / 0.310 0.465
0.064
m. Megatron, A100)
denoised
0.268 / 0.463 0.655
0.042
Anthropic/v4-s3
idealized
0.108 / 0.189 0.279
0.054
Megatron, A000)
denoised
0.193 / 0.191 0.380
0.057
Table 3: Models and estimated prompt encoding times per-output-token generation times (in seconds) for pidealized
(m, Megatron, A100) and ydenoised,
computed the coefficients of determination for runtimes of queries passed through black-box APIs; Table and Figure 10 show the results. Despite performance variance, we that the estimated
19

---

runtimes using the methodology based on linear regression are fairly accurate (R² > 0.9), giving us further confidence that our cost model is accurate.
Table 3 compares the learnt performance parameters for pddcllized (m,, Megatron, A100) and ydenoised for subset of "m
the considered models. As noted above, the estimated "(Megatron, A100) idealized" parameters for the A121 Labs and OpenAl models are higher than the estimated denoised parameters, indicating that both these providers use optimizations not present in the software stack we considered.
B.3 Comparison to Exhaustive Profiling
We model the exhaustive profiling costs for computing idealized runtime and denoised runtime separately. For the idealized runtime, we can compare the cost of exhaustively running all queries in the local environment with estimating and then using the cost model (the approach proposed $3.3). We run each query multiple times to get reliable runtime measurement. In both cases, we need to run all queries through the API once order to get the prompt size and number of generated tokens for each query.
We the following notation:
t: Number of trials.
Q: Set of user queries.
Q: Set of calibration queries (IQ) < IQ).
(x): Cost of invoking API on query
Gocal Cost of invoking local model on query q-
This produces the following expression:
t. EnEE Ooccal (4) + EneQ CAPI(4)
Idealized runtime savings t. EyeEQ Oucai (4') + EGEQ CAPI(g)
We can compute the savings concretely on the 4 HELM tasks evaluated in $5.3 We use the published 8-A 100 server
per-token costs from OpenAl for OpenAI IpenAI/davinci davinci and the per-hour cost from AWS (Amazon). With 50 trials per query (in practice, fewer trials are probably sufficient), we observe cost savings of 57x when using cost model compared to exhaustive execution.
We can derive a similar expression to estimate the savings from using our approach for the denoised runtime, with the main difference being calibration queries now need to run through the black-box API, instead on dedicated hardware.
EgEQ CAPi(q)
Denoised runtime savings
En εQ (AP((a) ΣnEQ CAPi(q)
Using the same parameters as above, we get a savings nearly 50x using the cost model compared to exhaustive execution (close to since Q << IQ).
20

---

B.4
Case Study: Using the Cost Model to Compare Different Hardware
OpenAl/davincl
A121/J1-Jumbo
Microsoft-NV/TNNG
Yandex/YALM
A121/J1-Large v1
Cohere/XL V20220609 Anthropic/v4-s3
EleutherAl/GPT-
BigSclence/BLOOM
AI21/J1-Graa (i//nn-rr
Idealized runtime,
Idealized runtime.
V100 V100 (secs)
1.0
V100 (secs)
0.0
1.0
8.00
0.1
0.2
0.3
Idealized runtime, A100 (secs)
Idealized runtime, A100 (secs)
Idealized cost,
V1000 (cents)
Idealized cost,
v100 (cents)
1.0
0.2
0.5
0.00
0.25
0.50
0.75
08.00
0.05
0.10
0.15
Idealized cost, A100 (cents)
Idealized cost, A100 (cents)
(a) IMDB,
(b) MMLU, chemistry.
Idealized runtime,
Idealized runtime.
1.0
V100 (secs)
V100 (secs)
0.5
0.0
0.5
1.0
1.5
0.00
0.75 Idealized runtime, A100 (secs)
0.25
0.50
Idealized runtime, A100 (secs)
Idealized cost,
V100 (cents)
1.0
Idealized cost,
V100 (cents)
1.0
0.5
0.5
0.00
0.0
0.2
0.4 0.6
0.2
0.4
0.6
Idealized cost, A100 (cents)
Idealized cost, A100 (cents)
(c) RAFT, terms of service.
(d) BoolQ.
Figure Comparison of idealized metrics estimated on different hardware.
Figure I1 shows a comparison between Megatron on NVIDIA A100 GPUs (the default configuration in this paper) to Megatron on NVIDIA V100 GPUs (an older generation of GPU). While we expect these GPUs be slower, we can also reasonably expect them be cheaper (due lower per-hour costs (Amazon)). In practice, we find that this is not the case, suggesting V100 GPUs are slower and more expensive. This is partially because we often have to use double the GPUs to fit the model parameters in GPU memory, since V100 GPUs only have 32GB of device memory compared 80GB on the A100 GPUs. This an example an analysis that we can quickly perform by just running calibration queries on V100 GPUs and fitting linear regression models to the resulting profiled runtimes.
21


Reviews:
[{"review_id": "aFPY0cs3R6", "reviewer": "Reviewer_Jz6a", "summary": "This paper presents a new metric that attempts to more fairly evaluate machine learning model inference runtime performance cost, which the authors refer to as idealized runtime. The authors argue that the black-box nature of many large language models (LLMs) does not necessarily provide accurate estimates of inference cost using raw runtime results alone, due to several external factors (e.g., hardware used, compute resource contention, etc.). Thus, a goal of idealized runtime is to create a fair(er) inference evaluation metric for LLMs.\n\nThe authors analyze 10 state-of-the-art LLMs using their idealized runtime metric and discuss the efficiency tradeoffs between them.\n", "questions": "None.", "limitations": "The authors seem to only focus on inference runtime overhead associated with GPUs (and a single class of GPU, as a reference point, Nvidia A100) and no other form of hardware compute. However, many other types of hardware are used to perform inference. Some of them are: CPUs, FPGAs, TPUs, CGRAs, etc. Moreover, each of these hardware classes have subclasses within that result in different temporal overhead and power footprint costs. None of these factors seem to be considered in the authors runtime equation.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- The paper is well-written\n- The research topic of creating a somewhat universal metric to measure spatial and temporal overhead of LLMs on inference is relevant and timely\n- The authors’ intuitions on focusing on compute operations (e.g., floating point instructions) and performing an inference runtime calculation that is not overtly computationally intense, is a good foundations for such research\n\n", "weaknesses": "While I agree with the authors that this topic is certainly worth deeply investigating, I have too many open concerns to recommend it for acceptance (at this time). Below I list two areas that I believe would need to be addressed before can be reasonably considered for publication.\n\n- floating point operation overhead (a critical paper weakness): in Section 3.1.1 the authors discuss some of the concrete details used for their idealized runtime equation. Of note is the focus on deriving expressions “for the number of floating-point operations required for each of the two steps, and then use these to derive expressions for runtime.” While the number of floating point operations required to perform a single forward pass of inference is quite likely to an important datum for estimating inference runtime overhead, it is (in my opinion) just the first of many (perhaps dozens?) of factors that may be required to make an accurate estimation of inference overhead. There are several reasons for this, I list a couple below (the below list is not exhaustive). \n\n(1) Not all floating point operations are spatially or temporally equivalent (e.g., a floating point ceiling or floor operation is notably less expensive than even a mathematical divide operation due to the execution “trap” that must exist for the divide, but not for the floor/ceil). Moreover, some floating point operations are orders of magnitude different in overheads (e.g., an FP divide is likely 10x-100x faster than a gather-scatter vector FP operation). To attempt to quantify all FP ops in a single computational class is logically unsound.\n\n(2) The spatial and computational complexity of FP operations between different hardware compute classes varies significantly (often times more than 10x-100x). For example, GPUs can handle up to their channel size limit of FP data and continue to emit a fixed computation time for equivalent class FP operations. This is due to GPUs inherent SIMD properties. CPUs, on the other hand, emit approximately linearly increasing FP operation overhead as FP operation count increases. On the other hand, if the GPU channel size is exceeded even by a single datum, the GPU execution time may increase by 2x or more because it will generally require a second issuance of GPU block execution. Yet, CPUs only suffer roughly additive execution overhead when additional datum are added for FP operations. This means there is generally an inversely proportional runtime cost associated with FP operations and data between CPUs and GPUs. The paper seems to only focus on GPUs, which, in commercial practice, tends not to be representative (or at a minimum highly specialized) of which compute is used for inference.\n\n- References: of the 50 or so references included by the authors, it appears that only around 10 of them are from peer-reviewed publications. The other 40 or so are from arxiv and websites. Moreover, the arxiv references are not necessarily recent (some date back over 5 years ago). The lack of peer-reviewed citation raises questions about the validity of substantiation as well as the authors’ understanding of the domain. A core problem with citing non-peer-reviewed published work is the lack of a fixed written artifact. Arxiv papers can have many versions as can website/webpages (which version are the authors referring to and how would a reviewer know this a priori?). I strongly encourage the authors to perform a more rigorous literature review and to substantiate their claims principally through peer-reviewed scientific research. It is a tall order to expect a reviewer to personally review / validate every non-peer-reviewed research paper to help verify its soundness and thus its appropriateness when used to substantiate the authors claims. Yet, not doing so would seem to imply that reviewers would need to implicitly “trust” the quality of such non-peer-reviewed artifacts. This seems antithetical to the evidence-based scientific research that is published at NeurIPS.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your main comments and questions below.\n\n---\n\n> Floating point operation overhead (a critical paper weakness): in Section 3.1.1 the authors discuss some of the concrete details used for their idealized runtime equation. Of note is the focus on deriving expressions “for the number of floating-point operations required for each of the two steps, and then use these to derive expressions for runtime.” While the number of floating point operations required to perform a single forward pass of inference is quite likely to an important datum for estimating inference runtime overhead, it is (in my opinion) just the first of many (perhaps dozens?) of factors that may be required to make an accurate estimation of inference overhead.\n\nWe want to clarify some misconceptions here.\n\n*Our method does not make the assumption that every floating-point operation has the same cost.* We certainly agree that not every floating-point operation has the same cost, and that the number of floating-point operations alone is insufficient to accurately estimate the runtime of a deep learning workload with many constituent operators. This is why we a) estimate prompt encoding (which is compute-bound) and token generation (which is memory-bandwidth-bound) runtime separately, b) use a piecewise linear function for prompt encoding runtime (since average throughput for matmuls increases with matmul size due to an increase in arithmetic intensity, as discussed in lines 145-150 in Section 3.2).\n\nMoreover, the metrics we propose (idealized runtime and denoised runtime) involve fitting a regression model to profiled *calibration runtimes directly* (Section 3.4) rather than relying on floating-point operation counts. The analytical expressions we show in Section 3.1 were intended to provide intuition and illustrate how the number of operations in each of these phases grow asymptotically, but our estimation method in practice does not directly use the number of floating-point operations. We can see from Figures 1 and 2 in the paper that the *measured end-to-end runtime for autoregressive inference* using existing optimized software and hardware implementations is linear in the number of generated output tokens and piecewise linear in the size of the prompt (only the best-fit line is estimated, the points are measured). We make similar observations when using raw runtimes of black-box APIs backed by similar Transformer models (Figure 2 in the PDF attached to this rebuttal).\n\n---\n\n> The spatial and computational complexity of FP operations between different hardware compute classes varies significantly (often times more than 10x-100x). [...]  The paper seems to only focus on GPUs, which, in commercial practice, tends not to be representative (or at a minimum highly specialized) of which compute is used for inference.\n\nWhile other resource types (including CPUs, FPGAs, etc.) can be used for inference workloads, we believe GPUs are still widely used for inference at scale (e.g., see the “AI for everyone” section in https://news.microsoft.com/source/features/ai/how-microsofts-bet-on-azure-unlocked-an-ai-revolution/). However, we expect our method to work well for other hardware architectures as well given profiled runtimes on the appropriate hardware (the profiled runtimes on the target hardware should take into account the architectural differences outlined above); verifying this is an interesting area of future work.\n\n---\n\n> References: of the 50 or so references included by the authors, it appears that only around 10 of them are from peer-reviewed publications. The other 40 or so are from arxiv and websites.\n\nApologies for this. Some of the referenced arXiv papers have been published at conferences (e.g., FlashAttention appeared at NeurIPS 2022, BERT appeared at NAACL 2019), and we can update these references to point to the conference versions in an updated copy of the paper (Google Scholar often shows arXiv references as the default unfortunately). Many of the referenced papers that have not appeared at conferences are widely cited (e.g., the GPT-4 and PaLM technical reports, scaling laws paper, Switch-Transformer).\n\n---\n\n> The authors seem to only focus on inference runtime overhead associated with GPUs (and a single class of GPU, as a reference point, Nvidia A100) and no other form of hardware compute. However, many other types of hardware are used to perform inference. Some of them are: CPUs, FPGAs, TPUs, CGRAs, etc. Moreover, each of these hardware classes have subclasses within that result in different temporal overhead and power footprint costs. None of these factors seem to be considered in the authors runtime equation.\n\nWe used our procedure to estimate runtimes on Nvidia V100 GPUs as well, shown in Figure 10 in the Appendix (attached in the Supplementary Material). As mentioned above, verifying that our method works well on other architectures like TPUs is an interesting area of future work."}, {"review_id": "DarqOGbRHv", "reviewer": "Reviewer_rBmS", "summary": "LLMs are dominating the NLP space at the moment, but efficient inference run-time and cost estimations have not been explicitly defined and tested. Raw run-times are not particularly useful across multiple services providing LLMs, as there is typically high variance overhead via the presence of black box prompt APIs. This paper proposes a few ways of estimating inference costs that attempt to provide an upper bound to the costs incurred by model computation. These metrics are demonstrated to be generally accurate and are cheap enough to be easily implemented, and the authors attempt to use them to generate new insights concerning various LLMs and generally large models across multiple services.", "questions": "1. Some noticeable run-to-run variance was observed for certain models, but I did not spot exactly how it was dealt with beyond the selection of a minimum time. Could you elaborate on this point, especially for AI21/J1-Grande v1?\n2. What would comparable exhaustive profiling cost a designer/user compared to your method? Can you provide some sort of comparison? Are there specific techniques you can compare your method to? Some materials from the appendix are mentioned, but this seems like it should be core content and included in an expanded Section 5.3. ", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Very relevant, as LLMs are dominating conversations in the NLP space. Paper as a whole does a good job of making a case for the necessity of their work. Methodologies to estimate a separation of overhead costs from core model computation are inherently useful. Most, if not all, assumptions are fairly setup and seem correct.\n2. The paper is very well-written. The problem setup is executed well and the initially proposed solution is very easy to follow. Explanations are mostly succinct, and most sections read as cogent. \n3. The results look to be generally correct and explanations related to them are succinct. The results are made more relevant by the scale of the experiments, rendering the paper somewhat unique. The applied denoising factor estimation technique seems to roughly work in providing an upper bound for expected model computation time. \n", "weaknesses": "1. Some areas of the paper are a bit overexplained for this venue. Readers and reviewers of NeurIPS should, for example, be very familiar with autoregressive frameworks and how they generally work, or the forward pass design of generative AI in general. \n2. Several early conclusions are mostly obvious. A lot of time is spent on specifying a procedure to estimate the number of floating point operations (and subsequently runtime based on throughput), but that isn't really all that interesting or new. For example, LLMs having roughly linear costs with respect to the size of their input (their quadratic attention costs are dwarfed) is probably well understood by your audience. Empirical proof related to this likely shouldn't be core to the paper.\n3. Unclear if particularly new insights were generated from the results. There were largely no clear winners in cost-capability tradeoff analyses, there is no comparison to other evaluation techniques in the core body of the paper, and insights into when models fall or do not fall on the Pareto frontier of a given task do not seem initially useful. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your main comments and questions below.\n\n---\n\n> Some areas of the paper are a bit overexplained for this venue. Readers and reviewers of NeurIPS should, for example, be very familiar with autoregressive frameworks and how they generally work, or the forward pass design of generative AI in general.\n\nThank you for this suggestion. We can defer lower-level details to the Appendix where appropriate.\n\n---\n\n> Several early conclusions are mostly obvious. A lot of time is spent on specifying a procedure to estimate the number of floating point operations, but that isn't really all that interesting or new. For example, LLMs having roughly linear costs with respect to the size of their input (their quadratic attention costs are dwarfed) is well understood by your audience.\n\nGiven that there hasn’t been a paper (or blogpost) describing these costs at length, we believe that our paper will be useful to the ML systems community, especially given the current interest in deploying efficient LLM inference services today. We have seen misconceptions on the internet (e.g., Twitter) regarding the relative cost of attention in the full model (we are withholding the link to preserve anonymity and double blind).\n\n---\n\n> Largely no clear winners in cost-capability tradeoff analyses / no comparison to other evaluation techniques in the paper.\n\nWe argue that the lack of clear winners is itself an interesting insight because it illustrates the complexity of evaluating the capabilities of LLMs: there is no one agreed upon task to evaluate the capabilities of LLMs, and no one LLM (at least among the ones we studied) dominates across a range of tasks. We want to emphasize that studying the capabilities of LLMs is not a direct goal of our work. Instead, we wish to augment such existing analyses by providing efficiency metrics comparable across different models, allowing us to provide additional context.\n\nWe disagree that we do not compare to other efficiency evaluation techniques. We compare our metrics of idealized runtime and denoised runtime to raw runtime measurements as well as heuristics such as the model size and the number of floating-point operations, and show that these can be misleading (see Figure 4).\n\n---\n\n> Some noticeable run-to-run variance was observed for certain models, but I did not spot exactly how it was dealt with beyond the selection of a minimum time. Could you elaborate on this point, especially for `AI21/J1-Grande v1`?\n\nAs we see in Figure 8b, the `AI21/J1-Grande v1` model experiences significant variation across requests, which indicates that raw runtimes are an unreliable measure of API performance. To address this limitation of raw runtimes, we proposed the denoised runtime metric which estimates runtimes using the linear regression model detailed in Section 4. The denoised runtime estimates factor out performance variation by using the minimum runtimes observed across a small sample of trials as input to the linear regression model; the regression fitting method is also able to factor out noise in individual measurements.\n\nFigure 1 in the PDF attached to this rebuttal shows the min, median, and max end-to-end runtime for a subset of these models, and illustrates the degree of variation that we can see in raw runtimes. Figure 2 shows the minimum end-to-end runtime plotted against the number of output tokens for 5 models considered in our evaluation (including AI21/J1-Grande v1), along with the best-fit line estimated with our method. We can see that our procedure is able to factor out a lot of the performance variation that we observe in the raw runtime measurements.\n\n---\n\n> What would comparable exhaustive profiling cost a designer/user compared to your method? Can you provide some sort of comparison? \n\nWe can model the exhaustive profiling costs for computing idealized runtime and denoised runtime separately.\n\nFor the idealized runtime, we can compare the cost of exhaustively running all queries in the local environment with running a comparatively small set of calibration queries as we propose in our paper (Section 3.4). We run each query multiple times to get a reliable runtime measurement. In both cases, we need to run all queries through the API once in order to get the number of prompt and output tokens for each query. We use the following notation:\n\n$x = \\text{Number of trials}$.\n\n$Q = \\text{Set of user queries}$.\n\n$\\hat{Q} = \\text{Set of calibration queries}\\ (|\\hat{Q}| \\ll |Q|)$.\n\n$c_{\\text{API}}(q) = \\text{Cost to invoke API on query } q$.\n\n$c_{\\text{local}}(q) = \\text{Cost to invoke local model on query } q$.\n\nThis gives the following expression:\n$$\\text{Idealized runtime savings} = \\frac{{x \\cdot \\sum_{q \\in Q}{c_\\text{local}(q)} + \\sum_{q \\in Q}{c_\\text{API}(q)}}}{x \\cdot \\sum_{q' \\in \\hat{Q}}{c_{\\text{local}}(q')} + \\sum_{q \\in Q}{c_\\text{API}(q)}}.$$\n\nWe can compute the savings concretely on the 4 HELM tasks evaluated in Section 5.4. We use publicly available per-token costs for the `OpenAI/davinci` model and the per-hour cost to rent an 8-A100 server from AWS. Assuming 50 trials per query (in practice, fewer trials are probably sufficient), this results in an overall savings of 57$\\times$ when using our approach with calibration queries compared to exhaustive execution.\n\nWe can derive a similar expression to estimate the savings from using our approach for the denoised runtime, with the main difference being calibration queries now need to be run through the black-box API, instead of locally.\n\n$$\\text{Denoised runtime savings} = \\frac{x \\cdot \\sum_{q \\in Q}{c_\\text{API}(q)}}{x \\cdot \\sum_{q' \\in \\hat{Q}}{c_{\\text{API}}(q')} + \\sum_{q \\in Q}{c_\\text{API}(q)}}.$$\n\nUsing the same parameters as above, we get a savings of 48$\\times$ using calibration queries compared to exhaustive execution (close to $x=50$, since both numerator and denominator in the above expression are dominated by the cost of running queries in $Q$).\n"}, {"review_id": "VE5juN5VDc", "reviewer": "Reviewer_x2WX", "summary": "\"This paper introduces a range of metrics designed to compare the tradeoffs between inference efficiency and capability of autoaggressive Language Model (LLM) based on Transformers. Alongside the raw run-time metric, which can be influenced by infrastructure optimization and performance variance, the paper proposes additional metrics, namely idealized runtime, denoised runtime, idealized dollar cost, and idealized energy cost. These metrics enable a comprehensive comparison of various LLMs, even when they are served through different black-box APIs and hardware/software implementations.", "questions": "* In Figure 3, the denoised runtimes of OpenAI models are often lower than the idealized runtime. Could you provide further explanation for this phenomenon? Is it possibly related to the presence of better-optimized hardware and software infrastructure?\n\n* Can you clarify which variables are being compared in Table 2? Are the comparisons made between Raw runtime and idealized runtime?", "limitations": "As mentioned in Section 3.1.1, the analytical model does not account for models with very large context windows (> 10,000); however, it is worth noting that these models are now becoming available.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: The paper introduces a novel analytical approach to estimate the inference efficiency of autoregressive Transformer models, overcoming challenges posed by models accessible only through black-box APIs and running on diverse hardware and software. This distinct proposal stands apart from previous analytical models designed for non LLMs. Moreover, it addresses the limitations associated with other proxies, such as model size and FLOPs.\n\n* Quality and clarity: The proposal is presented with a rigorous analytical description and validated through empirical verification. The paper's structure is well-organized, and the content is conveyed in a clear and concise manner.\n\n* Significance: The metrics, encompassing both the newly proposed and conventional ones, are effectively employed to compare the tradeoff between capability and efficiency. This comparison leads to several noteworthy insights, elaborated upon in Section 5.4.", "weaknesses": "There are a few minor formatting issues that require attention. For instance, in line 153, the notation of $o^2$ is confusing. The \"2\" is intended to refer to the footer, but it appears as if it denotes an exponent of 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your main comments and questions below.\n\n---\n\n> There are a few minor formatting issues that require attention. For instance, in line 153, the notation of o^2 is confusing. The \"2\" is intended to refer to the footer, but it appears as if it denotes an exponent of 2.\n\nThank you for pointing this out, we will fix these formatting issues in the revised version of this paper.\n\n---\n\n> In Figure 3, the denoised runtimes of OpenAI models are often lower than the idealized runtime. Could you provide further explanation for this phenomenon? Is it possibly related to the presence of better-optimized hardware and software infrastructure?\n\nYes, the idealized runtime being larger than the denoised runtime suggests that OpenAI models use software and/or hardware implementations which are faster than the reference implementations we used to compute our runtime estimates (in particular, the Megatron library with the minimum number of A100 GPUs needed to fit the model / other tensors required for inference). We detail this behavior in the “Denoised Runtime” part of Section 4. Possible optimizations could include newer hardware (e.g., H100 GPUs), quantization, sparsification, distillation, or additional parallelization. \n\n---\n\n> Can you clarify which variables are being compared in Table 2? Are the comparisons made between Raw runtime and idealized runtime?\n\nYes, we compare raw runtime with idealized runtime in Table 2.\n\n---\n\n> As mentioned in Section 3.1.1, the analytical model does not account for models with very large context windows (> 10,000); however, it is worth noting that these models are now becoming available.\n\nOne thing to note here is that we want to compare the context length to ($6 \\cdot \\text{hidden\\\\_size}$). Models like GPT-3 have quite a large hidden size already (12288), so it is a reasonable assumption that context length is generally smaller than $6 \\cdot 12288 = 73728$. One can only speculate that newer models like GPT-4 have larger hidden sizes.\n\nFor models where this approximation doesn’t quite hold, one can still fit a quadratic function to the runtime. This was not necessary when we initially did this work, but is not a dealbreaker going forward.\n\nWe also note that some of these models (e.g., MPT with the larger context length) were released close to the NeurIPS deadline (~10 days before), and a lot of the current model offerings still have much smaller maximum content lengths."}, {"review_id": "CjoL1KGHan", "reviewer": "Reviewer_a4zR", "summary": "This paper aims to better understand the tradeoff between the inference efficiency and capability of LLM. It comes up with a metric called “idealized runtime” to fairly evaluate the inference efficiency of several popular LLMs, which is designed to factor out the irrelevant noises such as hardware, code stack, network latency and so on. With this metric, the authors analyised the efficiency and capability of LLMs and got some inpsiring insights.", "questions": "- As I understand, raw runtime and denoised runtime are derived from the API of model providers, and the idealized runtime is derived from the local cluster. Then how to interpret the impact of batch-size? As we know, the batchsize plays a quite import role in the padding, scheduling and I/O overhead during the inference, which could lead to very difference runtime. Since the batch size behind the API is unknown, how do we know if the differences between raw/denoised time and idealized time is actually simply dominated by the batch-size, instead of hardware and software?\n- I wonder how to get the denoised runtime specifically? In the paper it says “we can run multiple trials in the profiling step and perform linear regression using the *minimum* obtained runtime across trials for each p and o”. So I guess it means you sent the same queries to a API n times, and chose the minimum runtime as denoised time? But how to do this sampling properly is important, for example, you may have quite different response time between morning and evening because of the traffic volume. And again, the impact of batch-size cannot be denoised here.\n- In Figure3, why there are two figures in each subfigure? In the left figure of Fig3(b), why the y-intercept is not zero?\n- Regarding the idealized runtime, it will be helpful if the authors list the detailed specifications of hardware and software setup, such as, CUDA version, Megatron version, tensor / pipeline parallel degrees, inter/intra node connections of GPUs, inference batch size.\n- From Fig4, we can learn that the idealized time of different models actually aligns relatively well with the model size and FLOPS (except the little difference of BLOOM and YaLM), whereas the raw and denoised runtime is way more different than model size. How can we interpret this observation? Could we just use the model size and number of GPUs needed to roughly estimate the idealized time?\n- Following the method described in section 3.4, we could estimate the linear regression function of prompting and generating time from an API, which is interesting. But in the experiment section, we only see the raw/denoised time in total, instead of breaking into prompting and generating time. I see a table in the Appendix about this data, but it would be nice to present a bit of them in the main paper as well. Otherwise the readers may wonder why do we need Section 3.4 if all the experiments are just end-2-end runtime.", "limitations": "The potential social impact is not discussed in the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all, I found the motivation of this paper very realistic and helpful to the community. We have been seeing lots of efforts made by the industry and academia to reduce the cost of the inference of LLMs, but how to compare those models from different parties fairly is yet to be figured out. This paper would be an inspiring exploration to this end. Besides, the analysis regarding the LLM inference is thorough and clear. ", "weaknesses": "- Some technical details are kind of missing, which I will specify in the Question section.\n- The concept of “idealized runtime” could be a bit unpractical because it requires one to implement and deploy the LLM on a very specific hardware and software setting, which may involve quite amount of proper engineering efforts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your main comments and questions below.\n\n---\n\n> The concept of “idealized runtime” could be a bit unpractical because it requires one to implement and deploy the LLM on a very specific hardware and software setting, which may involve quite amount of proper engineering efforts.\n\nIn this paper, we actually use an already-existing optimized open-source implementation (Megatron) on widely available hardware (a 8xA100 server with NVLink). To estimate a new model’s idealized runtime, one merely needs to instantiate a model of the right size (using appropriate command-line arguments) and run a few calibration queries to obtain runtimes on a few prompt sizes and number of output tokens. The resulting learnt regression parameters can then be used to estimate the idealized runtime for any arbitrary query with minimal engineering effort.\n\n---\n\n> As I understand, raw runtime and denoised runtime are derived from the API of model providers, and the idealized runtime is derived from the local cluster. Then how to interpret the impact of batch-size? As we know, the batchsize plays a quite import role in the padding, scheduling and I/O overhead during the inference, which could lead to very difference runtime. Since the batch size behind the API is unknown, how do we know if the differences between raw/denoised time and idealized time is actually simply dominated by the batch-size, instead of hardware and software?\n\nWe used a batch size of 1 since this gives us a lower-bound for latency for our assumed hardware and software. We agree that in practice, API providers might use a larger batch size to trade off latency for improved throughput. The denoised runtime is estimated using the minimum raw runtime seen across multiple runs of a given query; given enough runs, one of these runs should involve the query being executed as part of a batch of size 1.\n\n---\n\n> I wonder how to get the denoised runtime specifically? In the paper it says “we can run multiple trials in the profiling step and perform linear regression using the minimum obtained runtime across trials for each p and o”. So I guess it means you sent the same queries to a API n times, and chose the minimum runtime as denoised time? But how to do this sampling properly is important, for example, you may have quite different response time between morning and evening because of the traffic volume. And again, the impact of batch-size cannot be denoised here.\n\nThis is a good point. Your understanding is correct that naively, the denoised runtime is computed by running the same query multiple times and then choosing the minimum one. In practice, we only perform this procedure for a small set of calibration queries (Section 3.4), and then use the resulting profiled times to estimate the denoised runtime for all other queries. We make an implicit assumption that the smallest runtime across multiple runs of each calibration query corresponds to a batch of size 1. In general, we observed that the minimum raw runtime from our calibration queries have a largely linear relationship with the number of output tokens (see Figure 2 in the PDF attached to this rebuttal), similar to what we saw in our local cluster setting with Megatron.\n\nRuntime samples should be collected across different times of day in order to capture the effects of different traffic volumes..\n\n---\n\n> In Figure 3, why there are two figures in each subfigure? In the left figure of Fig3(b), why the y-intercept is not zero?\n\nApologies for not making this clear in the paper. The right subfigure in Figures 3a and 3b are a magnified version of the left subfigure in linear scale (instead of log-log scale). The y-intercept should be 0, and we will add a ytick to reflect this.\n\n---\n\n> Regarding the idealized runtime, it will be helpful if the authors list the detailed specifications of hardware and software setup, such as, CUDA version, Megatron version, tensor / pipeline parallel degrees, inter/intra node connections of GPUs, inference batch size.\n\nThank you, we will make this clear in the revised draft. We use CUDA version 11.5.0, Megatron version v2.6, tensor parallelism equal to the full number of GPUs used for inference (see Table 1 in the Appendix, attached as a part of the Supplementary Material) except for the MS+NV/TNLG v2 model, and batch size 1.\n\n---\n\n> From Figure 4, we can learn that the idealized time of different models actually aligns relatively well with the model size and FLOPS (except the little difference of BLOOM and YaLM), whereas the raw and denoised runtime is way more different than model size. How can we interpret this observation? Could we just use the model size and number of GPUs needed to roughly estimate the idealized time?\n\nThe model size and number of GPUs give a coarse directional estimate of the idealized runtime, but a) cannot answer questions such as “can queries be answered within a latency budget of 100ms?”, b) cannot directly predict the effect of hardware and software optimization (e.g., using H100 GPUs instead of A100 GPUs, or using FlashAttention instead of vanilla attention) on query runtime.\n\n---\n\n> Following the method described in section 3.4, we could estimate the linear regression function of prompting and generating time from an API, which is interesting. But in the experiment section, we only see the raw/denoised time in total, instead of breaking into prompting and generating time. I see a table in the Appendix about this data, but it would be nice to present a bit of them in the main paper as well. Otherwise the readers may wonder why do we need Section 3.4 if all the experiments are just end-2-end runtime.\n\nThis is a good point. Our main experiments focus on end-to-end runtime since we can only measure the raw end-to-end runtime using the black-box APIs. But we agree that it is interesting to present both of these separately, and we can add these to a revised version of the paper."}, {"review_id": "MmPXYtgkJi", "reviewer": "Reviewer_aZks", "summary": "The authors proposed idealized runtime, a metric for measuring inference efficiency in LLMs, which measures the performance of models as if they were executed on a given hardware and software platform. Idealized runtime efficiently can be extended to estimate the idealized energy and dollar cost. The metric also takes into account the amount of hardware required to perform inference on a given model. Using these metrics, the authors compare ten state-of-the-art LLMs to demonstrate inference efficiency-capability tradeoffs. Overall, the authors aim to provide a more accurate and fair comparison of LLMs across different providers and to shed light on the tradeoffs between inference efficiency and capability.", "questions": "I have raised my concerns in the previous section.", "limitations": "The authors may further discuss the potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper presentation is clear and the metric proposed is intuitive to understand.\n\n2. The empirical study is thorough and examines the LLM inference efficiency from various perspectives.\n\n3. The empirical findings demonstrate that the proposed metric can largely capture the efficiency-capability tradeoff in LLM inference.", "weaknesses": "1. The models on different tasks and metrics exhibit vastly different Pareto Frontiers (Figure 9). The cause of this variance hasn't been fully explored and explained in the work. Can these datasets/tasks properly evaluate the capabilities of the LLMs or is the variance due to something else?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your comment on weaknesses below.\n\n---\n\n> The models on different tasks and metrics exhibit vastly different Pareto Frontiers (Figure 9). The cause of this variance hasn't been fully explored and explained in the work. Can these datasets/tasks properly evaluate the capabilities of the LLMs or is the variance due to something else?\n\nWe think that this is in part due to the difficulty in evaluating the capabilities of LLMs: there is no one agreed upon task to evaluate the capabilities of LLMs, and no one LLM (at least among the ones we studied) dominates across a range of tasks. Studying the capabilities of LLMs (or designing a single task that reasonably tests all of the facets of a LLM) is not a direct objective of our work. Instead, we wish to augment such existing analyses by providing efficiency metrics comparable across different models, allowing us to provide additional context.\n"}, {"review_id": "CZbFxndigV", "reviewer": "Reviewer_93KK", "summary": "The authours proposed a metric called `idealized runtime` to evaluate the inference efficiency among different LLMs as if they were run on the same standard hardware/software system. The main contributions can be summarized as following:\n\n* the `idealized runtime` for a specific query with prompt length `p` and output length `o` on a target LLM can be collected by running the LLM on standard hardware/software (e.g., A100 and Megatron containers) **if target LLM's architecture is known/opensourced**\n* the `idealized runtime` of transformer models can be modeled as a function linear to `o` and piecewise linear to `p` **if context window size is much smaller than embedding size of the model**. So with a few queries with different `p` and `o`, the coefficients of this function can be extracted by fitting the runtime of these queries to the function.\n* with the fitted function, `idealized runtime` of queries can be estimated without running them exhaustively, so that the capacity of model (e.g., accuracy on a dataset/benchmark) can be compared against the pre-calculated/estimated `idealized runtime` of all the queries in the dataset to tradeoff/review scaling effect of LLMs.\n* for closed models with api access, the authors proposed an alternative `denoised runtime` to approximate their idealized runtime.", "questions": "Basically 1st point described in weakness. How far off is simple FLOPs estimated latency from the estimated `idealized runtime`, can you do a quantative study to support the necessity of doing this linear function fitting/approximation? And if you found the effective FLOP/s of an LLM is much lower/higher than 50%, can you study whether it is intrinsicaly due to the model (e..g, many non-dense ops) or the implementation.", "limitations": "The limitations are mainly the assumptions the authors made, as mentioned in weakness. The authors either didn't discuss them or without referring to SOTA LLM developments.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "* The authors pointed out the scaling law of LLMs should not just focus on accuracy vs FLOPs/Model size but accuracy vs inference latency/cost/power as well, which is a valuable point in LLM production.\n* The authors proposed a simple way to estimate the idealized runtime of queries on standard hardware/software systems given prompt length and (projected) output length.", "weaknesses": "* The inference latency of a well known (open sourced) LLM architecture can be easily estimated by total ops of the model (e.g., those calculated in line 105/115) and divided by hardware effective (typically around 50% for LLMs on A100, https://arxiv.org/pdf/2104.04473.pdf or BLOOM paper) FLOP/s. The authors only mentioned about the drawback of this proxy briefly in line 300-308, without evidences based on target LLMs. Thus the significance of approximating these latencies via fitting instead of analytical calculation is quesitonable.\n\n* The work makes too many assumptions that devalue its practice,\n  1. context length << embedding size. This is true when the common context lenght is only 2K, but new development on LLMs have pushed them to 32K(GPT4)/64K (MPT)/100K(Claude), so this assumption and therefore the linear function approximation is no longer valid. With long context/prompt length, the analytical equation is probably more accurate.\n  2. the `idealized runtime` is collected on standard Megatron container assuming model architectures are known. This limits the effectiveness of the work to open sourced models with efficient (distributed) implementations. \n  3. the work assumes the pretrained/finetuned model is the final service/inference model. However in practice, the service model can be quantized/sparsified/distilled, so apple-to-apple inference latency comparision on pretrained models doesn't direclty translate to service model and the gap is typically huge, e.g., as shown by FIG3, the variance in idealized-runtime vs denoised-runtime is significant.\n\n* The presentation of the work is not clear and sometimes repetitive/redudant.\n  1. using FIG3 again as example, what's the diff of the two subplots in (a)/(b), it is not clarified and seems just different scales in axis. And as mentioned above, this FIG suggests for a couple models, the denoised-runtime is smaller than idealized-runtime, which is not a good evidence that idealized-runtime is a good approximation of roofline runtime.\n  2. A lot of the equations are just rewrite/modifications to a simple linear function, and probably can be omitted or summarized in a table.\n\n* Some of the conclusions are drawn without evidence, e.g., in line 257, the authors claimed \"This suggests that scale alone does not predict model capabilities.\" Scaling law should be studied for the same model by varying model/dataset size. Comparing different model families with drastically different in pretraining dataset size/quality can not lead to any conclusion over model capacity over model size or inference latency.\n\n* Overall the work is oversimplified by the assumptions above to a linear regression problem, which can be analytically calculated (and is commonly adopted, e.g., like Palm2 tech report) and becomes trivial to the audience of NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your main comments and questions below.\n\n---\n\n> The inference latency of a well known (open sourced) LLM architecture can be easily estimated by total ops of the model and divided by hardware effective (typically around 50% for LLMs on A100s) FLOP/s. The authors only mentioned about the drawback of this proxy briefly in line 300-308.\n\nThis will not work well since the token generation part of the computation is not compute-bound, but is memory-bound; assuming 50% of peak compute throughput for this part of the workload will work _extremely poorly_ and leads to inaccurate runtime estimates (as pointed out below). Our point in lines 300-308 is slightly different: that the number of floating-point operations itself is not a good metric for practitioners since it is hard to interpret.\n\n---\n\n> Context length << embedding size. New LLMs have pushed context lengths to 100k, so the linear function approximation is no longer valid.\n\nOne thing to note here is that we want to compare the context length to ($6 \\cdot \\text{hidden\\\\_size}$). Models like GPT-3 have quite a large hidden size already (12288), so it is a reasonable assumption that context length is generally smaller than $6 \\cdot 12288 = 73728$. Newer models like GPT-4 should have larger hidden sizes.\n\nFor models where this approximation doesn’t quite hold, one can still fit a quadratic function to the runtime. This was not necessary when we initially did this work, but is not a dealbreaker going forward.\n\nWe also note that some of these models (e.g., MPT with the larger context length) were released close to the NeurIPS deadline (~10 days before), and many current model offerings still have smaller maximum content lengths.\n\n---\n\n> The idealized runtime is collected on standard Megatron container assuming model architectures are known. This limits the effectiveness of the work to open sourced models with efficient (distributed) implementations.\n\nThis is not quite true. It is true that we need to know key model hyperparameters to do anything interesting here (e.g., the hidden size, number of layers, number of attention heads, etc.). But given these details, we can instantiate a model of the right size and use the same *open-source* software stack (that already has an efficient distributed implementation) to obtain runtime estimates. In fact, many of the models that we have already evaluated are not open-source (e.g., `OpenAI/davinci` and `AI21/J1-Jumbo`).\n\n---\n\n> The work assumes the pretrained/finetuned model is the final service/inference model. However in practice, the service model can be quantized/sparsified/distilled.\n\nAs noted in the paper: “For AI21 Labs models, idealized runtimes are greater than denoised runtimes 15.7% of the time. For OpenAI models, idealized runtimes are greater than denoised runtimes 64.2% of the time. For all other models, idealized runtimes are always lower than the denoised runtimes, [...]”. It is likely that OpenAI is implementing optimizations such as quantization, sparsification, distillation, additional parallelization, or better software optimization to extract inference runtime improvements; we want to emphasize that these optimizations are *orthogonal to our work*. We want to provide a metric that is apples-to-apples comparable across model providers; other model providers can also implement such optimizations to optimize their API offerings. The purpose of Figure 3 is to show that our estimated idealized runtimes are largely consistent with raw runtime measurements (and denoised runtime estimates).\n\nThe denoised runtime metrics make more sense if one wants to compare the deployed models directly while taking out the effect of performance variation.\n\n---\n\n> What's the diff of the two subplots in (a)/(b), it is not clarified and seems just different scales in axis.\n\nApologies for not making this clear in the paper. The right subfigure in Figures 3a and 3b are a magnified version of the left subfigure in linear scale (instead of log-log scale). The y-intercept should be 0, and we will add a ytick to reflect this.\n\n---\n\n> Some of the conclusions are drawn without evidence, e.g., in line 257, the authors claimed \"This suggests that scale alone does not predict model capabilities.\"\n\nApologies for the confusion. We were only referring to model scale (or the number of parameters in the model) here; it is of course true that the training dataset size affects model quality / capabilities, but nothing in our claim was meant to support or contradict that statement. We can change the phrasing in the paper to “This suggests that the number of model parameters alone does not predict model capabilities”. We can also add citations to indicate that other papers have also made this observation.\n\n---\n\n> Overall the work is oversimplified; the analytical FLOPs for inference is commonly adopted, e.g., like Palm2 tech report.\n\nWe respectfully disagree that this is “trivial”. The PaLM-2 report does have a discussion for optimal *training FLOPs* (6ND where D is the number of training tokens and N is the number of parameters in the model), not inference FLOPs.\n\n---\n\n> How far off is simple FLOPs estimated latency from the estimated idealized runtime?\n\nThe FLOPs-estimated latency is inaccurate.\n\nThe effective FLOP/s is much lower than 50% because of the intrinsic structure of autoregressive inference: while training is largely compute-bound, autoregressive inference’s token generation stage is memory-bandwidth-bound. Depending on the number of output tokens generated and the prompt size, end-to-end autoregressive inference becomes largely memory-bandwidth-bound as well; the larger the number of generated output tokens, the lower the effective throughput since computation is in the memory-bandwidth-bound portion of the computation for longer. In our experiments, we found that effective inference FLOP/s for the various models studied in Section 5 varies from 15% to 41%."}]

Author rebuttal:
We thank all the reviewers for their time and effort in providing feedback on our paper, which will be useful in improving its quality.

All reviewers seem to agree that the problem addressed by the paper is important and well-motivated. However, the two lowest-score reviews seem to have serious misunderstandings of the paper:
- Our paper does not advocate using the number of floating-point operations to directly estimate end-to-end runtime (as reviewer Jz6a points out, the cost of each floating-point operation is not the same). Instead, we show the expressions for number of floating-point operations to provide intuition for our method (described in Section 3.4), which uses *runtimes of a set of calibration queries* to estimate runtime for any query (assuming a given hardware and software setup).
- Using the number of floating-point operations to estimate runtime works poorly. The effective FLOP/s for inference is much lower than 50% because of the intrinsic structure of the computation: while training is largely compute-bound, autoregressive inference’s token generation stage is memory-bandwidth-bound. Depending on the number of output tokens generated and the prompt size, end-to-end autoregressive inference becomes largely memory-bandwidth-bound as well; the larger the number of generated output tokens, the lower the effective throughput since computation is in the memory-bandwidth-bound portion of the computation for longer. In our experiments, we found that effective inference FLOP/s varies from 15% to 41%.

The main contribution of our paper is a general methodology to efficiently estimate the runtime of autoregressive inference of a dense Transformer model for a prompt of given size and number of output tokens on a given hardware and software stack, using profiling on a small set of calibration queries. The exact proposed method works best for prompts up to ~10k tokens for current-day models, but changing the output generation time from a linear model to a quadratic model could make runtime estimation accurate for even larger context sizes.

We respond to the other main concerns raised by reviewers in the per-reviewer comments.

Meta-review: Paper proposes a method to efficiently estimate the runtime of autoregressive inference of a (dense) Transformer model for a prompt of given size and number of output tokens on a given hardware and software stack, using profiling on a small set of calibration queries.

All of the reviewers appreciate the concept of the metric Idealized Runtime, however some didn't find its significance in practice. Authors addressed several comments by the reviewers and majority found it to be convincing. Considering the lack of any previous attempt in this space, and the abundance of LLM services lately, studies like this one has the potential to redistribute the focus of the field onto inference cost/latency and predictability. Hence I found it to be significant to be shared via NeurIPS.
Decision: Accept (poster)