SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Deepak Narayanan* NVIDIA
dnarayanan@nvidia.com
Keshav Santhanam
Stanford University keshav2@cs.stanford.edu
Peter Henderson
Stanford University phend@cs.stanford.edu
Rishi Bommasani
Stanford University
nlprishi@stanford.edu
Stanford University tonyhlee@stanford.edu
Stanford University  
pliang@cs.stanford.edu
Large language models (LLMs) are highly capable but also computationally expensive. Characterizing the fundamental tradeoff between inference efficiency and model capabilities is thus important, but requires an efficiency metric that is comparable across models from different providers. Unfortunately, raw runtimes measured through black-box APIs do not satisfy this property: model providers can implement software and hardware optimizations orthogonal to the model, and shared infrastructure introduces performance contention. We propose a new metric for inference efficiency called idealized runtime, that puts models on equal footing as though they were served on uniform hardware and software without performance contention, and a cost model to efficiently estimate this metric for autoregressive Transformer models. We also propose variants of the idealized runtime that incorporate the number and type of accelerators needed to serve the model. Using these metrics, we compare ten LLMs developed in 2022 to provide the first analysis of inference efficiency-capability tradeoffs; we make several observations from this analysis, including the fact that the superior inference runtime performance of certain APIs is often a byproduct of optimizations within the API rather than the underlying model. Our code is open sourced at https://github.com/stanford-crfm/helm-efficiency.
Large language models (LLMs; Devlin et al., 2019; Brown et al., 2020; Rae et al., 2021; Lieber et al., 2021; Smith et al., 2022; Black et al., 2022; Chowdhery et al., 2022; OpenAI, 2023) have grown in size by almost four orders of magnitude in recent years, achieving state-of-the-art accuracy on traditional tasks like question answering and summarization (Zellers et al., 2019; Hendrycks et al., 2020; Zhang et al., 2023). LLMs display many exciting new capabilities as well, like reasoning about the physical world (Bisk et al., 2020), solving math problems (Cobbe et al., 2021; Li et al., 2023), and generating code (Chen et al., 2021), to name a few. To capitalize on these capabilities, several organizations offer access to LLMs through black-box APIs (OpenAI; AI21; Cohere) and many companies are deploying LLM-powered products at scale like ChatGPT, Bing, jasper.ai and Github Copilot (Reuters; Microsoft; Scale VP).
When building models, both users and developers must balance the benefits of new capabilities against the costs of scale. Recent efforts have begun to systematically evaluate and compare the downstream
*Work done while author was at Microsoft Research.
Figure 1: Comparison of raw runtime to the two runtime metrics proposed in this work.
task accuracies of LLMs (Brown et al., 2020; Rae et al., 2021; Liang et al., 2023; Srivastava et al., 2023), while others have examined the massive energy, financial, and computational costs of model training (Cao et al., 2020; Henderson et al., 2020; Strubell et al., 2019; Bender et al., 2021; Patterson et al., 2021; Bommasani et al., 2021, §5.3). However, few have considered the trade-offs of inference efficiency vs. capability improvements (Bommasani et al., 2023). This is important since model inference costs might far outweigh training costs for certain applications (e.g., ChatGPT had about 180 million unique visitors in August 2023 (Reuters, 2023)).
Our main contribution in this paper is a cost model for the end-to-end runtime of autoregressive generation using Transformer models with multi-head attention (Vaswani et al., 2017). Since autoregressive inference is composed of two stages that have vastly different computational properties (Cursor, 2023), the end-to-end runtime cannot be estimated directly from the total number of floating-point operations. We instead consider each of these two stages independently, and observe that inference runtime can be expressed as the sum of a parameterized piecewise linear function of the number of prompt tokens and a linear function of the number of output tokens under certain assumptions<sup>2</sup>. These parameters are specific to the target model and the software / hardware stack used. Our cost model can be efficiently fit to runtime profiles collected on either dedicated hardware or through black-box APIs, and allows us to estimate the runtime of a workload without running it in its entirety.
Raw runtimes of inference queries to black-box APIs are not inherently comparable across model providers since the API can include optimizations orthogonal to the model (e.g., caching, customized hardware, etc.) and be susceptible to performance variance (e.g., in our experiments, we found that heavy load can worsen raw runtime by up to  $2 \times$  for certain model providers). This makes it hard to gauge the inference efficiency of models on a level playing field, which can be important for model creators and researchers to understand the full long-term costs of various training decisions (e.g., model architecture / number of parameters). Raw runtime is still a good metric for end users who are directly impacted by slow (or fast) predictions.
We propose inference efficiency metrics that facilitate apples-to-apples comparisons across models. The main metric we propose is the idealized runtime, which is the runtime of an inference query if run on a specified common software and hardware stack. The idealized runtime can also be used to compute idealized energy and dollar costs to take into account the cost of the hardware used to serve the model. All of these idealized metrics can be estimated efficiently with our cost model.
Using these metrics, we conduct a novel analysis of inference efficiency-capability tradeoffs for various Transformer models available through black-box APIs (§5.3). Our analysis reveals several insights about the relative performance of these models. For example, the vanilla 0penAI/davinci model is often on the Pareto frontier of the efficiency-capability trade-off landscape when using raw runtime as the efficiency metric on 4 NLP scenarios covering sentiment analysis, question answering and classification. However, this efficiency appears to come from optimizations within the API rather than the model; 0penAI/davinci is consistently not on the idealized runtime Pareto frontier. Overall, we found the idealized metrics to be a useful tool for model creators and researchers to understand the true inference runtime, energy, and dollar costs that result from a particular model architecture and training process.
2Our cost model focuses on dense Transformer models for now. We believe this is a reasonable compromise since modern text generation APIs are powered almost exclusively by such models. The cost model needs to be slightly modified to cover popular variants like mixture-of-expert (MoE) models. We also mainly concentrate on small to medium context windows in this paper; see §3.1.2 for a full discussion.
2 Autoregressive Inference of Language Models
The input to a language model is a sequence of tokens (e.g., words). Autoregressive language models like GPT-3 (Brown et al., 2020) estimate the conditional probability  $\operatorname{Pr}(x_i|x_{1:i - 1})$  of a token  $x_{i}$  given prefix tokens  $x_{1},x_{2},\ldots ,x_{i - 1}$ . During training, where we know all tokens in the training input a priori, the conditional probabilities  $\operatorname{Pr}(x_1|\emptyset),\operatorname{Pr}(x_2|x_{1:1}),\operatorname{Pr}(x_3|x_{1:2}),\ldots ,\operatorname{Pr}(x_s|x_{1:s - 1})$  can be estimated in parallel, and thus only a single forward pass needs to be executed in every iteration before the backward pass. However, at inference time, outputs of the model need to be fed back in as inputs to generate subsequent outputs. In particular, a token  $x_{i}$  is sampled from the conditional probability distribution obtained by running a forward pass through the model. Different sampling approaches can be used to obtain the token  $x_{i}$  from the conditional probability distribution  $\operatorname{Pr}(x_i|x_{1:i - 1})$ ; common approaches include greedy sampling, random sampling with temperature annealing, nucleus sampling, and beam search. The process then needs to be repeated for the next token  $x_{i + 1}$  and so on. Consequently, inference through an autoregressive language model needs to perform multiple forward passes. This entire procedure differs from traditional inference for other models which require just a single forward pass.
Inference queries to language models are seeded with a prompt, which is a set of initial tokens  $x_{1}, x_{2}, \ldots, x_{p}$  (we assume that the prompt has  $p$  tokens). The conditional distribution  $\operatorname{Pr}(x_{p+1} | x_{1:p})$  can then be computed through a forward pass. We call this the "prompt encoding" phase. Each subsequent generated token (sampled from  $\operatorname{Pr}(x_{i+1} | x_{1:i})$  where  $i > p$ ) needs its own forward pass through the model, which we term the "token generation" phase.
3 Cost Model for Autoregressive Inference Runtime
We now seek to derive a cost model  $T(o, p; \theta)$  for the runtime of autoregressive inference of Transformer models (Vaswani et al., 2017).  $p$  is the number of tokens in the input prompt,  $o$  is the number of generated output tokens, and  $\theta$  is a set of (learnt) parameters specific to a particular model, software, and hardware deployment.
In this section, we first specify the functional form of  $T$  based on the number of floating-point operations in the prompt encoding and token generation phases. We do not use the total number of floating-point operations to directly estimate runtime since every floating-point operation does not have the same "cost": in particular, the prompt-encoding phase is compute-bound while the token-generating phase is memory-bandwidth-bound for small batch sizes $^3$  (Cursor, 2023; Yu et al., 2022). We then validate the derived expression of  $T$  with experiments. Finally, we describe a procedure to estimate the  $\theta$  parameters from profiled runtimes; the profiled runtimes can be from a dedicated server deployment, or from black-box APIs.
3.1 Derivation of Cost Model
To generate  $o$  tokens,  $o - 1$  additional forward passes are needed (the first token is generated in the prompt encoding phase). The runtime of generating  $o$  tokens given a prompt with  $p$  tokens is the sum prompt_encoding_time  $(p)$  and output_generation_time  $(o)$ .
3.1.1 Number of Floating-Point Operations
To derive an expression for the end-to-end runtime of autoregressive inference, we first derive expressions for the number of floating-point operations required for each of the two steps.
In this paper, we are interested in the smallest possible inference runtime for a given query and assume inputs to the model are grouped into batches of size 1, but the batch size can be made larger in general to improve accelerator utilization and throughput at the cost of latency. Our cost model  $T$  can be adapted to accommodate other batch sizes  $b$ .
3Depending on the number of output tokens generated and the prompt size, end-to-end autoregressive inference becomes largely memory-bandwidth-bound as well; the larger the number of generated output tokens, the lower the effective throughput since computation is in the memory-bandwidth-bound portion of the computation for longer.
Prompt encoding. Transformer models consist of many Transformer layers, which themselves are composed of an attention layer (which measures the "importance" of input tokens to each other) and a two-layer FFN in traditional formulations. As outlined in §A, the total number of operations that need to be run in the prompt encoding phase for a single prompt of size  $p$  is  $24ph^2l \left(1 + \frac{p}{6h}\right)$ , where  $l$  is the number of Transformer layers in the model and  $h$  is the hidden size.  $p$  is usually  $\ll 6h$ , so the number of compute operations needed to encode prompts is  $24ph^2l$ . We assume that the costs of projecting into vocabulary space in the output layer of the model and sampling the next token given the distribution  $\operatorname{Pr}(x_{i+1}|x_{1:i})$  are cheap compared to the other operators in the Transformer layer (Narayanan et al., 2021).
Output token generation. When using language models autoregressively to generate new text, all operators in the transformer model must be performed incrementally. Concretely, the key, query, and value transformations in the self-attention layer need to be performed for just the new token, and self-attention scores need to be computed between the new token and all previous tokens. We can compute the number of floating-point operations needed per Transformer layer to perform these computations. Let  $i$  be the number of tokens generated so far (i.e., we are trying to generate the  $(i + 1)^{\text{th}}$  token, including the prompt). The total number of compute operations needed to generate the  $(i + 1)^{\text{th}}$  token for a single prompt is  $24h^2l + 4ihl = 24h^2l\left(1 + \frac{i}{6h}\right)$  (see §A.2 for details). If  $i \ll 6h$ , which is largely true in practice (e.g., for OpenAI/davinci, the maximum context length is 2048 and  $h = 12288$ ), the number of floating-point operations to generate a new token is roughly independent of the number of tokens generated so far. When context windows become large (e.g., in excess of tens of thousands of tokens as seen in some newer models (Anthropic, 2023; MosaicML, 2023)), the cost of generating each new token is no longer independent of the number of tokens seen so far. In this paper, we focus on smaller context windows (<10k tokens) where  $i \ll 6h$ .
3.1.2 Final Parametric Form
Runtime can be expressed as the ratio of the number of floating-point operations and throughput:
$$
\text {p r o m p t} \_ \text {e n c o d i n g} \_ \text {t i m e} (p; \alpha) = \frac {(2 4 h ^ {2} l) p}{\text {p r o m p t} \_ \text {e n c o d i n g} \_ \text {t h r o u g h p u t} (p)} = \alpha_ {p} p \tag {1}
$$
$$
\text {o u t p u t} _ {\text {g e n e r a t i o n}} \text {t i m e} (o; \beta) = \sum_ {i = 2} ^ {o} \frac {2 4 h ^ {2} l}{\text {o u t p u t} _ {\text {g e n e r a t i o n}} \text {t h r o u g h p u t} (i)} = \beta (o - 1) \tag {2}
$$
Here,  $\alpha_{p}$  is the average runtime per prompt token (different for different prompt sizes  $p$ ) and  $\beta$  is the runtime to generate each additional output token. prompt_encodingthroughput initially increases as  $p$  and arithmetic intensity (Williams et al., 2009) increase; we approximate prompt_encoding_time  $(p)$  as piecewise linear. output_generation_throughput is constant independent of the token being generated; this means output_generation_time is a linear function of  $o$ . prompt_encoding通過put and output_generation_throughput can differ by an order of magnitude or more.
We break up the space of all possible prompt sizes into disjoint intervals  $(p_0, p_1], (p_1, p_2], \ldots, (p_{n-1}, p_n]$ .  $p_0$  is assumed to be the smallest possible prompt size of 0, and  $p_n$  is assumed to be the largest possible prompt size (i.e., the maximum context window size supported by the model). For each interval  $(p_j, p_{j+1}]$ , we have a corresponding parameter  $\alpha_j$  that needs to be estimated;  $\alpha_j$  is the average runtime per prompt token when the prompt size is in the interval  $(p_j, p_{j+1}]$ .
Summing Equation 1 and Equation 2 gives us an expression for the end-to-end runtime<sup>4</sup>:
$$
\begin{array}{l} T (p, o; \theta = (\alpha , \beta)) = \text {p r o m p t} _ {\text {e n c o d i n g}} \text {t i m e} (p, \alpha) + (o - 1) \beta \\ = \sum_ {j = 0} ^ {n - 1} \mathbb {1} \left(p _ {j} <   p \leq p _ {j + 1}\right) \alpha_ {j} p + (o - 1) \beta . \tag {3} \\ \end{array}
$$
To support very large context windows, the above equation needs to be modified from the sum of a piecewise linear function of  $p$  and linear function of  $o$  to the sum of a piecewise quadratic function of  $p$  and a quadratic function of  $o$ .
4For simplicity, we will use prompt_encoding_time in all subsequent equations since piecewise linear functions are clunky to write.
(a) OpenAI/davinci.
(b) MS+NV/TNLG v2.
Figure 2: End-to-end runtimes for different prompt sizes (shown in legend in terms of number of tokens) and models as the number of generated output tokens is varied using Megatron. We also show a dotted best-fit line.
(a) Anthropic/v4-s3.
Figure 3: Prompt encoding runtimes versus prompt sizes. We also show a dotted best-fit line.
(b) AI21/J1-Jumbo v1.
(c) OpenAI/davinci.
(d) MS+NV/TNLG v2.
Model (owner/name)	Provider	h	l	n	# Params (B)	# GPUs×GPU type
OpenAI/davinci	OpenAI	12288	96	96	175	8×80GB-A100
AI21/J1-Large v1	AI21 Labs	4096	32	32	6.7	1×80GB-A100
AI21/J1-Grande v1	AI21 Labs	5120	50	40	17	1×80GB-A100
AI21/J1-Jumbo v1	AI21 Labs	13824	76	96	178	8×80GB-A100
Cohere/XL v20220609	Cohere	8192	64	64	52	4×80GB-A100
Anthropic/v4-s3	Anthropic	8192	64	64	52	4×80GB-A100
MS+NV/TNLG v2	Microsoft	20480	105	128	530	24×80GB-A100
EleutherAI/GPT-J	Together	4096	28	16	6	1×80GB-A100
Yandex/YaLM	Together	10240	80	128	100	4×80GB-A100
BigScience/BLOOM	Together	14336	70	112	176	8×80GB-A100
Table 1: Models studied in this paper. We also specify the number of GPUs / GPU type used to estimate the default idealized runtimes. Different configurations are used with 32GB-V100 GPUs.
3.2 Empirical Validation
We can validate the above equations empirically on instantiations of state-of-the-art LLMs.
Setup. We use Megatron (NVIDIA), a high-performance GPU implementation of Transformer models with support for autoregressive inference<sup>5</sup>. For a given model, we used the minimum number of GPUs necessary to minimize cost. For example, 0penAI/davinci cannot fit on a single 80-GB A100 GPU; we use tensor model parallelism (Shoeybi et al., 2019) to ensure that the model parameters fit in GPU memory in such cases. Tensor model parallelism works well within a multi-GPU server (Narayanan et al., 2021) since expensive all-to-all communication collectives like all-reduce are limited to fast high-bandwidth NVLink. For even larger models like MS+NV/TNLG v2, we need other forms of parallelism like pipeline model parallelism in order to fit the model in GPU memory without poor scaling. We used NVIDIA HGX servers with 8 NVIDIA A100 SXM4 80GB GPUs; A100 GPUs were the fastest widely available GPU as of October 2022, when we did this work.
$^{5}$ CUDA version 11.5.0, Megatron commit hash e156d2f and fp16 precision.
We evaluate 10 models, ranging in size from 6 to 530 billion parameters. Evaluated models are available in different ways: some were public via a commercial API (e.g., OpenAI/davinci, AI21/J1-Jumbo v1), some were private but the model owner provided research access for this effort (Anthropic/v4-s3, MS+NV/TNLG v2), and some were public and free (e.g., BigScience/BL00M) and were run using the Together API<sup>6</sup>. We do not evaluate models with withheld model architecture details (e.g., ChatGPT). Table 1 shows the full set of evaluated models, along with the key hyperparameters released by the respective model owner that determine their size.
Results. Figure 2 shows the end-to-end runtime measured using the above setup, versus number of generated output tokens for different prompt sizes and models. We instantiate models based on reported architectures, but with random (untrained) parameters, as we only care about estimating runtime, and runtime is independent of the model's parameters given a prompt size and number of output tokens. We randomly sampled 4 prompt sizes ( $\{1,512,1024,1536\}$ ) from the space of all possible prompt sizes and 7 different number of output tokens ( $\{1,2,4,8,16,32,64\}$ ). Runtime was averaged over 100 prompts of the same size. For each  $p$ , we compute a best-fit line using linear regression. We observe that the coefficients of determination  $(R^2)$  for the resulting time estimates are very close to 1.0 ( $>0.999$ ) for all models and conclude runtime shows a linear relationship with the number of output tokens for each prompt size (i.e., output_generation_time is a linear function of  $o$ ).
Runtime also increases with prompt size. Figure 3 shows the prompt encoding time (i.e., runtime when the number of generated output tokens is set to 1) versus prompt size  $(p)$  for 4 models. We show prompt encoding runtimes for all  $p \in \{1, 32, 64, 128, 192, \ldots, 1856, 1920\}$ . Runtime and the prompt size have a roughly linear relationship, especially at large prompt sizes (shown by the dotted best-fit line). At small prompt sizes,  $\alpha_{p}$  changes quickly; consequently, points deviate from the best-fit line on the left side of each figure.
3.3 Estimation Procedure
Equation 3 provides an expression for the end-to-end runtime of autoregressive Transformer LLMs for arbitrary prompt size  $p$  and number of generated tokens  $o$  that depends on  $(m, s, h)$ -specific parameters ( $m$  is model,  $s$  is software,  $h$  is hardware). This suggests an efficient way of estimating  $\theta = (\alpha, \beta)$  and consequently the runtime of a query with given prompt size and number of output tokens on a target system from a small number of profiled runtimes.
For each model and target system, we follow a two-step process. First, for each prompt size  $p \in \{p_0, p_1, \ldots, p_n\}$ , we profile the autoregressive Transformer LLM with numbers of output tokens equal to 1 to get the corresponding  $\alpha_j = \text{runtime}_j / p_j$ . Equipped with these  $\alpha_j$  values, we leverage the fact that total_runtime  $(p, o) - \text{prompt_encoding_time}(p)$  is a linear function in  $o$  to fit a single linear regression model with  $y = \text{runtime difference}$  and  $x = o - 1$  to estimate slope  $\beta$ . In practice, we profile the full cross product of a set of prompt sizes and number of output tokens (i.e.,  $(p, o) \in \{p_0, p_1, \ldots, p_n\} \times \{o_1, o_2, \ldots, o_m\}$ ) and use the resulting runtimes to estimate  $\alpha$  and  $\beta$  for the target model and system.
4 Idealized and Denoised Metrics
We now propose two runtime metrics that can be efficiently estimated using the cost function  $T$ .
Idealized runtime. The idealized runtime is the runtime of an inference query assuming a particular model architecture, software and hardware implementation (e.g., NVIDIA A100 GPUs and Megatron respectively). It allows for the inference efficiency of models to be directly compared with each other.
$$
T _ {(m, s, h)} ^ {\text {i d e a l i z e d}} (p, o; \left(\alpha_ {(m, s, h)} ^ {\text {i d e a l i z e d}}, \beta_ {(m, s, h)} ^ {\text {i d e a l i z e d}}\right)) = \text {p r o m p t} _ {-} \text {e n c o d i n g} _ {-} \text {t i m e} \left(p, \alpha_ {(m, s, h)} ^ {\text {i d e a l i z e d}}\right)) + (o - 1) \beta_ {(m, s, h)} ^ {\text {i d e a l i z e d}}). \tag {4}
$$
Denoised runtime. We also propose a runtime metric that factors out the noise from performance variation when using black-box APIs. We call this the denoised runtime; it is the best-case runtime of a particular query when given access to the same software and hardware used by the API provider.
$$
T _ {m \text {o n} \mathrm {A P I} a} ^ {\text {d e n o i s e d}} (p, o; (\alpha_ {m \text {o n} a} ^ {\text {d e n o i s e d}}, \beta_ {m \text {o n} a} ^ {\text {d e n o i s e d}})) = \text {p r o m p t} _ {\text {e n c o d i n g}} \text {t i m e} (p, \alpha_ {m \text {o n} a} ^ {\text {d e n o i s e d}}) + (o - 1) \beta_ {m \text {o n} a} ^ {\text {d e n o i s e d}}. \tag {5}
$$
$^{6}$ https://together.ai.
(a) Denoised runtime vs. raw runtime.
(b) Idealized runtime vs. denoised runtime.
Figure 4: Denoised vs. raw runtime and idealized vs. denoised runtime for various models across a range of queries along with a dotted  $y = x$  line in log scale on the left and linear scale on the right (for both Figures 4a and 4b). OpenAI model points are shown in green, AI21 Labs points are shown in red, and points corresponding to other models are shown in black.