SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The runtime for generation using a black-box API is similar to Equation 3, but with an additional overhead term added to the prompt encoding and token generation runtimes. overhead captures the fixed costs of using an API to serve model predictions instead of using accelerators locally (e.g., round-trip latency of communicating with a remote API server) and performance variability (e.g., queuing delay or performance contention across requests). We quantify these overheads in §B.1.
Estimating runtimes of models run in a black-box system introduces additional complexity: runtimes using black-box APIs display higher variance relative to dedicated hardware ( $\S$ B.1) due to performance contention, which can make it hard to estimate  $\alpha$  and  $\beta$ . Since the variable performance overhead is a random variable  $\eta \geq 0$ , we run multiple trials for each  $(p,o)$  pair in the profiling step and then perform linear regression using the minimum runtime (i.e., the runtime with minimum variable overhead across trials) for each  $p$  and  $o$ . We see in  $\S$ B.2 that this procedure is able to effectively factor out the performance variation observed in raw runtime measurements from black-box APIs.
Idealized dollar cost. Thus far, we have only focused on runtime. However, another important consideration is cost: larger models often require more accelerators just to fit the model in accelerator memory and so comparing runtimes alone might not be fair. We propose two metrics that explicitly take into account the number of accelerators for serving. Both metrics are derived from the idealized runtime. Unfortunately, we cannot similarly modify the denoised runtime since we do not know the number of chips used by the model provider. We compute the idealized dollar cost as  $t_{(m,s,h)}^{\text{idealized}}$  (secs) ×  $n_{\text{accelerator} h} \times c_{\text{accelerator} h}$  ($/sec).
$n_{\mathrm{accelerator}h}$  is the number of accelerators used at a time to serve a single request (1 if not using model parallelism,  $>1$  otherwise), and  $c_{\mathrm{accelerator}h}$  is the per-unit-time cost of the hardware  $h$  (e.g., if  $h$  is NVIDIA A100 GPUs, then  $c_{\mathrm{accelerator}}$  could then be the per-hour cost of renting an NVIDIA A100 GPU on AWS and the idealized dollar cost is the cost of serving the model on AWS A100 GPUs).
Idealized energy cost. Similar to work that has examined the energy cost of training (Henderson et al., 2020; Strubell et al., 2019; Patterson et al., 2021), we can estimate the idealized energy cost as  $t_{(m,s,h)}^{\text{idealized}}$  (secs)  $\times n_{\text{accelerator } h} \times p_{\text{accelerator } h}$  (W).  $p_{\text{accelerator } h}$  is the power draw of hardware  $h$ . We can compare the idealized energy cost of running an inference query to the energy cost of training a model to better understand the number of inference queries needed to amortize training costs.
In this section, we seek to answer the following: (a) Is the proposed methodology to estimate inference runtime of autoregressive Transformer models accurate and efficient compared to exhaustive profiling? (b) Can this method reveal interesting insights about models' efficiency-capability tradeoffs?
5.1 Accuracy of Runtime Estimation Procedure
We computed the coefficients of determination for runtimes of queries passed through black-box APIs for the models in Table 1. Despite performance variance, we see that the estimated runtimes using the methodology based on linear regression are fairly accurate  $(R^2 >0.9)$ , giving us further confidence that our cost model for runtime is accurate (full results in Table 2 in §B.2).
Figure 4a compares denoised runtimes to raw runtimes for a range of prompt sizes and number of generated output tokens. Experiments were conducted in September and October 2022. We observe that raw runtimes for the most part (96.6% of points) are greater than the estimated denoised runtimes (below the  $y = x$  dotted line). Figure 4b is similar, but shows idealized runtime with A100 GPUs and NVIDIA's Megatron (Shoeybi et al., 2019) versus denoised runtime (same hardware and software setup as described in §3.2). In a number of cases, the idealized runtime is much lower than the denoised runtime, since the relevant API uses slower hardware and/or software implementations and other features like batching that sacrifice higher throughput for lower latency. For AI21 Labs models, idealized runtimes are greater than denoised runtimes 15.7% of the time. For OpenAI models, idealized runtimes are greater than denoised runtimes 64.2% of the time suggesting that they are using better hardware and / or a more optimized software stack. For all other model providers, idealized runtimes are always lower than the denoised runtimes, indicating that our hardware and software stack assumptions are fairly accurate.
5.2 Efficiently Evaluating Other Hardware or Software Optimizations
We can use the cost model proposed in this paper to efficiently evaluate the effectiveness of other hardware accelerators such as TPUs (Jouppi et al., 2017) or other generations of GPUs. As an example, Figure 11 in §B.4 shows how runtime and cost for various models is affected by using older NVIDIA V100 GPUs instead of NVIDIA A100 GPUs (the default hardware accelerator used in this paper). While we expect these GPUs to be slower, we can also reasonably expect them to be cheaper (due to lower per-hour costs (Amazon)). In practice, we find that this is not the case, suggesting V100 GPUs are slower and more expensive. This is partially because we often have to use double the GPUs to fit the model parameters in GPU memory, since V100 GPUs only have 32GB of device memory compared to 80GB on the A100 GPUs. This analysis requires profiling on the order of hours ( $< 2$  hours for most models, depending on the number of  $(p,o)$  values profiled) once, compared to hours per benchmark (depending on number of queries in the benchmark) for exhaustive profiling. As we show in §B.3, running a few calibration queries and then fitting the cost model to obtain  $(m,s,h)$ -specific parameters is much more efficient than exhaustive profiling. Our cost model also allows us to evaluate different software implementations and optimizations such as FasterTransformer (NVIDIA) or Flash-Decoding (Dao et al., 2023).
5.3 Efficiency-Capability Tradeoffs
We can now use the metrics proposed in this paper to evaluate the efficiency-capability tradeoffs of various language models accessible through black-box APIs. We focus on the few-shot evaluation setting adopted by BIG-Bench (Srivastava et al., 2023) and HELM (Liang et al., 2023). We consider four tasks in HELM: a sentiment analysis task (IMDB), two question answering tasks (MMLU [college chemistry] (Hendrycks et al., 2020) and BoolQ (Clark et al., 2019)), and a classification task (RAFT [terms of service] (Alex et al., 2021)). Figure 5 presents the results, with each row of graphs comparing average accuracy to a different efficiency metric. We highlight a few takeaways.
Effect of model scale. We observe that only a subset of the evaluated models fall on a Pareto frontier, with different models on the Pareto frontier for different tasks. This suggests that model scale alone does not predict model capabilities.
Heterogeneous software / hardware. The OpenAI/davinci model appears in the Pareto frontier for each benchmark when using raw runtimes but not the idealized metrics. This suggests that the OpenAI API implementation is more optimized than others: this could be due to a number of systems optimizations, such as query caching, or even model compression techniques like distillation, quantization or sparsification (Polino et al., 2018). Comparing these models on a level footing (same software and hardware) requires metrics that can factor out the effect of performance optimizations orthogonal to the model, such as idealized runtime.
Model architecture design. The relative positions of BigScience/BL00M and Yandex/YaLM on the idealized cost and floating-point operations (+ model size) graphs are sometimes reversed: while BigScience/BL00M achieves cheaper idealized cost (which takes into account the lower number of GPUs that Yandex/YaLM requires), Yandex/YaLM uses fewer floating-point operations. BigScience/BL00M's improved performance can be at least partially attributed to a more thorough
- OpenAI/davinci
AI21/J1-Large v1
Al21/J1-Grande v1
AI21/J1-Jumbo v1
Cohere/XL v20220609
Anthropic/v4-s3
Microsoft+NV/TNLG v2
EleutherAl/GPT-J
Yandex/YALM
BigScience/BLOOM
Number of floating-point operations
Denoised runtime (secs)
Idealized runtime (secs)
(a)IMDB.
Figure 5: Capability vs. efficiency tradeoff graphs. Capability is shown as accuracy on the target task. Six efficiency metrics are shown: model size (billions of parameters), per-query number of floating-point operations (FLOPs), raw runtime, denoised runtime, idealized runtime (all in seconds), and idealized cost (in cents). Idealized metrics were estimated on the hardware and software setup described in §3.2. Metrics are averaged over all scenario instances. Models on the Pareto efficiency frontier are shown as squares with a black dotted line connecting the points.
(b) MMLU, chemistry.
(c) RAFT, terms of service.
(d)  $\mathbf{BoolQ}$
search through model architectures for minimum runtime with a given number of floating-point operations in the forward pass (Scao et al., 2022).
Run-to-run variance. AI21/J1-Grande v1 often achieves worse raw runtime than AI21/J1-Jumbo v1 despite having  $10 \times$  fewer parameters, since the Grande model experiences higher performance variance (Figure 7). The idealized metrics make it easier to see the true efficiency-capability tradeoffs.
The denoised runtime metric can also help an end user reason through whether the observed inference performance is an artifact of performance contention.
Cost comparison. We can also compare these estimated inference costs to the costs charged by the black-box API provider. We observe that they are up to an order of magnitude lower than the charged actual costs. However, we note that these reported costs do not incorporate the significant cost of training models, which presumably gets amortized into the cost users pay with black-box APIs.
A large body of work has looked at studying the impact of model scale on model capabilities.
Scaling laws and other benchmarking efforts. By fitting a curve to dozens of training runs, scaling laws (Kaplan et al., 2020) show how the size of a model in a model family affects the training and validation loss of these models. While these scaling laws are instructive, we also care about the capabilities of models along other axes beyond validation loss (e.g., are models robust; do they exhibit stereotypes?). Moreover, large language models have been shown to exhibit emergent behavior that cannot easily be expressed as a continuous function of scale (Wei et al., 2022). Similarly, even though model size is used as a proxy for training and inference runtime performance, it is not useful when trying to answer questions like "Can model  $X$  meet a latency SLO of 100 milliseconds?" Consequently, empirical analysis is still needed to understand the capabilities of these models.
Floating-point operations and other proxy metrics for efficiency. The number of floating-point operations (FLOPs) required for the forward pass of a model has also often been used to estimate inference efficiency. While this is a fine approximation, it is not ideal for a couple different reasons. First, runtime does not correlate exactly with the number of FLOPs required (Scao et al., 2022). In particular, two operators with the same number of FLOPs could be executed with different efficiencies if one of the operators involves more memory accesses, preventing execution at peak device throughput. Second, as with model size, the number of FLOPs is hard to interpret. LLMs are often part of larger applications, and the performance requirements of these applications impose runtime constraints on LLM inference. It is hard to translate FLOPs to something actionable.
Inference runtime estimation for other types of models. Typically, inference for ML models is straightforward: an input of a particular size is passed through the model, in the process generating intermediate outputs and eventually a final prediction from a single forward pass. The sizes of intermediate outputs do not change from input to input, resulting in negligible runtime variance. This consequently makes inference runtime estimation easy. However, LLMs are different: while the hidden size does not change from input to input, the prompt size (in number of tokens) can be different for various inputs; runtime also depends on the number of output tokens generated.
Carbon costs of ML computation. Many papers (Canziani et al., 2016; Cao et al., 2020; Henderson et al., 2020; Strubell et al., 2019; Bender et al., 2021; Patterson et al., 2021) have discussed the importance of quantifying the cost of training models, both from an energy and emitted  $\mathrm{CO}_{2}$  perspective. This is often possible because model providers are open about details on training necessary to compute these metrics (Black et al., 2022; Patterson et al., 2021). While recent work has emphasized the need for considering inference-time efficiency (Henderson et al., 2020; Bommasani et al., 2021, §5.3), information on inference-time costs of LLMs is more scant for a multitude of reasons (e.g., optimizations powering an API might be part of a company's competitive advantage).
This work presents a systematic study of inference efficiency for autoregressive Transformer models accessible through black-box APIs. We showed both analytically and empirically that the inference runtime for these models is the sum of a piecewise linear function of the prompt size and a linear function of the number of output tokens, and designed an idealized runtime metric that can be estimated efficiently with minimal extra profiling. We are hopeful that our work helps model creators make better informed decisions about long-term model investments spanning training and serving.
We thank the anonymous reviewers, Trevor Gale, Siddharth Karamcheti and Matei Zaharia for their help and feedback that improved this work. This work was supported in part by the AI2050 program at Schmidt Futures (Grant G-22-63429). This work builds on top of the Stanford HELM project; we are grateful to its many contributors. We also thank the following individuals at their respective organizations for the access, support, and/or credits required to evaluate their models:
AI21 Labs: Opher Lieber, Barak Lenz, Dan Padnos, Yoav Shoham.
- Anthropic: Ben Mann, Jackson Kernion, Deep Ganguli, Jack Clark, Dario Amodei.
- Cohere: Lewis Stott, Ellie Evans, Bill McCartney, Aidan Gomez.
- Microsoft and the Turing Academic Program: Payal Bajaj, Barun Patra, Ahmed H. Awadal-lah, Saurabh Tiwary, Eric Horvitz.
- OpenAI: Lama Ahmad, Miles Brundage.
AI21. AI21 Models API. https://www.ai21.com/studio.
Alex, N., Lifland, E., Tunstall, L., Thakur, A., Maham, P., Riedel, C. J., Hine, E., Ashurst, C., Sedille, P., Carlier, A., et al. RAFT: A Real-World Few-Shot Text Classification Benchmark. arXiv preprint arXiv:2109.14076, 2021.
Amazon. AWS Pricing for GPU Instances. https://app.holori.com/compare?max_price=2365.89545&min_gpu=8&company=1&tab=Compute.
Anthropic. Claude 2. https://www.anthropic.com/index/claude-2, 2023.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PiQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Michael, P., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., et al. On the Opportunities and Risks of Foundation Models. arXiv preprint arXiv:2108.07258, 2021.
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Masleh, N., Xiong, B., Zhang, D., and Liang, P. The Foundation Model Transparency Index. arXiv preprint arXiv:2310.12941, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Canziani, A., Paszke, A., and Culurciello, E. An Analysis of Deep Neural Network Models for Practical Applications. arXiv preprint arXiv:1605.07678, 2016.
Cao, Q., Balasubramanian, A., and Balasubramanian, N. Towards Accurate and Reliable Energy Measurement of NLP Models. Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing, 2020.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking Large Language models for News Summarization. arXiv preprint arXiv:2301.13848, 2023.
Figure 6: High-level schematic of a Transformer model with  $l$  Transformer layers generating text at inference time given a prompt "The brown fox jumps".
A Derivation for Number of Floating-Point Operations
This paper focuses on language applications for Transformer models, where the model input is text. The input text is first preprocessed into a sequence of tokens (e.g., words) through a process called tokenization. Feature representations for each token (obtained by passing one-hot representations of the tokens through an embedding layer) are passed through multiple Transformer layers. Inputs to each Transformer layer are typically 3-dimensional tensors of shape  $(b,s,h)$  where  $b$  is the microbatch size (number of sequences),  $s$  is the sequence length (number of tokens in each sequence), and  $h$  is the hidden size (dimensionality of the model). For simplicity, we denote inputs as  $X$ .
Transformer layers in language models use self-attention to allow tokens to "interact" with each other. We assume multi-head attention; other forms of attention like multi-query attention (Shazeer, 2019) slightly change the analysis.
Self-attention is composed of the following operations:
- Attention key  $(K)$ , value  $(V)$ , query  $(Q)$  transformations. Given input  $X$ , we perform matrix multiplications  $K = X \times W^{K}$ ,  $V = X \times W^{V}$ , and  $Q = X \times W^{Q}$ .  $W^{K}$ ,  $W^{V}$ , and  $W^{Q}$  are learned parameters.
- Attention score computation. Matrix multiplication  $Q \times K^T$ , followed by application of the softmax function to obtain score tensor  $Z$ . Each element  $Z_{ij}$  is an importance score between query token  $i$  and key token  $j$ . This is the primary mechanism that allows interaction across tokens in a sequence.
- Attention over value computation. Matrix multiplication of scores  $Z$  by values  $V$ .
The subsequent two-layer feed forward network (FFN) consists of two linear layers (implemented as matrix multiplications). For most models, this involves multiplying the output of the self-attention layer by a matrix with dimension  $h \times 4h$  and then multiplying the resulting output (after other operators like layer norm) by a matrix with dimension  $4h \times h$ . Figure 6 shows how these operators are connected in a typical "decoder-only" Transformer model.
We can now derive an expression for the number of floating-point operations in a typical forward pass through the model, using a similar form of analysis as prior work (Narayanan et al., 2021).
A.1 Training
We use the same notation as before:  $b$  is the microbatch size (number of sequences) and  $h$  is the hidden size of the model. In practice, the self-attention layer computation described in §A is performed with different parameter matrices  $W_{i}^{K}, W_{i}^{V}$  and  $W_{i}^{Q}$ . This is called running the self-attention layer with multiple attention heads. We assume that the Transformer model has  $n$  attention heads.
$s$  is the sequence length in terms of number of tokens. Inputs  $X$  to the Transformer layer have shape  $(b,s,h)$ . The Transformer layer's computation during training can then be reduced to the following matrix multiplication operations.
- Attention key, value, query transformations: These can be expressed as a single matrix multiplication of size:  $(bs,h)\times (h,3h)$ . Output is of size  $(bs,3h)$ .
- Attention score computation:  $bn$  batched matrix multiplications (BMMs), each of size  $(s, h/n) \times (h/n, s)$ . Output is of size  $(bn, s, s)$ .
- Attention over value computation:  $bn$  batched matrix multiplications of size  $(s, s) \times (s, h / n)$ . Output is of size  $(bn, s, h / n)$ .
- Post-attention linear projection: a single matrix multiplication of size  $(bs, h) \times (h, h)$  to coalesce outputs of  $n$  attention heads to a single per-sequence vector of size  $h$ . Output is of total size  $(bs, h)$ .
- Matrix multiplications in the MLP layer of size  $(bs, h) \times (h, 4h)$  and  $(bs, 4h) \times (4h, h)$ . Outputs are of size  $(bs, 4h)$  and  $(bs, h)$ .
Using the fact that a  $(m,n)\times (n,k)$  matrix multiplication needs  $2mnk$  floating-point operations, the total number of compute operations is to complete the forward pass through a Transformer layer during training is  $24bsh^{2}\left(1 + \frac{s}{6h}\right)$ . A Transformer model typically has  $l$  Transformer layers, resulting in a total of  $24bsh^{2}l\left(1 + \frac{s}{6h}\right)$  floating-point operations in a single forward pass through the model. For most LLMs,  $s\ll 6h$ , meaning our expression for the number of floating-point operations in a single training forward pass can be simplified to  $24bsh^{2}l$ .
A.2 Autoregressive Inference
We can similarly compute the number of floating-point operations needed to generate a single output token during autoregressive inference.  $i$  is the number of tokens generated so far (i.e., the  $(i + 1)^{\mathrm{th}}$  token, including the prompt, needs to be generated next). The operators to be run in each Transformer layer in this phase are:
- Attention key  $(K)$ , value  $(V)$ , query  $(Q)$  transformations: These can be expressed as a single matrix multiplication of size  $(b, h) \times (h, 3h)$ .
- Attention score computation:  $bn$  batched matrix multiplications (BMMs), each of size  $(1, h / n) \times (h / n, i)$  (only  $Q$  value for the latest token is used;  $K$  and  $V$  values accumulated over all tokens so far).
- Attention over value computation:  ${bn}$  batched matrix multiplication of size  $\left( {1,i}\right)  \times  \left( {i,h/n}\right)$
- Post-attention linear projection: a single matrix multiplication of size  $(b, h) \times (h, h)$ .
- Matrix multiplications in the MLP layer of size  $(b,h)\times (h,4h)$  and  $(b,4h)\times (4h,h)$ .
Consequently, the total number of compute operations needed to generate the  $(i + 1)^{\mathrm{th}}$  token is  $24bh^{2}l + 4bihl = 24bh^{2}l\left(1 + \frac{i}{6h}\right)$ .
In this section, we show additional results. In §B.1, we quantify the performance variation of using black-box APIs; in §B.2, we provide additional evidence that the estimation procedure described in §3.3 is accurate; in §B.3, we quantitatively show the cost benefits of our procedure compared to exhaustive profiling; and in §B.4, we show a case study of the type of quick analysis that this procedure can enable.
B.1 Performance Variation in Black-Box APIs
We first measure the performance variation seen while using black-box text generation APIs.
(a) AI21/J1-Large v1.