SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) AI21/J1-Grande v1.
(c) AI21/J1-Jumbo v1.
Figure 7: Per-instance runtimes using black-box APIs to access LLMs for multiple instances (prompt size,  $p = 512$ ).
(a) AI21/J1-Large v1.
(c) Cohere/XL v20220609.
(b) AI21/J1-Grande v1.
(d) Anthropic/v4-s3.
Figure 8: Median (dotted line) and min / max (lower and upper boundaries of shaded region) end-to-end runtimes across 5 trials for different prompt sizes (shown in legend in terms of number of tokens) as the number of generated output tokens is varied using black-box APIs. End-to-end runtimes show variation across the 5 trials for a given prompt size and number of output tokens.
Figure 9: Minimum runtime across 10 trials as number of parallel queries increases for the Anthropic/v4-s3 model. Prompt size is 512 tokens and the number of output tokens is varied (shown in legend). Experiment was run in 10/2022.
Variation of runtimes across trials. To better quantify performance variability when using black-box APIs, we run multiple trials of synthetic queries where we control the size of the prompt and the number of generated output tokens. Figure 7 and Figure 8 shows per-trial runtimes for different model offerings; Figure 7 shows runtimes for models from the same model provider (AI21). Unless otherwise noted, all experiments in this paper were run in September or October 2022 with the latest API versions available at the time.
We see discernible performance variance across multiple trials for different models, across prompt sizes and number of generated output tokens. Certain models experience higher performance variability: Figure 7 shows AI21/J1-Grande v1 has much higher performance variance than AI21/J1-Large v1 or AI21/J1-Jumbo v1 (larger spread among points for a query of given size). AI21/J1-Grande v1 has an average coefficient of variation of about 0.55 compared to much smaller coefficients of variation ( $\sim 0.2$ ) for the other AI21 models. Even for models with lower spreads (e.g., AI21/J1-Large v1), we see that outlier points can have as much as  $3 \times$  higher runtime.
Figure 8 also interestingly shows that this performance variance can obfuscate the linear relationship between model runtime and number of output tokens.
Variation of runtimes with load. To understand the impact of load on performance contention and end-to-end runtime, we measured query runtime as we increase the number of queries sent in parallel to the various black-box APIs. Figure 9 shows runtime versus number of parallel queries for different numbers of output tokens and a fixed prompt size of 512 tokens for the Anthropic/v4-s3 model. We observe as much as a  $2 \times$  increase in runtime, indicating that load can lead to increased contention on API servers and consequently increased observed runtime.
B.2 Accuracy of Runtime Estimation Procedure
Model (owner/name)	R2
OpenAI/davinci	0.976
AI21/J1-Large v1	0.987
AI21/J1-Grande v1	0.916
AI21/J1-Jumbo v1	0.995
Cohere/XL v20220609	0.995
Anthropic/v4-s3	0.997
(a) AI21/J1-Large v1.
(c) AI21/J1-Jumbo v1.
(b) AI21/J1-Grande v1.
(d) Cohere/XL v20220609.
Figure 10: Minimum end-to-end runtimes for different prompt sizes (shown in legend in terms of number of tokens) as the number of generated output tokens is varied using black-box APIs, along with a best fit line estimated using the linear regression method described in §3.3 of this paper.
Table 2: Models and coefficient of determination  $(R^2)$  of time estimates for end-to-end text generation for various models using black-box APIs.
Model (owner/name)	Metric	prompt_encoding_time (p = 512/1024/1536)	Per-output-token generation time (β)
OpenAI/davinci	tidealized	0.178 / 0.323 / 0.476	0.081
tmdenoised	0.045 / 0.033 / 0.142	0.030
AI21/J1-Grande v1	tidealized	0.097 / 0.190 / 0.298	0.038
tmdenoised	0.172 / 0.351 / 0.519	0.021
AI21/J1-Jumbo v1	tidealized	0.164 / 0.310 / 0.465	0.064
tmdenoised	0.268 / 0.463 / 0.655	0.042
Anthropic/v4-s3	tidealized	0.108 / 0.189 / 0.279	0.054
tmdenoised	0.193 / 0.191 / 0.380	0.057
Table 3: Models and estimated prompt encoding times / per-output-token generation times (in seconds) for  $t_{(m, \text{Megatron}, A100)}^{\text{idealized}}$  and  $t_m^{\text{denoised}}$ .
We computed the coefficients of determination for runtimes of queries passed through black-box APIs; Table 2 and Figure 10 show the results. Despite performance variance, we see that the estimated
runtimes using the methodology based on linear regression are fairly accurate ( $R^2 > 0.9$ ), giving us further confidence that our cost model is accurate.
Table 3 compares the learnt performance parameters for  $t_{(m, \text{Megatron}, \text{A100})}^{\text{idealized}}$  and  $t_m^{\text{denoised}}$  for a subset of the considered models. As noted above, the estimated "(Megatron, A100) idealized" parameters for the AI21 Labs and OpenAI models are higher than the estimated denoised parameters, indicating that both these providers use optimizations not present in the software stack we considered.
B.3 Comparison to Exhaustive Profiling
We can model the exhaustive profiling costs for computing idealized runtime and denoised runtime separately. For the idealized runtime, we can compare the cost of exhaustively running all queries in the local environment with estimating and then using the cost model (the approach proposed in §3.3). We run each query multiple times to get a reliable runtime measurement. In both cases, we need to run all queries through the API once in order to get the prompt size and number of generated tokens for each query.
We use the following notation:
-  $t$  : Number of trials.
-  $Q$ : Set of user queries.
-  $\hat{Q}$ : Set of calibration queries  $(|\hat{Q}|\ll |Q|)$ .
-  $c_{\mathrm{API}}(q)$ : Cost of invoking API on query  $q$ .
-  $c_{\mathrm{local}}(q)$ : Cost of invoking local model on query  $q$ .
This produces the following expression:
$$
\text {I d e a l i z e d r u n t i m e s a v i n g s} = \frac {t \cdot \sum_ {q \in Q} c _ {\text {l o c a l}} (q) + \sum_ {q \in Q} c _ {\text {A P I}} (q)}{t \cdot \sum_ {q ^ {\prime} \in \hat {Q}} c _ {\text {l o c a l}} \left(q ^ {\prime}\right) + \sum_ {q \in Q} c _ {\text {A P I}} (q)}.
$$
We can compute the savings concretely on the 4 HELM tasks evaluated in §5.3. We use the published per-token costs from OpenAI for OpenAI/davinci and the per-hour cost to rent an 8-A100 server from AWS (Amazon). With 50 trials per query (in practice, fewer trials are probably sufficient), we observe cost savings of  $57 \times$  when using our cost model compared to exhaustive execution.
We can derive a similar expression to estimate the savings from using our approach for the denoised runtime, with the main difference being calibration queries now need to run through the black-box API, instead of on dedicated hardware.
$$
\text {D e n o i s e d r u n t i m e s a v i n g s} = \frac {t \cdot \sum_ {q \in Q} c _ {\mathrm {A P I}} (q)}{t \cdot \sum_ {q ^ {\prime} \in \hat {Q}} c _ {\mathrm {A P I}} \left(q ^ {\prime}\right) + \sum_ {q \in Q} c _ {\mathrm {A P I}} (q)}.
$$
Using the same parameters as above, we get a savings of nearly  $50 \times$  using the cost model compared to exhaustive execution (close to  $t$  since  $|\hat{Q}| \ll |Q|$ ).
B.4 Case Study: Using the Cost Model to Compare Different Hardware
Figure 11: Comparison of idealized metrics estimated on different hardware.
Figure 11 shows a comparison between Megatron on NVIDIA A100 GPUs (the default configuration in this paper) to Megatron on NVIDIA V100 GPUs (an older generation of GPU). While we expect these GPUs to be slower, we can also reasonably expect them to be cheaper (due to lower per-hour costs (Amazon)). In practice, we find that this is not the case, suggesting V100 GPUs are slower and more expensive. This is partially because we often have to use double the GPUs to fit the model parameters in GPU memory, since V100 GPUs only have 32GB of device memory compared to 80GB on the A100 GPUs. This is an example of an analysis that we can quickly perform by just running calibration queries on V100 GPUs and fitting linear regression models to the resulting profiled runtimes.