FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Authors: Lingjiao Chen
Matei Zaharia
James Zou
Venue: Accepted by TMLR ()

Abstract: The rapid adoption of large language models (LLMs) has led to a growing number of companies offering generative LLMs as callable services at varying costs. We find that popular generative LLM APIs, such as GPT-4, Gemini 1.5, and Claude 3.5, exhibit heterogeneous pricing structures, with fees that can differ by two orders of magnitude and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs to use for different queries to reduce cost and improve accuracy. Our experiments demonstrate that, for a range of natural language tasks including news classification, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to a 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMs sustainably and efficiently.

Published in Transactions on Machine Learning Research (12/2024)
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Lingjiao Chen
Department of Computer Science Stanford University
Matei Zaharia
matei@@berkeley.cco edu
Department Electrical Engineering and Computer Sciences University of California, Berkeley
James Zou
jumeszOstan edu
Department of Biomedical Data Science Stanford University
Reviewed on OpenReview: /ttps:///pperevetee net/ forum? id- cStnku5p6R
Abstract
The rapid adoption of large language models (LLMSs) has led to growing number of com- panies offering generative LLMs as callable services varying costs. We find that popular generative LLM APIs, such as GPT-4, Gemini 1.5, and Claude 3.5, exhibit heterogencous pricing structures, with fees that can differ by two orders of magnitude and heterogeneous performance across tasks and input queries. This makes it challenging for users to decide which generative LLM APIs to utilize for their applications and budget. Motivated by these findings, we propose FrugalGPT, an algorithmic framework that adaptively selects which generative LLMs use for different queries reduce cost and improve accuracy. Our experiments demonstrate that, for range of natural language tasks including news classi- fication, reading comprehension, and scientific question answering, FrugalGPT can match the performance of the best individual generative LLM (e.g., GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% at the same cost. The ideas and findings presented in this paper lay a foundation for using LLMSs sustainably and efficiently.
1 Introduction
We are currently witnessing surge in the adoption of generative large language models (LLMs). The en- ticing potential of generative LLMs has led to a growing number of companies (such as AT21, Anthropic, Google, OpenAl, etc.) offering LLMs as callable services. Consequently, ML practitioners now frequently build applications by invoking these foundation models. For example, Tweet sentiment analysis is an of- ficial use case of ChatGPT (OpenAT 2024a), Strabag uses Microsoft services predict construction site risks Microsoft] 2024 and Stripe uses GPT-4 detect fraudulent behavior OOpenAl 2024b).
However, practitioners often face challenges in deciding which generative LLM services to utilize for their applications and optimizing their budgets. The cost of generative LLM services vary by up to two orders of magnitude: for instance, the prompt cost for 10M tokens is $300 for OpenAl's GPT-4 Turbo but only $1.5 for Google's Gemini 1.5 Flash 8B (as shown in Table 3. Smaller LLMs are generally more affordable, but their performance comparatively limited (as depicted in Figure d)). Larger LLMs like GPT-4 Turbo offer better performance but the risk of escalating costs. In addition their financial burden, employing larger LLMs incurs significant environmental and energy impacts (Bender et al. 2021 Wu et al.] 2022
Schwartz
al.
2020

---

Published in Transactions on Machine Learning Research (12/2024)
Llama (70B) Tie Better
GPT-4 Turbo Better
GPT-4 Turbo
Query
Answer
5%
87%
8%
Existing LLM Usage
(c) LLM pairwise comparison
FrugaiGPT
Llama 3
GPT-4
0.88
(70B)
Turbo
0.87
$1.5
accept
Query
Accuracy
0.85
GPT-4Turbo
Gem GPT-4o-mini
0.84
Gem 8BClaude 3.5
Answer
0.83
GM2
GPT-40
accept
0.82
Liama Gemini Pro
Budget
Jamba 1.5
accept
0.8
10
15
20
Cost ($)
(b) Proposed FrugaIGPT
(d) Performance and cost tradeoffs
(a) The standard usage sends queries to single LLM (e.g., GPT-4 Turbo), which can be expensive. (b) FrugalGPT, adaptively decides which LLMs to trigger for different user queries to reduce the inference cost. By optimizing the selection of different LLM APIs (e.g,, Gemini Flash, Llama 3 (70B), and GPT-4 Turbo), we can achieve substantial efficiency gains. (c) LLM performance breakdown on HEADLINES (a financial news dataset). Llama 3 (70B) outperforms GPT-4 Turbo on 5% of queries and produces identical generations on 87% of queries. (d) FrugalGPT can reduce the inference cost by 98% while exceeding the performance the best individual LLM (GPT-4 Turbo) on HEADLINES. This bocause FrugalGPT successfully learns data subsets on which inexpensive LLMs like Gemini Flash 8B are good better than GPT-4 Turbo, and directs these data to the corresponding low-cost LLMs only.
Figure 1:
Comparisons of different approaches using LLM services.
In this paper, we empirically demonstrate that for many of the lasks that generalive LLMs are used for, il is possible to evaluate a result's 's quality using an inerpensive model. For example, we found that DistillBERT accurately predict the answer quality of many LLMs including GPT-4 Turbo and Llama 3 (70B) on common natural language tasks like classification and question answering. Furthermore, generative L.L is "universally" superior to others. Take the task of classifying price sentiments from news headlines as an example (Sinha & Khandait 2021 There are 5% of queries where Llama 3 (70B) is entirely accurate while GPT-4 Turbo makes errors, and 87% of queries where both models provide identical responses (as illustrated in Figure (()).. Directing 92% (=87%+5%%) of queries Llama 3 (70B) and the remaining 8% GPT-4 Turbo is considerably more cost-cffective and performant than relying solely on GPT-4 Turbo. These discoveries suggest the possibility of routing queries adoplively to different LLMs both lower the cost and enhance the performance of LLM applications.
find that
Inspired by these findings, we propose FrugalGPT, an algorithmic framework that adaptively determines which LLMs to use given a user's budget. FrugalGPT first learns a generation judger that assigns a score to indicate the quality different LLMs' generations for any given query. It then invokes a list of LLMs soquentially until the judger's score for an answer surpasses a threshold. For example, FrugaIGPT may initially call Google Gemini 1.5 Flash 8B obtain an answer. the judger's score for this answer lower than threshold of 0.9, Llama. 3 (70B) is subsequently invoked to generate
response. The judger's
2

---

Published in Transactions on Machine Learning Research (12/2024)
score for Llama 3's answer exceeds threshold of 0.8, so no further LLMs are needed, and Llama 3's answer is returned to the user. Otherwise, GPT-4 Turbo is called to generate the final answer. We developed an efficient optimization technique to determine the optimal order of LLMs to call and the stopping threshold for each LLM as the core of FrugalGPT.
To demonstrate the potential of FrugalGPT, we implement and evaluate on various tasks, such as news classification, reading comprehension, and scientific question answering, using real-world LLMs, includ- ing Claude 3.5 (Anthropic 2024 Gemini 1.5 (Team et al. 2023 GPT-4o (Hurst al. 2024]. GPT-4
Turbo [OpenAl 2023b), Jamba 1.5 Large (Team et al. 2024), and LLama 3 (70B) (Dubey et al. 2024 Our experiments show that FrugalGPT can save up to 98% of the inference cost of the best individual LLM API while matching its performance on the downstream task. On the other hand, FrugalGPT can improve performance by up to 4% at the same cost. This is because FrugalGPT accurately identifies queries on which some inexpensive LLMs are correct but the most powerful LLM (e.g., GPT-4) incorrect, and directs these queries exclusively to the low-cost LLMs. We have also released the code and datascts usod in our experiments al /https://gtt htt.ggtth...oo.eeeeeeeeeeeeeeeee stanf ood ottp://sttaaoodddouuuuuttttttt/e fut.dduuuuuurrattarraaa... way for optimizing LLMs' inference cost and performance.
We hope FrugalGPT paves the
2 Related Works
Model Ensembles.
Model ensembles (Dong et al.
2020 Ganaie et al.
2022 which involve combining
multiple ML models for prediction, have gained popularity in supervised learning Garrii-Peedraaas 2009
Friedman 2002), unsupervised learning (Yang et al. 2014), semi-supervised learning Guppa et al.) (002)) and weakly supervised learning (Diba et al., 2017). Recent work (Arora et al. 2022 shows that fusing multiple gencrations from GPT-J (Wang & Komatsuzaki
2021 can compote with GPT-3's performance, and syn- thesizing multiple open-source LLMs' generations leads to better performance than individual LLMs (Jiang et al. 2023). Model ensembles typically require white-box access to multiple models for training, but LLM APls are often black-box. Moreover, model ensembles necessitate querying all models for any single query, thereby increasing costs.
MI-as-a-Service and Cascades. Generative LLM APIs constitute a crucial component of the rapidly expanding beodiine-lermiigg-a---seee
(MLaaS) industry. Recent studies have demonstrated the diversity
of different ML APIs' predictions (Buolamwini & Gebru, 2018 Koenecke et al.
2020 Chen et al. 2022b a
The concept of using multiple services for speed is known as model cascade (Viola & Jones] 2004). which has
been applicd in predictive ML domains such pedestrian detection CCa et al.) 2015). analytic systems (Kng
et al. 2017) and facial recognition (Li et al. 2015 Sun el al.] 2013 Recent work (Chen et al. 2020 2022c)
builds customized cascade for cost reduction, with a focus on classification ML APIs. However, their approach needs to estimate the performance of an ML API without querying it, based on simple signals such as labels from a proxy model. Such pre-query estimation is challenging for generative LLM APIs, whose outputs encompass a much larger space. FrugalGPT overcomes this by creating post-query quality estimator. Furthermore, for a given query, previous work invokes at most two APIs, while FrugalGPTT allows invoking three more given the vast number of LLM APIs. This renders it computationally more challenging to find the best calling strategies, and thus we also develop novel techniques to identify the optimal strategies efficiently (Section g
Speculative Decoding, Speculative decoding has recently emerged for LLM inference acceleration without
retraining or model architecture modification (Leviathan al. 2023; Chen et al. 2023 Sun et al. 2023
Spector & Re 2023 Liu ct al.) 2023). Its goal to provide the same output as a large LLM at lower latency. It relies on inexpensive LLMs for most generations and switches to costly LLMs when necessary. However, it requires access to the decoding module, which is not available for proprietary LLMs like GPT-4, and because it aims to give the same answer as the large LLM, it missos the opportunity to provide a better answer in cases where the small LLM is more accurate.
The remainder of the paper is organized as follows. We start by offering more context and the problem statement in Section 3 We present how FrugalGPT works in Section + Section 5 shows the empirical benefits of FrugalGPT using real-world LLM APIs (including GPT-3, ChatGPT, and GPT-4). We discuss future prospects in Section
3

---

Published in Transactions on Machine Learning Research (12/2024)
3 Scope and Problem Statement
Natural language query answering. In this paper, we concentrate on the standard natural language query answering task, where the objective is to answer query sampled from a natural language query distribution Q. Various real-world natural language tasks, such as news classification and commonsense reasoning, can be formulated as query-answering problems.
LLM marketplace. We consider answering queries via the LLM market, which comprises K different LLM APIs, denoted by {{()) Each fi(:) P is a function that, given a prompt p from the prompt space P, generates an answer from the answer distribution A. Note that to use LLM APIs, one has convert each query q to some corresponding prompt first. L.I.M APTs are associated with their own cost, typically consisting of three components: portion proportional to the length of the prompt, portion proportional to the length of the generated answer, and (somelines) fixed cost per query. Formally, given a prompl p, the cost of using the ith API is (x(p) - (.2l + &.1 + č,o, where J..j = 0, 1,2 are constants.
An illustrative example. Adapting the case study provided by (Kasser & Slowik 2023 assume a small business operates a customer service using GPT-4. The company caters to 15,000 customers each month, with each customer asking three questions twice a week, totaling 360,000 queries per month. Suppose for each question, its prompt averages 1800 tokens and the answer is around 80 tokens (as estimated by (Kaiser) & Slowik 2023)). ). Considering that the input and response costs of GPT-4 are $0.03 and $0.06 per thousand tokens, the total monthly cost amounts to 360 x ($0.03 x 1800 + $0.06 80) x $21.2K. Such high cost is prohibitive for many small businesses.
Problem statement: budget-aware LLM API usage. Our primary goal in this paper is leveraging LLM APIs wilhin a budgel construint. Formally, this can be formulated as maximixing the overall lask performance (x g)]l, while ensuring the average cost is bounded by a user-defined value b, i.., B(n.))@xxA q)] I b. Hore, a denotes the correct answer to the query g, ā(s, q) is the generated answer by some strategy 8 for query q, and c(s, q) is the associated cost for processing query q using strategy s. The reward function r(:-) measures how closely the generated answer aligns with the user query.
4 FrugalGPT: A Cost-aware Paradigm to Leverage LLMs
This section presents FrugalGPT, a cost-aware approach designed to harness the power of multiple LLM services. We begin by outlining the FrugalGPT pipeline and explaining the functionality of each component. Subsequently, we delve into the construction of the FrugalGPT pipeline for any user budget.
FrugalGPT Pipeline. FrugalGPT comprises three main components: an LLM router, an answer scorer, and a stop judger. Given a user query q, the LLM router is first invoked select an LLM obtain its response to the query. Next, the generation scorer takes the qucry, the answer, and the sclected LLM input and generates quality measurement output. Based the quality measurement and the invoked LLM service, the stop judger determines whether (i) to stop and return the answer, or (ii) to repeat the process of invoking the LLM router and generation scorer.
The LLM router consists of two parts. First, given the previously invoked LLM service k', it selects the next LLM service to use, denoted by k é (kk), where a : 1,,, K) -> 1,, K} a permutation of all LLM services (with Ø representing no invocation). Second, sends the query q to the kth service and obtains the generation (9) as output. Although the service permutation could depend on the input query in principle, consider the case of three models: GPT-4, GPT-Nco, and GPT-J. In this case K = 3. The LLM router may return GPT-J's generation for the first. time it is invoked. For the second and third time, il gives output by GPT-Neo and by GPT-4, respectively. For the fourth time and beyond, it simply returns empty. This corresponds to the permutation a (GPT-J) = GPT-Neo, a (GPT-Neo) = CPT-4.o(( (GPT-4) Ø. Another instance that the LLM router first invokes GPT-Nco, then GPT-J, and finally GPT-4. In this case, the permutation becomes a (GPT-Neo) = GPT-J. = CPT-4,0(GPP) =0
instantiation adopts query-agnostic permutation a(-) for simplicity. As an example,
4

---

Published in Transactions on Machine Learning Research (12/2024)
The generation scorer, denoted by 9.(:.)) Q A [0,1], generates quality measurement given query and an answer produced by the ith LLM API. Generally, the generation scorer can be any function such that its higher values strongly correlate with the input generation's quality. In our instantiation, we adopt a DistilBERT (Sanh et al. 2019) model tailored for regression as the gencration scorer, as it is smaller, cheaper, and faster than LLM services while still providing a reliable quality measurement. Specifically, we have added a linear layer on top of the original DistilBERT that takes the last representation layer (768- dimensional) as input and produces a 2-dimensional output encode the answer correctness. The maximum value of the last layer, normalized by softmax, is returned as the final score. We utilize the same embedding as DistiBBERT, ensuring compatibility and seamless integration. For each LLM service, we have trained the model weights with (i) the quety appended by the service's response as input features, and (ii) whether the response is correct as labels. We will present an ablation study on the generation scorer in Section S
The stop judger is responsible for deciding when to stop and return the answer to the user. As higher quality measurements indicate better generation quality, we use a threshold-basee stop judger: return answer a if the quality measurement gi gi a) is higher than threshold t; and go back to the router otherwise. The threshold vector controls the trade-offs between performance and cost: larger values often lead to better performance, while smaller values favor lower cost.
Joint optimization of the FrugalGPT Pipeline. Configuring the LLM router and stop judger appro- priately is erucial to FrugalGPT. Technically, we need to configure (i) the LLM router's service permutation function ot-) and (ii) the stop judger's threshold vector т. Our goal is to maximize the expected reward on the query distribution while satisfying the user budget. This problem can be formally modeled as the following optimization problem:
max E fa(() aUT
s.l. E
Σ
(z((1) sb.
(1)
(2),
E - (0) 92 f2(1)) T2,
Vt < ool' g fz (4)) ST
Here, the objective is the expected performance (reward), the first constraint ensures the average cost is bounded by the budget, the second constraint indicates that the stop judge returns the answer at the t-th iteration, and the last constraint indicates that the LLM router and the generation scorer are called repeatedly for previous iterations. I is a hyperparameter that controls the maximum number of L.MM services to call for a query. Solving this problem is inherently challenging because the optimization space is vastly large. a(-) is a permutation function over all possible LLM services, and exhaustive search takes O(K²) computations. Moreover, even if (-)) is fixed, the problem is non-convex with respect to the threshold vector T. In fact, we can show that even if the scorers are of high quality, the optimization problem is still NP-hard, formally stated as follows. We leave the proof in the appendix due to space constraints. Suppose the scorers are perfect, ie., (i(9.a) > gi(g,a") + r(g.a) > r(g. d') Then Problem I is an NP-hard problem.
To overcome this computational obstacle, we design specialized optimizer for this problem. It (i) prunes the search space of (-) by ignoring any consecutive selection of LLMs with small answer disagreement, and (ii) approximates the objective by interpolating it within few samples.
Search space pruning removes candidate permutation functions with relatively small maximum performance improvement, or MPI. Here, MPI is a function of two LLMs, ki k2, that measures al most how many mistakes ly incurs can be fixed by k1. Formally, MPI(k1, k2) & Prir(g. fe, (q)) > >r(g, fx,()) Suppose k is called from the last itcration in the cascade. Then in the next iteration, calling any LLMs with small MPI would not yield significant performance gains and thus could be avoided. Inspired by this, we introduce the following pruning condition
5

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Summary of commercial LLM APIs. We use 14 LLM APIs from 6 providers. The cost was retrieved in March 2023. The cost includes three components: imput (proportional to the number of input tokens), output (proportional to the number of generated tokens), and fixed cost per request. LLMs' costs can differ by up to 2 orders of magnitude. For example, to process 1M input tokens, GPT-J from Textsynth costs only 80.2, but OpenAl's GPT-4 costs $30.
Cost (USD)
Provider
API
Size/ /B
1M Input tok.
1M Output tok.
Request
GPT-Curie
6.7
2.00
2.00
ChatGPT
NA
2.00
2.00
0
OpenAI
GPT-3
175
20.00
20.00
GPT-4
NA
30.00
60.00
0
J1-Large
7.5
0.00
30.00
0.0003
A121
J1-Grande
17
0.00
80.00
0.0008
J1-Jumbo
178
0.00
250.00
0.005
Xlarge
52
10.00
10.00
Cohere
Medium
6.1
10.00
10.00
GPT-J
6
0.20
5.00
0
Textsynth
FAIRSEQ
13
0.60
15.00
GPT-Neox
20
1.40
35.00
Databricks Model Serving
Dolly
7
0.27
0.27
ForeFrontAl
QA
16
5.80
5.80
(k)) E E MPI(N", MPT(W, (PT((,))) R), for at most. other values of k" K}
(2)
That is to say, given the previously invoked LLM k the next. LLM to call must hold the top-m value MPT with respect to k. This reduces the search complexity from O(K4) to O(m²). In practice, we found that m = often suffices to identify a high-quality cascade.
Now suppose the function af-) is fixed. The remaining step is to find the optimal threshold vector This be resolved via two-stage approximation. First, divide the scarch space [0, 1]2h into few equal-size grids. Next, within each grid, we approximate the objective by quadratic function of the threshold vector, whose parameters are determined by the grid bound values. Then within each grid, we can leverage QP solver to find the optimal solution. The final solution is the best among all grids. The combination of the above two techniques provides an efficient implementation with satisfactory performance, as demonstrratd later Figure g
5 Experiments
In this section, we present an empirical study on FrugalGPT. Our goals are four-fold: (i) understand when and why FrugalGPT lowers the cost, (ii) quantify the cost savings attained by FrugalGPT while matching the best individual LLM API's performance, (iit) measure the trade-offs between performance and cost enabled by FrugaIGPT, and (iv) explore how different factors including data distribution shifts and scorer's quality affect FrugaIGPT.
Setup: LLM APIs, Tasks, Datasets, and FrugalGPT instances. We have selected 14 LLM APIs from 6 mainstream providers, namely, OpenAT (OpenAl 2023a), AT21 (A12T 2023), Cohere CCohere 2023 Textsynth [Textsynh] 2023), Databricks (Databricks) 2023), and ForeFrontAl (Forefront Al 2023 The pricing information and all empirical results with these models collected in March 2023, and results

---

Published in Transactions on Machine Learning Research (12/2024)
Yes
Financial News
GPT-J
Score=0
J1-L
Score-0
GPT4
No
(a) Learned FrugaIGPT strategy
Approach
Accuracy
Cost ($)
Nellagio rectieg
GPT-4
neural
GPT-4
85.7%
$33.1
FrugalGPT
price down
FrugaiGPT
87.2%
$6.5
(b) query and response example
(c) Overall performance and cost
GPT
18
16
ChatGPT
0
8
7
9
7
20
100
GPT-3
4
0
5
5
7
Query Percentage
100%
GPT-4
2
5
5
6
80
11-L
15
G1
7
60
6
0
5
6
Cohere
5 10 9
12
0
5
10
10
40
33.7%
18
20
FO
15
16
18
13
16
14
14
34
12
16.6%
GPT:
12 10
34
10
6
8
0
12
10
F5Q
15 16
GPT-J
GPT-4
GPT-Nee
LLM APIs
GPT-C
Chatopt
P.3
GPT4
Cohere Coteredd
Dolly
1D
GPT-
GPT-Neo
(d) Each LLM pair's MPI
(e) FrugalGPT's Query Percentage
Figure 2: A case study of FrugalGPT the HEADLINES dataset. (a) The learned FrugalGPT this dataset with overall budget of $6.5, onf-fifth of GPT-4's cost. FrugalGPPT avoids querying GPT-4 4 as long GPT-J and J1-L produce high-quality answers. (b) Sometimes GPT-4 makes mistake, but FrugalGPT learns to use the correct answers by J-1 and GPT-J. (c) Overall, FrugalGPT reduces the cost by 80%, while improving the accuracy by 1.5% compared to GPT-4. (d) The maximum possible improvement (MPT) for each LLM pair, measuring how often one LLM (each row) makes a mistake while another (each column) is correct. Even for the best individual LLM, GPT-4, cheap LLMs (c.g., GPT-J) can be better 6% of the data. (e) FrugalGPT sends only 16.6% queries uo GPT-4 and thus saves cost.
with newer models would be discussed later. The details are summarized in Table FrugalGPT has been developed top of these APIs and evaluated range of datasets belonging to different tasks, including HEADLINES (Sinha & Khandait 2021). OVERRULING (Zheng et al. 2021 COQA (Reddy et al. 2019 AGNEWS Zhang et al., (0155) and SCIQ (Webll et al. 2017). More details of the datasets and tasks can be found in the Appendix. We focus on FrugalGPTT with the hyperparameter = 3, as this simplifies the optimization space and shows exciting results. Each dataset is randomly split into a training set (50%) to learn FrugalGPT and a test sot for ovaluation (50%).
A Case Study. We begin with a case study on the HEADLINES dataset. We set the budget be $6.5. which is one-fifth of GPT-4's cost. As depicted in Figure 2 (a), the learned FrugalGPT sequentially calls GPT-J, J1-L, and GPT-4. For any given query, it first extracts answer from GPT-J. If the score of this answer is greater than 0.96, the answer is accepted answer is accepted as the final response if its score is greater than 0.37; otherwise, GPT-4 is invoked to obtain the final answer. Interestingly, this approach outperforms GPT-4 for numerous queries. For instance, given a headline "Gold prices trade near 3-month high as Fed begins meeting' from NASDAQ, FrugalGPT accurately predicts that the price is going down, while GPT-4 provides an incorrect answer (as shown Figure (b). Overall, FrugalGPT results in both accuracy gains and cost reduction, illustrated Figure 2 Mcc. This is partially because FrugalGPT only sends 16.6% queries to GPT-4 (see Figure 2 ee))
the final response. Otherwise, J1-L is queried. J1-L's

---

Published in Transactions on Machine Learning Research (12/2024)
Table 2: Cost (USD) savings by FrugalGPT to match the best individual LLM's performance.
Cost to reach the same accuracy (USD)
Dataset
Type
Best single LLM
Cost. Savings
Best single LLM
FrugalGPT
HEADLINES
4 way classification
GPT-4
33.1
0.6
98.3%
OVERRULING
2-way classification
GPT-4
9.7
73.3%
COQA
Free-form QA
GPT-3
72.5
29.6
59.2%
AGNEWS
4-way classification
GPT-4
64.6
15.9
75.4%
SCIQ
Free-form QA
GPT-3
132.4
63.1
52.3%
LLM diversity. Why can multiple LLM APIs potentially produce better performance than the best individual LLM? Similar to how ensemble methods can improve accuracy on many tasks, this often due to generalion diversity: even an inexpensive LLM sometimes correctly answer queries on which more expensive LLM fails. Recall that we introduce maximum performance improvement (MP)) in Section 1 pruming metric. It also measures the generation diversity well: larger value of MPI indicates that one generative LLM give more responses different from another one. As shown in Figure 2 (d), MPI is indeed large for many pairs of generative LLMs. For instance, there are 6% queries where GPT-4 incorrect but GPT-J (and GPT-C, JI-L, or Dolly) can give desired answers. This indicates the potential of combining multiple generative LLMs, and verilies why FrugalGPT offers cost reduction without performance drops.
Cost Savings. Subsequently, we examine if FrugalGPT can reduce costs while maintaining accuracy and, if so, by how much. Table 2 displays the overall cost savings of FrugalGPT,, which range from 50% to 98%. This is feasible because FrugalGPT identifies the queries that can be accurately answered by smaller LLMs and, as a result, only invokes those cost cest-effective LLMs. Powerful but expensive LLMs, such as GPT-4, are utilized only for challenging queries detected by FrugalGPT.
Performance and Cost Trade-offs. Now, we investigate the trade-offss between performance and cost achieved by FrugalGPT, as illustrated in Figure D Here we focus on three datasets due to space limitations; more results on other datasets can be found in the Appendix.
Several interesting observations can be made. First, the cost ranking of different LLM APIs is not fixed. For instance, J1 is the second most expensive LLM on the HEADLINES dataset, while GPT-3 holds that position on the OVERRULING and COQA datasets. This is primarily due to the heterogeneous pricing mechanism: J1 incurs a high cost for each generated token but charges nothing for input tokens, whereas GPT-3 charges for both input and output tokens. Moreover, more expensive LLM APIs sometimes result worse performance than their cheaper counterparts. For example, J1 costlier than GPT-3 on HEADLINES, but its performance is inferior. These obscrvations underscore the importance of aptly selecting LLM APIs, even in the absence of budget constraints. Next, note that FrugalGPT enables smooth performance-cost trade-offs across all evaluated datasets. This offers flexible choices to L.M users and potentially helps LLM API providers save energy and reduce carbon emissions. In fact, FrugalGPT can simultaneously reduce costs and improve accuracy. For example, on the OVERRULING dataset, FrugaIGPT achieves a 1% accuracy gain while reducing costs by 73% compared to the best LLM APT, GPT-4. This is likely because FrugalGPT integrates knowlodge from multiple LLMs.
The example queries shown in Figure g further aid in understanding why FrugalGPT can simultancously improve performance and reduce costs. GPT-4 makes mistakes some queries (e.g., the first example in part (a)), but some low-cost APIs provide correct predictions. FrugalGPT accurately identifies those queries and relies solely on the inexpensive APIs. For example, GPT-4 incorrectly infers no overruling from the legal statement 'The time has come to reconcile and regularize our cases in this field,' as shown in Figure g b). However, FrugalGPT accepts GPT-J's correct answer, avoiding the use of expensive LLMs and improving overall performance. Naturally, single LLM API is not always correct; FrugalGPT overcomes this by employing chain of LLM APIs. For example, in the second example shown in Figure g aa, FrugalGPT
8

---

Published in Transactions on Machine Learning Research (12/2024)
0.86
0.88
FrugeloPT
GPPT
Gold holds gains in Asia on risk-on sentiment over U.S. politics
GPT-J
0.99-0.66
0.84
GPT-3J1
GPT-4
0.882
GPT-NNo ChaiGPT
0.78
0.8
Cohere
Cohere- M
halds ground
GPTJ
0
J1
0:44 0.37
Down
0.76
J1-L
comments
GPT-4
Kouuraa
X
0.74
0.72
FSQ
DallyGPT-C
upgraded outperformer from neutral at CIBC
0.7% 277400
0.13 4- 0:37
GPT-4
None
10
20
GPT-4
None
Cost
(a) HEADLINES
PPT-
come regularize our this field. Q:
GPT-J
0.91
0.9
[CHATPPT
Cohere
GPT-4
X
Accuracy
overuulin??
0.7
Mep
JJ1
The court [..] was expressly overruled by this court lima, overruling?
Yes
0.6 0
Yes
0.0.0.
GPT-4
GPT-F
[-] denying available and accommodation.
ChatGPT
GPT-3
0.5
Cohere-M
Yes
0.2
No
0.8
No
6
10
12
Is overruling?
GPT-4
Cost (S)
(b) OVERRULING
[-] Cap Winters [-] added a thousand hairs to his
GPT-3
0.45
GPT-3
0.6
Accuracy
0.4
GPT-4
head Q: Did he
red hair?
GPT-4
does not mention this
The lest
0.35
OPTIEO
PPT-NNo
Cohere
[-)] told every Tuesday for their story time. [..]. Q:
GPT-3
0.3
01:00
0.8
0.
Tuesday
V
Cohere-M M
kamaschol
0.25
when did they
time free?
GPT-4
thair day off
from school
X
6 ATGTT
0.15
20
60
80
100
120
140
[-.] Q: What was the
black-walnut sheff
When [-.] little
Black wainut
GPT-3
blook walnud
biack-wamul
GPT-4
black walnuf
Cost
(S)
shelf made of?
GPT-4
waalkw
(c) COQA
Figure 3: Accuracy and cost tradeoffs achieved by FrugalGPT. Overall, FrugalGPTT often achieves the same performance of the best individual LLM API (e.g., GPT-4) with orders of magnitudes smaller cost. When incurring the same cost, FrugalGPT can improve the accuracy by up to 5%. Examples of FrugalGPT for each dataset are shown the right. We show similar performance-cosl tradeoff improvements for FrugalGPT for AGNEWS and SCIQ in the Appendix.
identifies that GPT-J's generation may not be reliable and turns to the second LLM in the chain, J1-L, to find the correct answer. Again, GPT-4 provides the wrong answer. FrugalGPT is not perfect, and there remains ample room for cost reduction. For example, the third example Figure S c). all LLM APIs in the chain give the same answer. However, FrugalGPT unsure the first. LLMs are correct, resulting the need to query all LLMs in the chain. How to avoid such cases is an interesting direction of future work.
Comparison with threshold baseline. To further understand the benefits of FrugalGPT, we also compare it against a simple thresholding method. In particular, first invokes GPT-J, and computes the log probability averaged over all output tokens. If the averaged log probability is larger than a threshold, GPT-J's response is accepted. Otherwise, GPT-4 is invoked. This can be seen variant of FrugalGPT with L = 2, except that the scorer is simply the first generator's own judgment. We evaluate this method with different threshold values on the HEADLINES dataset, and show its performance in Figure as the blue dashed line (the blue texts correspond to the threshold values).
There are several interesting obscrvations. First, this simple threshold method enables good tradcoff between performance and cost. In fact, the corresponding blue in Figure g is much higher than a straight line between GPT-J and GPT-4. This suggests that the log probability returned by GPT-J indeed reflects
9

---

Published in Transactions on Machine Learning Research (12/2024)
0.88
0.86
GPT-4
-0.8
-0.4
0.84
GGTT-B
0.82
GPT-Neo
Accuracy
ChatGPT
0.8
0.78
GPT-J
Cohere
-1.6
0.76
J11L
0.74
Simple Thres
0.72
FQ
FrugalGPT
0.7
GPT-C
10
20
30
40
Cost
Figure 4: Comparison of FrugalGPT with simple threshold bascline the HEADLINES datasct. Overall, observe that PrugalGPT consistently putperforms the threshold method given dilferent cost constraints.
how likely the answer correct. Second, threshold value in the middle (-0.8) leads the best performance. This again verifies our intuition that no single model is the best. Finally, FrugalGPT consistently outperforms this threshold method. For example, to match GPT-4's performance, this simple baseline reduces 60% cost while FrugalGPT reduces more than 90%. This is because FrugalGPT uses three models sequentially, and also uses an optimized generation scorer.
Table 3: Price comparison of models used for evaluation in 2024. The price was collected in Novem- ber 2024. As of November 2024, Claude 3.5, GPT-4o, L.aama 3 (70B), and Gamma 2 (9B) had dif- ferent versions, and in our experiments we used and gemma-2-9b-it offered by the correspouding providers, respec- tively. Overall, the prices are lower than those in 2023, but there is still large price difference between these new models. For example, the generation cost of GPT-4 Turbo is 200x of Google Flash 8B's cost.
gpt-4o-2024-05-11
Cost (USD) / 1M tokens)
Provider
Model
Input
Output
A121
Jamba 1.5 Large
2.00
8.00
Authropic
Claude 3.5 Sonnet
3.00
15.00
Gemini 1.5 Pro
1.25
5.00
Google
Gemini 1.5 Flash
0.075
0.30
Gemini 1.5 Flash 8B
0.0375
0.15
GPT-4o
5.00
15.00
OpenAI
GPT-4 Turbo
10.00
30.00
GPT-4o mini
0.15
0.60
Llama (70B)
0.88
0.88
Together Al
Gamma 2 (9B)
0.30
0.30
Evaluation using more recent models. While the main evaluation was conducted using models available in 2023, many new models with lower prices have been developed since 2024. To understand how FrugalGPT works with these newly developed models, we evaluate its performance using a few more recent models on the HEADLINES and SCIQ datasets. In particular, we use eight proprietary models, GPT-4-Turbo GPT- 4o, and GPT-4o-mini offered by OpenAI, Jamba 1.5 Large from AI21, Claude 3.5 Sonnet from Anthropic, Gemini 1.5 Pro, Gemini 1.5 Flash, and Gemini 1.5 Flash 8B from Google, and two open-source models, LLama 3 (70B) and Gamma. (9B) provided by Together AT. This is because these models show superior
10

---

Published in Transactions on Machine Learning Research (12/2024)
FrugalGPT
FrugalGPT
0.8
0.88
0.87
0.78
0.86
GPT-4o
GPT-4Turbo
Accuracy
0.85
GPT-4Turbo
Accuracy
0.76
0.84
Gem GPT-4o-mini
Gem 8BClaude 3.5
GM2
GPT-4o-mini
0.74
0.83
GM2
GPT-4o
0.82
Llama 3 Gemini
Gem
Jamba 1.5
0.72
Claude 3.5
0.81
Jamba 1.5
0.8
0.7
5
10 Cost ($)
15
20
0
20
40
60
80
100
Cost ($)
Figure 5: FrugalGPPTss performance on the HEADLINES (left) and SCIQ (right) datasets using more recent models. In particular, we use eight proprictary models, GPT-4-Turbo, GPT-4o, and GPT-4o-mini offered by OpenAT, Jamba 1.5 Large from A121, Claude 3.5 Sonnet from Anthropic, Gemini 1.5 Pro, Gemini 1.5 Flash, and Gemini 1.5 Flash 8B from Google, and two open-source models, LLama 3 (70B) and Gamma 2 (9B) provided by Together AI. We use the same prompts and temperatures for all these experiments. On SCIQ, omit three models since their accuracy is much lower than other models, namely, LLama 70B (24.1%), Gemimi Pro (53.9%) and Gemini Flash 8B (43.6%). FrugalGPP can reduce the cost by more than 90% while matching the best individual model's performance on these datasets.
performance and are widely used in many applications. Their prices are summarized in Table B collected in November 2024. Compared to Table I the overall prices are indeed lower, but there is still large price difference between these models.
Figure F shows the performance-cost tradcoff achieved by FrugalGPT using these new models on HEAD- LINES (left) and SCIQ (right). Overall, FrugalGPT consistently offers large cost savings. For example, to match GPT-4-Turbo's performance, it requires less than 10% of the cost. It is also interesting to note that expensive models again are not necessarily better than cheap models. For example, GPTT4o's performance is worse than CPT-4o-mini's on this task, although the former's price is much higher. This further underscores the importance of choosing models carefully even if the budget is large.
Performance Resilience to Data Distribution Shifts. A common challenge when deploying ML sys- tems in practice is data distribution shifts, i.e., the queries encountered during deployment differ from those in development. To understand the robustness of FrugalGPT against data distribution shifts, we trained Fru- galGPT on the original HEADLINES training data and evaluated its performance on four testing datasets with different distributions. Specifically, we created these testing datasets by altering the distribution of labels. For instance, in Variant the label distribution is 33% (up), 17% (down), 17% (none), and 33% (neutral). Conversely, the original dataset's label distribution is balanced (25% for each label). Details be found in Table 5 in the Appendix. As depicted in Figure 7 in the appendix, the performance of both FrugaIGPT and GPT-4 remains relatively consistent across different data distributions. Interestingly, while using only 10% of GPT-4's cost, FrugalGPT often delivers similar or superior performance compared to GPT-4 under several testing data distributions.
Effeets of Scorer Functions. The scorer plays crucial role in FrugalGPT. Therefore, it is essential to study how the scorer's quality impacts FrugalGPT'ss performance. In this regard, we focused on three backbones for the scorer with varying numbers of parameters: ALBERT (11M), DistilBERT (67M), and BERT (110M). We trained the scorer on the HEADLINES dataset using different backbone models and compared the performance of the resulting FrugalGPT, with budget of 10% of GPT-4. As illustrated
11

---

Published in Transactions on Machine Learning Research (12/2024)
in Figure [b), low-quality scorer (such ALBERT) indeed leads to limited performance, as expected. Conversely, larger scorers with better quality, such DistilBERT and BERT, offer higher performance.
6 Discussions and Future Prospects
The substantial cost of employing LLMs in real-world scenarios presents considerable barrier their widesproad usage. In this paper, we discovered that for many tasks on which LLMs are commonly used today, (i) small models can predict the quality of LLMs accurately, and (ii) no LLM is universally better than others. Based on these findings, we introduce FrugalGPT, our approach to resolving the cost challenge. Our empirical findings show that FrugalGPT can reduce costs by up to 98% while preserving the performance of cutting-edge LLMs.
There are many interesting directions for future exploration. For example, while FrugalGPT concen- trates on balancing performance and cost, real-world applications call for the evaluation of other crit- ical factors, including latency, fairness, and environmental impact. Incorporating these elements into optimization methodologies while maintaining performance and cost-effectiveness is an important avenue for future research. Furthermore, utilizing LLMs in risk-critical applications necessitates the careful quantification of uncertainly LLM-generated outputs. We have released our code and datasets at https://github. ative service utilization.
hoping to stimulate further research on gener-
References
AI21. AI21 LLM API. https://www... www..aii...c...o 2023. Accessod: 2023-03-31.
AI Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3, 2024.
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: simple strategy for prompting language models. ar preprint (rXim:2210.02444 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountabillty, and Transporency, pp. 610 623, 2021.
Joy Buolamwini and Timmit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In ACM Conference on Fairness, Accountability and Transparency, pp. 77 91, 2018.
Zhaowei Cai, Mohammad Saberian, and Nuno Vasconcelos. Learning complexity-awaro cascades for deep
pedestrian detection. In IEFE International Conference on Computer Vision, pp. 3361-3369, 2015.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv, 2023.
Lingjiao Chen, Matei Zaharia, and James Y Zou. FrugalML: How to use ML prediction APIs more accurately and cheaply. In Advances in Neural Information Processing Systems, volume 33, pp. 10685 10696, 2020.
Lingjiao Chen, Zhihua Jin, Evan Sabri Eyuboglu, Christopher Ré, Matei Zaharia, and James Y Zou. HAP1: A largo-scale longitudinal dataset of commercial ML API predictions. In Advances in Newral Information Processing Systems, volume 35, pp. 24571 24585, 2022a.
Lingjiao Chen, Matei Zaharia, and James Zou. How did the model change? Efficiently assessing machine learning API shifts. In International Conference on Learning Representation, 2022b.
Lingjiao Chen, Matei Zaharia, and James Zou. Efficient online ML API selection for multi- classification tasks. In International Conference on Machine Learning, volume 163, pp. 3716-3746 2022c.
Cohere. Cohere LLM API. https://cohere.ce 2023. Accessed: 2023-03-31.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Databricks. Dolly deployed on Databricks Model Serving.
http:://www..daaaaaraaaccccc//aa http:/iiiiwalelleeettte..ee htg.ap/ee......ii.o............a bprpddeeeeeeaalllaaeeeeeettttttr 2023.
Accessed: 2023-03-31.
Ali Diba, Vivek Sharma, Ali Pazandeh, Hamed Pirsiavash, and Luc Van Gool. Weakly supervised cascaded convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognation, pp. 914-922, 2017.
Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science, 14:241-258, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.211 2024.
Forefront AI. Porefront AI LLM API. http::///beaa...ffoooooa ai/ 2023. Accessed: 2023-03-31.
Jerome H Friedman. Stochastic gradient boosting. Computational Statistics 8 Data Analysis, 38(4):367-38) 2002.
Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
Nicolás Garcfa-Podrajas. Constructing ensembles of classifiers by mcans of weighted instance selection. JEEE Transaclions on Neural Nelworks, 20(2):258 2777,87777722 2009.
Ashit Gupta, Anirudh Deodhar, Tathagata Mukherjee, and Venkataramana Runkana. Semi-supervisod cascaded clustering for classification of noisy label data. arXiv preprint arXiv:2205.0220 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adain Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. orXiv preprint 0rXiv:2410.2197 21276 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Annual Meeting of the Association for Computational Linguistics, 2023.
Filip Kaiser and Claudia Slowik. Cost estimation of using GPT-3 for real applications.
http::/:hheoeerrrr.caaa//o//ooor tos-------------t--------------- ppotepttttootttteeealltll mmoells-gpte Srrpelssgggg---ppiiiingeeeeppppp
2023. Accessed: 2023-03-31.
Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, and Matei Zaharia. Noscope: optimizing neural network queries over video at scale. ar preprint orXiu: 123.0029,,, 2017.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14)76684-768) 2020.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decod- ing. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, and Gang Hua. A convolutional neural network cascade for face detection. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5325 5334, 2015.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijic Deng, Alvin Cheung, and Hao Zhang. Online speculative decoding. Xiviv preprint arXio:2310. 071777.777777 2023.
Microsoft. Strabag uses OpenAI service to predict risks.
https://eustomers.
http:///B88B20000020200--a2aeee0 wtw.estttteeaacceecpoooetttooeee mmanufacturring--zure www.:uuuuree-auuuiraaaaaa 2024.
Accessed: 2024-01-01.
13

---

Published in Transactions on Machine Learning Research (12/2024)
OpenAL. OpenAl LLM API. https://platform.oppeett openai com/ 2023a. Accessed: 2023-03-31.
OpenAL. New Models and Developer Products Announced at DevDay, 2023b. URL https://openai. com/
Accessed: 2024-10-01.
2024a. Accessed: 2024-
01-01.
OpenAL. Stripe uses OpenAI service to detect fraud. pttpss//ppeaaa.oomm///sssseeeee2
2024b. Accessed: 2024-01-01.
Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Associalion for Compulational Linguistics, 7:249-20 2609, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. Xiv preprinl 19x.::1110.01110 2019.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green AL. Communications of the ACM, 63 (12):54-63 63, 2020.
Ankur Sinha and Tanmay Khandait. Impact of news on the commodity market: Dataset and results, In Future of Information and Communécation Conference, volume 2, pp. 589-601, 2021.
Benjamin Spector and Chris Re. Accelerating LLM inference with staged speculative decoding, arXiv preprint Xrxv:2338.. 04628, 2023.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep convolutional network cascade for facial point detection. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3476-3484 3483, 2013.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami,, Himanshu Jain, Felix Yu, Michael Riley, and Sanjiv Kumar. Spectr: Fast speculative decoding via optimal transport. In International Conference on Machine Learning Efficient Systems for Foundation Models Workshop, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11123322.1180 11805, 2023.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid moaaffrmer-mamble models at scale. arXiv preprint arXiv:2408. 12570, 2024.
Textsynth. Textsynth L.M API. httpps:/texttyntth.o
2023. Accessed: 2023-03-31.
Paul Viola and Michael J Jones. Robust real-time face detection. International Journal of Computer Vision, 57:137 154, 2004.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A billion parameter autoregressive language model, 2021.
Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint XrXii:1707. 170,00697,,, 2017.
Carole-Jean WI, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Glo- ria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable Al: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 2022.
Fan Yang, Xuan Li, Qianmu Li, and Tao Li. Exploring the diversity in cluster ensemble generation: Random sampling and random projection. Expert Systems with Applications, 2014.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classifi- cation. In Advances in Neural Information Processing Systems, volume 28, 2015.
Lucia Zheng, Noel Guha, Brandon Anderson, Peter Henderson, and Daniel Ho. When does pretraining help? assessing self-supervised learning for law and the caschold dataset of 53,000+ legal holdings. In International Conference on Artificial Intelligence and Lawa pp. 159 168, 2021.
14


Reviews:
[{"review_id": "P2OtlUZiw9", "reviewer": "Reviewer_ymTi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed framework is simple but effective.\n2. There is a significant cost reduction while using reasonable scorer models to route the queries.\n\n\n**Weaknesses**\n1. In page 4, the input and response costs of GPT-4 are $30 and $60 per million tokens. The calculation is wrong which should be $2.12K instead. Also this price does not match the numbers in Table 1.\n2. While the chose datasets represent a subset of real-world applications, there is not enough justification for these classification datasets. Some smaller models, such as roberta, should be able to match the performance of many LLMs. In general, LLMs are better at generative tasks, such as open domain QAs, which lacks experimental analysis.\n3. It is not clear whether the proposed framework is task-specific or task-agnostic. Does the hyperparameters need to be tuned for each task?\n4. It seems that the proposed framework is highly dependent on the behavior of several LLM APIs. This may make the proposed framework less stable when applied to real-world applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To save cost and improve accuracy, this paper proposes FrugalGPT, an algorithmic framework to adaptively select the generative LLM APIs to sue for different queries. FrugalGPT has 3 components: a LLM router, an answer-scorer, and a stop judger. The experiments demonstrate that the proposed framework can match the performance of the best individual generative LLM or betterwith up to a 98% cost reduction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weaknesses. While the proposed framework demonstrates superior performance on several QA and classification tasks, there are several concerns: 1. LLMs are better at generative tasks other than classification tasks. The main usage of LLMs is focused on generative tasks; 2. Since there are consistent prices drops of state-of-the-art LLMs and the gap between open-source LLMs (e.g., Llama3.1) is smaller, is it necessary to use the proposed framework?", "broader_impact_concerns": "I did not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JmGPCMCNmS", "reviewer": "Reviewer_YQvh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a critical challenge in widespread adoption of LLMs: the high cost of inference and offers a very simple and practical solution: intelligently routing queries to different LLMs based on their cost and performance characteristics.\n- The paper is generally well-written and organized, making it easy to follow the authors' ideas and contributions.\n\nWeaknesses:\n- The framework relies on the black-box nature of LLM APIs, which may limit its applicability in scenarios where greater transparency and control over the models are required.\n- The paper considers models that are slightly out of date. Models such as GPT-4o, GPT-4o mini and Gemma-2B aren’t included.\n- The paper assumes the availability of a reliable answer scorer that can accurately assess the quality of LLM-generated answers. However, the quality of such a scorer can significantly impact the overall performance of FrugalGPT. The paper briefly mentions an ablation study on the generation scorer in Section 5, but a more detailed discussion of the scorer's limitations and potential impact on FrugalGPT would be valuable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces FrugalGPT, a framework designed to optimize the utilization of Large Language Models (LLMs) by dynamically selecting the most suitable LLM for specific tasks and queries, thereby minimizing costs.\nFrugal GPT learns a generation judger to assess the quality of LLM outputs and then sequentially invoking LLMs until the judger's score meets a predefined threshold.\nThey empirically show that FrugalGPT can save up to 98% of the inference cost of the best individual LLM API while matching its performance on the downstream task. Additionally, FrugalGPT can improve performance by up to 4% at the same cost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The prices of LLM APIs for Inference has gone down significantly compared to the prices reported in Table 1. Including the cost savings considering the new prices will strengthen this paper. \n- Adding more information on how the LLM router works in the Frugal GPT pipeline will strengthen the paper.\n- Further elaboration on implementation details, theoretical analysis, and potential overfitting concerns would enhance the paper’s technical depth and contribution.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9TcFG1bTyh", "reviewer": "Reviewer_knVh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is that the general approach of calling smaller models before bigger models is solid, and the results seem solid as well besides the dataset type issue below.\n\nThe main critique of the paper is that it is quite elaborate, and there is no comparison to a far simpler available baseline: send the answer to GPT-J and accept sufficiently high probability answers (this is a simplified version of their L = 2 setup, but requiring training only a single scalar for the classification case). There is a brief allusion to this strategy in the first paragraph of Appendix E, implying they intentionally avoided such strategies because some LLM APIs do not provide log probs. However, returning log probs does not add any cost to an API, and the feature is missing from proprietary model services for security reasons alone (anti-model theft and jailbreaking). Smaller open models such as GPT-J do provide log probs: https://textsynth.com/documentation.html#logprob\n\nAnother concern is that there is no discussion of differences between the datasets that might be contributing the quality variation. Citing a clear outlier in performance in the abstract is bad: any sufficiently easy classification dataset would get such a result (which the simpler baseline from the previous paragraph), the other classification datasets get only 4x, and the QA datasets can only 2x. This distinction is hard to see from the main text of the paper: it occurs only in prose in an appendix. The type of dataset (n-way classification vs. multiple choice QA vs. free-text QA) should be given in the main body, ideally in Table 2.\n\nSmaller notes are listed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method which jointly trains a routing model and a scorer to (1) choose which of several LLMs to send a query to and (2) decide whether the result is of sufficient quality. The method results in significant time savings on 2 to 4-way classification datasets (>50x on one likely easy dataset and 4x on two others), and ~2x speedups up two question answer datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The key two changes are as mentioned above:\n\n1. Compare against the baseline of using a single scalar threshold on GPT-J logits to decide whether to accept the result of one GPT-J call or call up to GPT-4. This is a simplified version of L = 2 that avoids a training step. For classification and multiple choice tasks this is immediate; for the free-text QA task it may be useful to divide by the number of tokens.\n2. Highlight the types of datasets in Table 2, with a column that says \"<n>-way classification\", \"multiple-choice QA\", or \"free-text QA\" (or equivalent, and discuss the differences in results between the different types.\n\nHere are some smaller requested changes:\n\n1. CoHere shouldn’t have a capital H.\n2. The “Ope (b)” citation looks like a typo. There are a few others like this.\n3. Add a limitation about long-form queries, as I would expect this method to produce smaller speedups there.\n4. The 5% quality improvement on some datasets seems likely to be a real ensembling effect, but it should be presented as such (that it likely has the same cause as ensembling methods).\n5. Figure 5 in the appendix says the details can be found in the appendix.", "broader_impact_concerns": "No broader concerns: I agree with the authors' description of ethical aspects based on efficiency gains.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the authors' revision, two reviewers lean toward acceptance, acknowledging the positive contributions of this paper. One reviewer (Reviewer ymTi), however, leans toward rejection, with remaining concerns including: (1) the sustainability of the proposed framework given the rise of proprietary LLM APIs with substantially reduced costs and the availability of open-source alternatives; (2) the need for a more comprehensive evaluation on generative tasks; and (3) the complexity of the proposed framework.

I agree with Reviewer ymTi that these concerns are valid. However, I believe that the contributions of this work offer value to the community despite these limitations. In the revision, the authors included a comparison with a simpler baseline, which helps validate the necessity of the proposed approach’s complexity. The authors also demonstrated the effectiveness of their approach with more recent LLM models and on two generative tasks.

​​Given these points, I recommend acceptance of this paper. I have the following suggestions for the authors as they prepare the final version:

* The newly added comparisons with a simpler baseline (C.6) and more recent LLM models (C.7) are essential to demonstrate the necessity of the approach’s complexity and its current effectiveness. Including these results in the main paper would strengthen the validation of the method.

* If possible, adding further results on generative tasks could enhance the paper, as suggested by Reviewer ymTi.
Decision: Accept (poster)