Pricing and Competition for Generative AI

Authors: rafid  mahmood
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:10:20

Abstract: Compared to classical machine learning (ML) models, generative models offer a new usage paradigm where (i) a single model can be used for many different tasks out-of-the-box; (ii) users interact with this model over a series of natural language prompts; and (iii) the model is ideally evaluated on binary user satisfaction with respect to model outputs. Given these characteristics, we explore the problem of how developers of new generative AI software can release and price their technology. We first develop a comparison of two different models for a specific task with respect to user cost-effectiveness. We then model the pricing problem of generative AI software as a game between two different companies who sequentially release their models before users choose their preferred model for each task. Here, the price optimization problem becomes piecewise continuous where the companies must choose a subset of the tasks on which to be cost-effective and forgo revenue for the remaining tasks. In particular, we reveal the value of market information by showing that a company who deploys later after knowing their competitor’s price can always secure cost-effectiveness on at least one task, whereas the company who is the first-to-market must price their model in a way that incentivizes higher prices from the latecomer in order to gain revenue. Most importantly, we find that if the different tasks are sufficiently similar, the first-to-market model may become cost-ineffective on all tasks regardless of how this technology is priced.

Main paper: 
 introduction: 
 The recent explosion of generative artificial intelligence (AI) has introduced new machine learning (ML) frameworks for applications from chatbots to robotics [Wu et al., 2023 , Nasiriany et al., 2024] . Whereas in classical ML, a user interacted with a single model designed for a specific predictive task (e.g., classification) via input data and output predictions, a single generative AI model can solve a variety of tasks for a user out-of-the-box [Brown et al., 2020] . Moreover, users interact with the generative model over a universal interface of natural language prompting [Arora et al., 2022] .The prompt-based paradigm has fostered two recent human-AI interaction trends. First, prompting facilitates such a wide distribution of tasks (i.e., user inputs and model outputs) that conventional metrics for evaluating models have become insufficient, leaving the most effective evaluation metric to be a binary score of whether the user is satisfied with the model output [Li et al., 2024 , Chiang et al., 2024] . For example, Ziegler et al. [2024] empirically analyzed the GitHub Copilot software to reveal that the frequency of generated code approved by a user 'is a better predictor of perceived [user] productivity than alternative measures.' Second, if a user does not receive a satisfactory output, they can try again in another prompting round by inputting to the model additional information [Castro et al., 2023] . For instance, the Anthropic HH and the Chatbot Arena datasets report on average 2.3 and 1.3 prompting rounds per conversation, respectively [Bai et al., 2022 , Chiang et al., 2024] .In this work, we study the impact of these interaction characteristics on the pricing of generative AI technology. While classical ML products can be priced by analyzing the user demand for a model that can achieve a given performance metric on a specific task [Gurkan and de Véricourt, 2022, Mahmood Figure 1 : Overview of the competitive pricing problem for generative AI models. et al., 2022] , a generative AI model is priced per user prompt1 . This set price determines the user cost for multiple different tasks and variable number of prompting rounds, e.g., the cost of using GPT-4 for math reasoning or code generation depends only on the per-token price, and the length and number of prompts. Thus, developers of a generative AI product must factor the demand for all potential use-case tasks of the technology when setting a price. This pricing problem becomes further challenging when considering the rapidly growing marketplace of competing generative AI models, since companies must also ensure that their products do not become unattractive to users as soon as a competitor develops a newer and better model.We first characterize when, for a given task, a user will prefer one generative AI model versus another. We argue that users minimize their total cost, measured by the cost-per-prompt times the number of prompting rounds needed for the model to produce a satisfactory output; this leads to a comparison of price-performance competitiveness between AI models. We then study a game with two firms developing competing models used for a set of tasks. Both firms know each other's model's performance on the tasks. The first firm deploys their product and sets a price, followed by the second firm with their product and price. Finally, a user decides which models to use for each task. Both firms seek to maximize revenue, but the first firm acts without knowledge of their competitor's price. Figure 1 summarizes the problem setting and insights. Our key observations include:1. The pricing problem reduces to a piecewise optimization problem, where firms price their model to be competitive on a subset of the tasks while forgoing revenue from the others. This subset can be determined by ranking the tasks on the competitive ratio between the two models for each task and selecting the most competitive tasks.2. A firm who deploys late always obtains revenue from at least one task by leveraging the available market information. In contrast, the first-to-market must strategically set their prices to encourage the latecomer to set higher prices and focus on fewer tasks.Under certain conditions on model performance and user demand, the first-to-market may acquire zero revenue regardless of their price. In these settings, the latecomer naturally maximizes their revenue by being competitive for all tasks. Thus, developers that are first should have a minimum model performance before deploying their product. 
related literature: 
 Evaluating ML models. ML models are typically evaluated on generalization error for a task via out-of-sample test dataset benchmarks and competitions [Deng et al., 2009] . Generative AI and large language models (LLMs) are evaluated on a suite of benchmark tasks such as for coding [Chen et al., 2021] , math [Cobbe et al., 2021] , and problem solving [Hendrycks et al., 2021] . However, standardized benchmarks become uninformative over time as new models are trained to overfit to these metrics [Roelofs et al., 2019 , Koch et al., 2021] . Recently, Chiang et al. [2024] introduced the Chatbot Arena for comparing LLMs head-to-head on human preference. Here, a user poses a real-world prompt which is input to two models. The user reviews both model outputs and can even continue multiple conversation rounds, before ranking which model generated a satisfactory answer first. Although difficult to measure, user satisfaction rate is increasingly viewed as the most informative metric as seen from generated code approvals on GitHub Copilot [Ziegler et al., 2024] , or perceived aesthetic quality of text-to-image generation such as MidJourney and Playground [Li et al., 2024] . Our work combines user satisfaction with a user cost to construct a price-performance ratio for comparing different models.Human-generative AI interaction. Prompt-based interaction has increased the diversity of tasks where these models can be applied [Eloundou et al., 2023] . Castro et al. [2023] analyze how and when human users will use a generative AI model for a task versus performing it manually, as well as the characteristics of interacting over multiple prompt rounds. The quality of prompts is crucial to generating higher-quality answers [Liu et al., 2023, Binz and Schulz, 2023] . This has motivated studies on prompt techniques, such as chain-of-thought [Wei et al., 2022] and self-consistency [Wang et al., 2022] . In our work, we treat prompt quality as a random variable and to simplify the structural analysis, assume that users will interact with a generative AI model for as many prompting rounds as needed to get a satisfactory answer.Pricing and competition. Duopolies of competitive products use game theoretic models such as the Bertrand (i.e., simultaneous pricing) and Stackelberg (i.e., sequential pricing) models of interaction [Gibbons, 1992] . Both deterministic and probabilistic demand can be used to study oligopolistic pricing of a single or multiple products Chintagunta and Rao [1996] , Gallego and Hu [2014] . Specific structured demand frameworks allow for identifying market equilibria and failure settings where revenue is unobtainable [Federgruen and Hu, 2015, 2019] . Our work is most closely related to the Stackelberg literature by modeling a sequential game and exploring the conditions under exponential demand that make certain generative AI models unattractive [Hamilton and Slutsky, 1990] .Pricing AI products. AI technology can be priced at various levels, ranging from training data to model queries [Liu et al., 2021 , Chen et al., 2019 , Cong et al., 2022] . A core aspect of the pricing problem involves valuating the ML model based on performance [Xu et al., 2024] . ML products are further susceptible to an AI flywheel effect where the release and price of an AI product will affect the subsequent collection of new training data from users, leading to a dynamic pricing problem [Gurkan and de Véricourt, 2022, Chen and Xue, 2023] . More generally, novel technology products such as a new generative AI model with emergent use-cases may feature social-learning and dynamically growing market sizes [Feldman et al., 2019 , Zhang et al., 2022] . The closest to our work is Gurkan and de Véricourt [2022] who explore pricing and contracting the development of a classical ML model under the AI flywheel. In contrast, our work explores competition between ML model developers when faced with a diverse set of potential downstream tasks for which the model can be used. 
main model: 
 We first define the characteristics of the pricing problem. We then propose a model of user choice between two competing models from a price-performance perspective.Tasks. We define a task as a set of independent problem instances where for each instance, a user queries a machine learning model with an input prompt and receives an output generated the model. For example, a programming task may have instances where a user inputs a commented function definition and the model must complete the code to perform the function Chen et al. [2021] , Ziegler et al. [2024] . Task instances are evaluated by a user via a binary correctness score. For tasks where correctness is unambiguous (e.g., whether the program runs), the score is equivalent to accuracy, whereas for open-ended tasks (e.g., whether the output meets the stylistic preferences of the user), we treat correctness simply as whether the user is satisfied with the output2 Generative AI model. Given a set of T different tasks, a generative model is an ML model that can be used to solve instances of any of the different tasks via prompts. We define this model as a tuple (p, V 1 , V 2 , . . . , V T ) where p denotes the price for using the model, as measured in dollars-per-prompt (see Appendix B for the extension to pricing per-token), and for each t ∈ [T ], V t ∈ (0, 1) denotes the average score of the model over instances of each task. We assume that V t → 1 implies that the model can always generate a correct output for any task instance and V t → 0 implies that the model will always generate an incorrect output. Therefore, for any random instance of task t, V t can also be interpreted as a Bernoulli probability of the generative model producing a satisfactory output in a single attempt.Users will not use the generative model if it's price is too high with respect to the user's valuation of the specific task. For any given task t, let D t (p) ∈ R + be the demand function, i.e., the number of users who will use a generative AI model for task t as a function of the price p. Following standard assumptions, we assume that D t (p) is differentiable and non-increasing in the model price, as well as being known to the developer of the AI model [Gallego and Van Ryzin, 1994] .Multi-round use. Most ML benchmarks typically evaluate models on whether the models can generate the correct output under a single prompt round [Chen et al., 2021 , Cobbe et al., 2021 , Hendrycks et al., 2021] . However, in-the-wild users of generative models typically have interactive multi-round conversations where if the model generates an unsatisfactory solution after the first prompt, the user can provide feedback via their preferences or corrections in a second prompt round [Castro et al., 2023 , Liao et al., 2024] . For example in code completion, if the model fails to account for an edge-case input to the function, the user can identify the edge-case and ask the model to account for it. To characterize this multi-round use, we extend the single prompt to a sequence of Bernoulli trials that continue until the user is satisfied with the model output. For simplicity of modeling, we make the following assumptions on user behavior. Assumption 1. The total number of prompting rounds n t (V t ) that a user will engage with the model: (i) has a finite mean; and (ii) is independent of the model price p conditioned on the user knowing V t .Assumption 1 implies that the price of a model only determines demand via whether the model is used at all, rather than how many times (i.e., prompting rounds) the model is used. Under this assumption, there are many choices for modeling the distribution of n t (V t ). We give three examples:• Geometric: Because non-expert users tend to design uninformative prompts [Zamfirescu-Pereira et al., 2023] , we may suppose the probability of success on any round does not depend on user input, and each round is an i.i.d. Bernoulli trial with probability V t . Then, the total number of rounds is n t ∼ Geom(V t ), following a Geometric distribution.• Truncated Geometric: Users may quit the model after a maximum T t rounds if it fails to generate a satisfactory response [Castro et al., 2023] . For instance, conversations in the Chatbot Arena dataset terminate after an average 1.3 rounds [Chiang et al., 2024] . Here,Pr{n t = n} := (1 -V t ) n-1 V t for 1 ≤ n < T t and Pr{n t = T t } := (1 -V t ) Tt-1 .• Prompt-dependent: Suppose the success probability is prompt-dependent V t (x i ) where x i ∼ Pr{x} is the prompt on the i-th round. If users prompt until the model generates a satisfactory answer, we havePr{n t = n} := E x1,••• ,xn [V t (x n ) n-1 i=1 (1 -V t (x i ))].Ultimately, the choice of characterizing n t depends on the information available to the generative AI model provider. With limited information, the Geometric assumption may be most practical, but given knowledge of user prompts, we may consider more sophisticated models. Our results all hold independent of the distribution as long as Assumption 1 is satisfied. For ease of notation and interpretability, we assume n t ∼ Geom(V t ) in the remainder of this work.Given the above task and user behavior framework, we analyze the problem of a user who must choose between two competing generative models to solve instances of their tasks. Since a user may prefer different models for different tasks, we consider a single task and omit subscript t.Consider two generative models: model A (q, W ) and model B (p, V ). Under Assumption 1, given a sufficient number of prompt rounds, both models can eventually solve every task instance. Thus, a rational user will seek to minimize the expected cost of completing a task instance, measured by the average price-per-prompt times the expected number of rounds required to complete the task instance.In this comparison, model B will incur a lower cost for the user ifEQUATIONOtherwise, model A incurs lower user costs. We assume that model B is preferred in ties. Note that the implication follows from our Geometric assumption of n(V ).Condition (1) states that for any task, a specific generative AI model is preferred if the priceperformance ratio for this task (i.e., the cost of using this model over the model's performance) is lower than any other competing generative model. For example, if model B is twice as likely to generate a user-satisfactory output as model A for a given input prompt, i.e., V = 2W , then the user will prefer model B as long the cost of prompting this model is not twice as high, i.e., p ≤ 2q.Otherwise, the user will incur lower costs and still obtain their desired outputs by simply prompting the weaker model for twice as many rounds.We note that (1) can also compare the use of a generative AI model versus manually performing the task [Castro et al., 2023] . For instance, if we treat model B as a human and a prompt round as a timed attempt at completing a task (e.g., coding the function within one hour), then V is the probability of a human being able to perform the task in the single attempt and p is the time-value of this labor.Finally, this framework can also compare free-to-use generative AI models such as LLaMA [Touvron et al., 2023] . Although there may not be a given price-per-token for using these models, there is a fixed cost to set up the infrastructure and environment. Given an expected total number of task instances, this fixed cost can be approximated to an equivalent p. 
pricing generative ai models: 
 We now develop a general framework under which a provider of a generative AI model can price their product. We first define the pricing problem as a game between two firms developing competing generative AI models. We then create tractable reformulations for these problems for both firms, representing pricing with and without considering competition.Pricing problem. Consider a set of T tasks. Let (q, W 1 , W 2 , . . . , W T ) and (p, V 1 , V 2 , . . . , V T ) be the generative models released by two competing firms, A and B, respectively. Firm A deploys their generative AI model (i.e., model A) first and sets the price q for this product. Then, firm B deploys their competing model (i.e., model B) and sets their price p. After both firms deploy their models, a user with demand functions D t (•) for each t will decide which models to use as determined by (1). We evaluate the total revenue obtained by each firm, given by R A (q|p) and R B (p|q) respectively:EQUATIONNote that in practice, the user demand may depend on p and q simultaneously; for simplicity, we assume the demand for a specific model to be determined only after the user preference condition (1) is resolved. The results in this section easily generalize to more complex demand functions. The revenue functions are composed of the demand for each task times the price set by the firm [Van Ryzin and Talluri, 2005] , summed over all tasks for which the firm's model is competitive according to the price-performance ratios. Each firm's objective is to maximize their revenue. However, because firm A is the first-to-market and they do not know the action that firm B will take; instead, firm A must optimize for the worst-case scenario as determined by firm B. On the other hand, firm B first observes the price set by firm A. Thus, the two firms set prices as follows:EQUATIONWe assume both firms know V t and W t for all t ∈ [T ]. This is motivated by the availability of research papers and reported benchmark scores, and the predictability of model performance via power laws [Kaplan et al., 2020] . In practice, a firm may not know their competitor's performance, but they can forecast the state-of-the-art score in the short term.We first consider firm B's problem, who set their price after firm A has already released a competing model with price q * . Firm B's problem depends on the competitive ratioκ t := E[n(W t )]/E[n(V t )]for task t ∈ [T ], i.e., the relative ratio of number of prompting rounds between the two firms, which under a Geometric distribution assumption, simplifies to κ t := V t /W t . To maximize their revenue, firm B can rank the tasks in order of κ t , select a subset of tasks in this order, and solve the optimization problem for each subset. This results in an overall piecewise optimization problem. The vertical lines correspond to κ 1 q, κ 2 q, and κ 3 q, where κ 1 > κ 2 > κ 3 . For p < κ 3 q, revenue is obtained from all three tasks, for p ∈ (κ 3 q, κ 2 q], revenue is obtained from only the first two tasks, and for p ∈ (κ 2 q, κ 1 q], revenue is only obtained from the first task. No revenue can be obtained if p > κ 1 q.Theorem 1. Consider the following ordering σ :EQUATIONThen, firm B's pricing problem is equivalent to the piecewise optimization problem:EQUATIONTheorem 1 shows that a generative AI model should be priced by prioritizing a subset of the tasks and making the model price-performance competitive for those tasks only. Consequently, the firm ignores the remaining tasks for which the model has low competitive ratios κ t , since they would need extremely low prices in order to satisfy (1) for these tasks. This strategy is due to the observation that for any task σ(t), if a model satisfies (1), then the model will also satisfy the condition for all σ(t ′ ) for t ′ ≤ t. Furthermore, Theorem holds without loss of generality with respect to the strict inequalities on (4), since tasks with the same competitive ratios can be grouped together by summing the constituent demand functions. Finally, Theorem 1 holds for arbitrary demand functions. In practice, the demand functions are known and typically have a parametric structure. Figure 2 gives an example using three tasks with demand that decays exponentially with price.Theorem 1 reveals two key implications on how a firm can price a generative mode when given competitor information. First, pricing reduces to solving T optimization problems, where each of these problems are of a single variable with a differentiable objective and boundary constraints. Thus, the inner problems can be solved via gradient descent. Second, as long as firm B sets a price p ≤ κ σ(1) , they will obtain some non-zero revenue, i.e., problem (5) always has a feasible solution. This advantage is due to the fact that firm B sets their price given a fixed q.We next consider firm A's problem of setting a problem while assuming that firm B will act optimally next. This pricing problem is a bi-level optimization problem, but it can be solved by ranking the tasks according to the competitive ratios for firm A and prioritizing a subset of these tasks.Theorem 2. Firm A's pricing problem is equivalent to the piecewise bi-level optimization problem:EQUATIONTheorem 2 relies on the observation that the reverse order of σ(•) in ( 4) cn rank the most to least competitive tasks for firm A. For any t, if q < pκ -1 σ(t) , then model A is price-performance competitive for all tasks σ(t), • • • , σ(T ) and firm A will acquire revenue from all these tasks. Firm A may not always be able to obtain revenue. Problem ( 6) is infeasible if for every t ∈ [T -1], there is no q ≥ 0 that satisfies the bi-level constraint. This infeasibility implies for any q ≥ 0, firm B always maximizes their revenue by setting a low price p ≤ κ σ(T ) q. Thus, the key motivation of firm A is that the firm benefits only when their competitor is incentivized to set high prices. 
structural analysis under exponential demand: 
 Demand is typically modeled via structured parametric functions [Van Ryzin and Talluri, 2005] . In this section, we specialize the pricing problem to the standard choice of exponentially decaying parametric demand to extend the previous general results. See Figure 2 (Left) for an example. Assumption 2. For each task t, the demand function decays exponentially in price D t (p) := a t exp(-bp), where a t > 0 is the zero-price base demand and b > 0 is the price-sensitivity of users. Furthermore, all tasks have the same price-sensitivty.Under the exponential demand model, the demand for each task t is equal to a t when p = 0, and decays at a rate -b. We assume that the decay rate is the same for each task; this is motivated by the practical consideration that the different tasks should have relatively similar 'user value' to have the same price. If one task is uniquely price-sensitive to users, the firm may instead develop a finetuned model for that task or propose incentives such as task-specific discounts to better optimize revenue.Below, we revisit both firm B's and firm A's problems under this demand model to derive globally optimal solutions and structural insights on the market dynamics.Under Assumption 2, firm B's problem (5) now simplifies to the maximum of up to T possible values that are the globally optimal solution to each of the individual inner optimization problems.Theorem 3. Suppose Assumption 2 holds. For any t, let āσ(t) := t s=1 a σ(t) . Without loss of generality, let t * ∈ [T ] be the task index that satisfies κ σ(t * ) q ≥ 1/b > κ σ(t * +1) q. Then, firm B's pricing problem is equivalent to the following maximum value:EQUATIONTheorem 3 states that problem (5) can be solved by by solving each of the inner problems starting from the lowest price range up until we arrive at the zero-gradient solution of the revenue function, i.e., 1/b. Furthermore, for each of the price ranges before this point, the optimal solution is the upper price boundary. Note that if 1/b > κ σ(1) q, i.e., there does not exist any t * satisfying the condition, then we take the maximum of all T problems. Figure 2 (Right) visualizes the revenue function R B (p|q) for T = 3 tasks with exponential demand. Finally, this structure also reveals that the optimal price is bounded from both above and below via these optima.Corollary 1. The optimal price for firm B is bounded 1/b ≥ p * ≥ κ σ(1) q.Figure 3 : The relationship between κ 2 /κ 1 and a 1 /(a 1 + a 2 ) for firm A. In the blue region, firm B will always set a price that is competitive on both tasks and firm A will acquire zero revenue. In the orange region, the maximum price that firm A can set is upper bounded (see problem ( 11)). In the green region, the maximum price that firm A can set has a higher upper bound (see problem ( 10)).We now revisit firm A's pricing problem, which is a bi-level problem with multiple inner optimization constraints. To obtain structural insights on the challenges of pricing under competition, we explore a special case with T = 2 tasks. Without loss of generality, we assume κ 1 > κ 2 , i.e., σ(t) = t.When there are only two tasks, firm B will either set their price to be competitive for the first task only (i.e., the task with the higher competitive ratio) or for both tasks. In the latter case, model A would be unattractive for both tasks and firm A would obtain zero revenue. Therefore, firm A should set their price in such a way that they maximize revenue on the second task, while ensuring that firm B is motivated to be competitive only for the first task. Thus, problem (6) simplifies to max q qD 2 (q) s. t. maxEQUATIONIf firm A sets a price q that is infeasible for this problem, they will get zero revenue. Furthermore, this problem reduces to two single-level optimization problems with only bounding constraints. Theorem 4. Suppose that T = 2, that Assumption 2 holds, and without loss of generality, assume σ(t) = t. For z ∈ R, let W(z) be the Lambert W function defined only over z > -e -1 . Ifκ 2 κ 1 ≤ -W - a 1 e -1 a 1 + a 2 (9)then, the firm A's pricing problem is equivalent toEQUATION)Otherwise, firm A's problem is equivalent toEQUATIONRecall from Theorem 3, the second-level problems in (8) can be solved analytically by checking the boundary points and zero-gradient solution. Theorem 4 uses this property to map problem (8) to two sub-problems based on whether condition (9) holds.Determining which problem to solve to obtain the optimal price depends on a relationship between two constants κ 2 /κ 1 and a 1 /(a 1 + a 2 ). Here, κ 2 /κ 1 is the relative competitive ratio with respect to the two tasks for firm B, where κ 1 > κ 2 > 0. Thus, κ 2 /κ 1 → 1 suggests that the relative performance differences between model A and model B are similar for both tasks, whereas κ 2 /κ 1 → 0 suggests that relative to model A, model B's performance is much worse on the second task than for the first task. The second parameter a 1 /(a 1 + a 2 ) is the fraction of the total demand that is occupied by the first task. If this is close to 1, then the first task has significantly higher demand than the second, but if it is close to 0, then the first task has significantly lower demand than the second.We now discuss the structural insights obtained from Theorem 4 (see Figure 3 for a visualization). First, note that when condition ( 9) is satisfied, firm A is able to set higher prices than it could if the condition were not satisfied. In the latter case, the optimal price that firm A can set must be upper bounded by the constraint in problem ( 11), which in this scenario, is less than the upper bound in problem (10). Intuitively, this condition partitions firm A's pricing problem into two regimes: a high-price and low-price regime. Furthermore, the high-price regime is only attainable if the relative performance difference between the two tasks is greater than the Lambert W function of the fraction of total demand occupied by the first task.Second, if firm B's relative performance difference between the two tasks is larger than the fraction of demand occupied by the first task, then firm A's pricing problem is infeasible. That is, no matter what price that firm A sets, firm B is always incentivized to set a price that ensures users will prefer model B and consequently, leave no revenue for firm A. Proposition 1. If κ 2 /κ 1 > a 1 /(a 1 + a 2 ), then for any price that firm A sets, firm B will always set a price that allows them to be price-performance competitive for both tasks.Intuitively, if the relative performance difference is high, then model B's performance relative to model A is approximately the same for both tasks. Consequently, the range (κ 2 q, κ 1 q] is small, meaning that a small perturbation from this price would not significantly decrease firm B's revenue from the first task, while permitting the firm to obtain revenue from the second task as well. Note that model B does not necessarily have to be 'better' than model A, only that the performance on the tasks must be similar. Finally, this ratio κ 2 /κ 1 only needs to be larger than the fraction of total demand that is occupied by the first task. Consequently, even if the ratio is small, firm B can still be incentivized to set a competitive price for both tasks if the potential revenue that can be obtained from the second task is high.Theorem 4 and Proposition 1 sets a guideline for when a firm should deploy their generative AI model. Consider a company that has developed a model for a new application area with no competitors. This company knows that upon releasing their model to the public, competitors will release their own models. If the first-to-market company believes that the use-cases for this model are sufficiently similar to each other with respect to model performance, but are differentiated with respect to user demand, then they must ensure that the model performs exceedingly well on at least one specific use-case that their competitors cannot match. That is, the company must differentiate their product [Schmalensee, 1982] , or competitors can outprice the initial company with similar products.Although firm A must price accounting for firm B's actions, if model A is sufficiently differentiated from model B, then firm A can acquire their maximum possible revenue. Recall that the zero-gradient optimal price for problem ( 8) is q * = 1/b. Furthermore recall that if condition ( 9) is met, then firm A can set higher prices for their model. We show below that if condition ( 9) is satisfied and if κ 2 is sufficiently low with respect to the demand ratio, then the optimal price is feasible. Proposition 2. If κ 2 ≤ -W(a 1 e -1 /(a 1 + a 2 )) min(κ 1 , 1), then firm A maximizes their revenue by setting q * = 1/b. 
conclusion: 
 We study how a company developing a generative AI model can set the price for this technology.Generative AI models are a fundamentally different technology product compared to classical AI models due to two factors. First, a single generative AI model is performant on multiple distinct use-cases that each invite individual respective user demands. Second, generative AI models offer interactivity via prompting, thereby encouraging 'geometric' user interaction where users can repeatedly prompt the model until it generates a satisfactory answer, compared to classical non-interactive AI models that invite one-shot 'Bernoulli' interaction. These features combined with a singular unit price per-prompt for model use warrant new revenue maximization frameworks for this technology.We find that for generative AI models, the pricing problem reduces to ranking the different tasks in order of the relative performance of the model versus a competing alternative. In isolation, a company can then always obtain non-zero revenue by setting a price to be competitive for at least one downstream task. However, when considering the competition from alternative models that may be released after the price is set, the company faces strict upper bounds on the prices they can set based on the performance of the company's model relative to the latecomers. In particular, if the relative difference in performances on the tasks is sufficiently small, then a competitor will always be able to outprice the company on all tasks. This result reveals that an outsized performance improvement on at least one of the downstream applications of a generative AI model is essential to maximizing revenue.Limitations. We explore a theoretical market problem where firms know user demand and the performance of competing AI models. In practice, these would be estimated with some noise. Furthermore, we study a static marketplace where each firm sets their price once. In contrast, AI technologies feature a flywheel where a low price and an early release of a product can allow for a firm to collect more training data and improve their model downstream [Gurkan and de Véricourt, 2022] . In particular, the opportunity to acquire high-quality data can significantly improve model performance, thereby motivating the first mover position. Finally, we do not include the cost of developing the generative AI models themselves, but assumes that the fims have already decided to build these models. The market dynamics can change when considering how large of a model to build, how much resources (e.g., compute hours) to spend, or even whether to build a generative AI model. We see these scenarios as important future extensions.Societal impact. Pricing of generative AI can significantly affect the democratization of generative AI technology. This paper explores the conditions that incentivize firms to develop or deploy this technology, e.g., when firms can obtain revenue. Setting appropriate prices for these models can allow for this technology to be more easily accessible to a wider set of users. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [NA] Justification: There is no compute required. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have reviewed the code of ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Broader societal impacts are discussed in the conclusion. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to 
 
 Appendix: Proof of Theorem 1. Consider the ordering σ defined by (4) and note that for any t, if we constrain p to satisfyκ σ(t) ≥ p q > κ σ(t+1) ,then all of the tasks σ(1), σ(2), . . . , σ(t) become competitive for firm B's generative model compared to firm A's model, i.e., by satisfying (1), whereas tasks σ(t + 1), . . . , σ(T ) are competitive for firm A's model instead. As a result, for prices constrained in this range, the resulting problem becomes max p p t s=1 D σ(s) (p). Therefore, to solve the overall pricing problem, we only need to solve this resulting constrained problem for each value of t.Proof of Theorem 2. From Theorem 1, for any value of q, there exists a t ∈ [T ] such that firm B will prioritize revenue from tasks σ(1), σ(2), • • • , σ(t). This value of t will satisfymax p p t s=1 D σ(s) (p) κ σ(t) q ≥ p > κ σ(t+1) q > max p ′    p ′ t ′ s=1 D σ(s) (p ′ ) κ σ(t ′ ) q ≥ p ′ > κ σ(t ′ +1) q   for all t ′ ̸ = t. Furthermore, given this t, firm A will only obtain revenue from tasks σ(t + 1), σ(t + 2), • • • , σ(T ), thereby revealing the objective function. Finally, note that if t = T , then firm A will not acquire any revenue since firm B will be price-performance competitive on all tasks. Therefore, firm A's optimization problem is to simultaneously solve for q and for t ∈[T -1].Proof of Theorem 3. First, note that under an exponential demand function the revenue function for each piece t of problem ( 5) is pā σ(t) exp(-bp), and therefore has a zero-gradient point at p * = 1/b. Thus, when solving problem (5), we can solve the inner problem by breaking into 3 cases based on which t to consider.For t * satisfying the condition. Here, the zero-gradient price p * is a feasible solution to the inner pricing problem, meaning that this must be the optimal price. Substituting this into the revenue function yields āσ(t * ) exp(-1)/b.For t > t * . Note that for any t > t * , the price is constrained to be less than κ σ(t * +1) < 1/b. In this regime, the revenue function pā σ(t) exp(-bp) is monotonically increasing, meaning that the optimal price will be the maximum possible value, i.e., p = qκ σ(t) . Substituting this into the revenue function yields κ σ(t) qā σ(t) exp -bκ σ(t) q .For t < t * . Here, the set of feasible prices is strictly greater than κ σ(t * ) > 1/b. In this regime, the revenue function is monotonically decreasing, meaning that the optimal price will be the minimum value, i.e., p → qκ σ(t+1) , approaching from above. However, for any such setting, decreasing p to equal the infinum would make firm B price-performance competitive for the t + 1-th task as well and thereby yield a higher return. Therefore, the optimal solution for any t < t * can always be upper bounded by the optimal solution for t + 1, leading up to t * .Proof of Corollary 1. The upper bound follows from the proof of Theorem 3, as we show that the optimal revenue cannot be achieved without satisfying task σ(t * ). The lower bound follows from observing the optimal solution for t = T .Proof of Theorem 4. Let p 1 := arg max{pD 1 (p) | κ 1 q ≥ p > κ 2 q}. Furthermore, let p 2 := arg max{p(D 1 (p) + D 2 (p)) | κ 2 q ≥ p > 0}. Although p 1 and p 2 depend on q, we omit this dependency to simplify the notation. We prove our theorem by exploring three regimes for q: (0, 1/(bκ 1 )], (1/(bκ 1 ), 1/(bκ 2 )], and (1/(bκ 2 ), ∞). We then show that whether condition (9), the problems for each regime simplify further into problems (10) and (11).For q ∈ (1/(bκ 2 ), ∞). For any q > 1/(bκ 2 ), we have κ 2 q ≥ 1/b > 0. From Theorem 3, p 2 = 1/b is the optimal price. Furthermore, since κ 1 q ≥ p 1 > κ 2 q, we have p 1 > 1/b. From Corollary 1, p 1 is priced too high and cannot achieve a higher revenue than p 2 . Therefore in this regime, firm B will set a price low enough that their model is competitive for both tasks, and consequently firm A will achieve zero revenue.For q ∈ (1/(bκ 1 ), 1/(bκ 2 )]. From Theorem 3, p 1 = 1/b and p 2 = κ 2 q are the optimal prices that firm B can set for their two sub-problems. For firm A to achieve any revenue, the constraint in problem (8) becomes 1 b a 1 exp(-1) > κ 2 q(a 1 + a 2 ) exp(-bκ 2 q) ⇒ -bκ 2 q exp(-bκ 2 q) > -a 1 exp(-1)a 1 + a 2 ⇒ -bκ 2 q > W - a 1 exp(-1) a 1 + a 2 ⇒ q < - 1 bκ 2 W - a 1 exp(-1) a 1 + a 2 ≤ 1 bκ 2Above, the first line follows from rearranging the terms and the second line follows from applying the definition of the Lambert W function, i.e., W(z) = y ⇔ y exp(y) = z. The third line follows again from rearranging the terms. Finally, note that the Lambert W function is bounded in [-1, 0], meaning that this constraint dominates the original upper bound. Therefore, for q in this regime, we can solve the problemmax qa 2 exp(-bq) - 1 bκ 2 W - a 1 exp(-1) a 1 + a 2 ≥ q ≥ 1 bκ 1 (12)For q ∈ (0, 1/(bκ 1 )]. From Theorem 3, p 1 = κ 1 q and p 2 = κ 2 q are the optimal prices that firm B can set when q ≤ 1/(bκ 1 ). Here, the constraint that ensures firm B will only prioritize the first task reduces toqκ 1 a 1 exp(-bκ 1 q) > qκ 2 (a 1 + a 2 ) exp(-bκ 2 q) ⇒ κ 1 κ 2 a 1 a 1 + a 2 > exp(bq(κ 1 -κ 2 )q) ⇒ b(κ 1 -κ 2 )q < log κ 1 κ 2 + log a 1 a 1 + a 2 ⇒ q < log κ1 κ2 + log a1 a1+a2 b(κ 1 -κ 2 ) .Above, the first line follows from rearranging the terms and the second line follows from taking the log on both sides. The third line follows from rearranging the terms. Therefore, for q in this regime, we can solve the problemEQUATION)We now show that when condition ( 9) is satisfied, problems ( 12) and ( 13) join to have one continuous feasible set, whereas when the condition is not satisfied, problem ( 12) is infeasible and the minimum disappears.First, to see that when the condition is satisfied, the two problems have one feasible set, we must prove1 b(κ 1 -κ 2 ) log κ 1 κ 2 + log a 1 a 1 + a 2 ≥ 1 bκ 1 . (14) Specifically, - κ 2 κ 1 ≥ W - a 1 exp(-1) a 1 + a 2 ⇒ κ 2 κ 1 exp - κ 2 κ 1 ≤ a 1 exp(-1) a 1 + a 2 ⇒ exp 1 - κ 2 κ 1 ≤ κ 1 κ 2 a 1 a 1 + a 2 ⇒ 1 - κ 2 κ 1 ≤ log κ 1 κ 2 + log a 1 a 1 + a 2 ⇒ κ 1 -κ 2 κ 1 ≤ log κ 1 κ 2 + log a 1 a 1 + a 2 ⇒ 1 bκ 1 ≤ 1 b(κ 1 -κ 2 ) log κ 1 κ 2 + log a 1 a 1 + a 2 .Above, the first line follows from applying the Lambert W function. The second line follows from rearranging the terms and the third line follows from taking the log on both sides. The fourth and fifth lines follow from rearranging the terms and multiplying both sides by 1/b. Note that when this condition is satisfied, the lower bound for problem (12) and the upper bound for problem (13) are equivalent, meaning that we can merge the two disjunctive regions. This results in obtaining problem (10).Next, we first note that when the condition is not satisfied, the inequality in ( 14) is reversed and the minimum on the left-hand-side of the constraint in ( 13) can be removed. We then show that problem (12) becomes infeasible. Specifically,κ 2 κ 1 > -W - a 1 exp(-1) a 1 + a 2 ⇒ 1 κ 1 > - 1 κ 2 W - a 1 exp(-1) a 1 + a 2 ⇒ 1 bκ 1 > - 1 bκ 2 W - a 1 exp(-1) a 1 + a 2The second line follows from multiplying both sides by 1/b. This condition means that problem (12) does not have a feasible region, meaning that the optimal price can only be obtained by solving problem (11).Proof of Proposition 1. The proof of the follows from observing that when this condition is satisfied, the intermediate problems from the proof of Theorem 4, i.e., problems ( 12) and ( 13) both become infeasible. If those problems become infeasible, the nominal problems ( 10) and ( 11) also must be infeasible. First,κ 1 κ 2 < a 1 + a 2 a 1 ⇒ log κ 1 κ 2 < log a 1 + a 2 a 1 ⇒ log κ 1 κ 2 + log a 1 a 1 + a 2 < 0The first line follows from taking the log and the second line follows from rearranging the terms and simplifying. When this condition is satisfied, problem (13) becomes infeasible.Next, observe that for any z ∈ [0, 1], we have z ≥ -W(z exp(-1)). Because a 1 /(a 1 + a 2 ) ∈ [0, 1], for any value of a 1 and a 2 , we havea 1 a 1 + a 2 ≥ -W a 1 exp(-1) a 1 + a 2 ⇒ κ 2 κ 1 > -W a 1 exp(-1) a 1 + a 2 ⇒ 1 bκ 1 > - 1 bκ 2 W a 1 exp(-1) a 1 + a 2Thus, problem ( 12) is also infeasible.Proof of Proposition 2. We show that if this condition holds, then problem (10) is the active problemto-solve and that the zero-gradient solution 1/b is a feasible solution. First, note that the condition implies that condition (9) holds, meaning that problem ( 10) is the problem-to-solve. We then note that condition implies1 ≤ - 1 κ 2 W - a 1 exp(-1) a 1 + a 2 min V 1 W 1 , 1 ≤ - 1 κ 2 W - a 1 exp(-1) a 1 + a 2Multiplying both sides by 1/b completes the proof.In this paper, we assume that the generative AI models can be used at a fixed price per-prompt. In practice, generative AI models are priced either per-token or priced via a subscription mechanism where users pay a fixed cost for (potentially unlimited) use. Furthermore, different tasks may feature statistically different numbers of tokens either via longer (or shorter) prompts, or longer (or shorter) model outputs. Consequently under per-token pricing, different tasks will have different costs on average. The price-per-prompt framework of this paper naturally extends to pricing per-token with a small change of variables that preserves all fundamental results.Suppose there are two firms, A and B, developing generative AI models with fixed price per-tokens q 0 and p 0 , respectively. We assume that the price is the same for both input and output tokens. For each task t and model, there is now a distribution of the number of tokens ϕ t and θ t , that the user sends and receives through the respective model in a given prompt. Note that the corresponding price-per-prompt p t and q t are now random variables p t = p 0 θ t and q t = q 0 ϕ t , that depend on the specific task. For any given task t, a user will prefer model B ifp 0 E[θ t n(V t )] ≤ q 0 E[ϕ t n(W t )]where the expectation is taken over the randomness in the number of prompting rounds times the number of tokens per prompting round. The corresponding revenue functions for the two firms areR A (q 0 |p 0 ) := q 0 T t=1 D t (q 0 )1 {q 0 E[ϕ t n(W t )] ≤ p 0 E[θ t n(V t )]} R B (p 0 |q 0 ) := p 0 T t=1 D t (p 0 )1 {p 0 E[θ t n(V t )] ≤ q 0 E[ϕ t n(W t )]} .To solve this problem, note that the inherent structure is equivalent to equation (3). Let κ t := E[ϕ t n(W t )]/E[θ t n(V t )] be the corresponding competitive ratio of the two models and note that the set of tasks can be ranked according to this re-defined competitive ratio. Then, Theorem 1 follows using the same steps and the re-defined κ t . Moreover, all subsequent results carry over from this result.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.In practice, generative AI models are typically priced-per-token. In Appendix B, we show that all our results extend to the price-per-token setting with a minor change of variables. For a list of prices for current generative AI prices, see: https://docsbot.ai/tools/gpt-openai-api-pricing-calculator.For example in code completion, users prefer generated code that provides a good starting point for the users to improve on rather than code that is technically correct but confusing[Ziegler et al., 2024]. 


Decision: The paper studies the problem of pricing generative AI technologies. Their setting assumes different tasks and two firms that sequentially price their tasks based on their performance to maximize their profit.

While the reviewers generally liked the motivation behind the paper and the analysis of the simplified model, they were concerned about several assumptions of the paper. These include the one-shot nature of the game, the assumption of iid results for repeated queries and the lack of any accounting for the cost it takes to generate the technology. The authors had provided to some of these concerns in the rebuttal. While I do not expect a published work to address all these issues, sI encourage the authors to incorporate these to the limitations/discussion of the paper. I would also encourage the authors to include a more in-depth discussion of why the game applies to generative AI companies more clearly in the final version.

Reviews: review: paper_summary: paper_summary: This paper develops a theoretical model of users deciding which generative AI system to use based on the price and probability of performing a task satisfactorily. Based on this theoretical model, the paper then explains how a firm should price their generative AI model in response to other firms, and then how a firm should price their model knowing that other firms will respond. They show that, depending on the ratio of ability to perform a task between the two models, a firm will either get no revenue since they are not price-competitive for anything, charge low enough prices to be price competitive on some tasks, or be so much better at a task than the other firm that they can charge prices which are bound based only on decreasing demand as price increases.
strength_weakness: strength_weakness: 1) The generative AI task performance model seems somewhat unrealistic -- it assumes that given enough user queries the model will always eventually succeed.
questions: questions: 1) Is there an alternative way of modeling task performance that does not assume that the AI system will always succeed?
2) Different tasks might use different numbers of tokens while performing the task, which would result in higher costs for that specific task. Would it change the results a lot if all tasks shared a per-token price but had variable costs for the different tasks based on how many tokens the task requires?
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact for their work.
paper_summary: paper_summary: The paper identifies some unique characteristics of modern generative AI software which affect their pricing. It uses a notion of user cost-effectiveness to compare two models, capturing the cost per prompt and the number of prompting rounds needed to reach a satisfactory answer. The authors propose to model the pricing problem as a game between two companies sequentially releasing their models before users choose their preferred model for each task. A set of tasks is assumed and each task is priced separately. They show that the price optimization problem is piecewise continuous i.e. the companies must choose a subset of tasks on which to be cost-effective and forgo revenue on other tasks. Their analysis indicates that being the first to market may become cost-ineffective, as companies entering the market later can benefit from knowing their competitor's pricing.
strength_weakness: strength_weakness: 1. The setup seems far from reality and loosely related to generative AI. First, in real life even if we assume two companies, there is nothing prohibiting them from updating their prices any number of times and the first mover is probably always in a better position due to customer acquisition, branding, and acquiring real data, system testing, etc. So the analysis seems counterintuitive. 


2. The connection with generative AI is not very clear. What unique characteristics of generative AI are actually used in the pricing model? It is generally applicable to any model with different performance on a set of tasks. 


3. There are no real data experiments or simulations provided in support of the modeling assumptions and analytical results. It is understandable the reality is complex and there are a lot of variables involved in pricing. It would be unreasonable to expect to model all of that but I do expect this study to be a bit more closer to reality than it is. It could be done by showing the real data and formulating the modeling assumptions from it and the pricing game with multiple parties going over multiple rounds and also improving their models. It would be great if with the proposed model the future price can be predicted accurately.
questions: questions: See above.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper studies the pricing and competition of companies providing services using generative AI models. This paper proposes a stylized economic model that abstracts away from the technical details, and in particular, considers two companies entering the market sequentially. This paper assumes that the customer chooses the model that is cheaper per prompt, and each company solves a revenue maximization problem. Based on the model and analysis, this paper argues that companies should adopt different pricing strategies based on their order of entering the market.
strength_weakness: strength_weakness: 1. Several fundamental model setups and assumptions are not very justified. 
1. The insights derived are not very informative nor it provide very meaningful guidance.  Overall, I understand that the parsimonious model aims to highlight certain main trade-offs, but this paper somewhat errs on the side of oversimplification. 
1. Another big concern is that: the nature of analysis in this paper is very stylized and economic, and it is unclear if this is the right venue for such kind of papers..
questions: questions: 1. Is the sequential order of companies true? Would simultaneous competition be more reasonable? Furthermore, if so, do they get any first-mover advantage? The current model assumes that the first company is disadvantaged due to a lack of price information. 
1. The current modeling seems to assume that customers are very rational and could make a sensible decision based on knowledge of price and number of prompts $n$. What about the cases when consumers do not have a good idea of the number of prompts needed or the success rate? What if customers have heterogeneous types or they have heterogeneous tasks?
1. The i.i.d. assumption of $V_t$ is concerning. Due to the conversational nature, the prompts are naturally highly correlated, and thus the independence of the success rate is not justified. 
1. Can authors start the motivation with some more practical background? 
1. To what extent does the current analysis extend to a more general scenario with multiple companies? 
1. Punctuations are missing from several math equations.
limitations: limitations: N/A
paper_summary: paper_summary: * The paper explores the optimal pricing problem for generative AI services in a two-firm Stackelberg competition setting.
* In the setting, two generative AI firms compete over users. Each generative model is characterized by a fixed price $p$ per query, and success probabilities $(V_1, \\ldots, V_T)\\in[0,1]^T$ for each task. The demand for task $t$ at price $p$ is characterized by a non-increasing demand function $D_t(p)$. Users are assumed to query the generative model repeatedly until a satisfactory result is obtained, and success probability of each query ($V_t$) is assumed to be independent and identical. Two generative AI firms compete over users in a Stackelberg setting (firm A sets prices first, firm B follows), and their goal is to maximize revenue.
* For firm B, Theorem 1 characterizes the price optimization problem as a piecewise optimization problem. For firm A, Theorem 2 characterizes the price optimization problem as a bi-level optimization problem. Section 5 assumes an exponential parametric form for the demand function, and derives a globally-optimal solution based on them.
strength_weakness: strength_weakness: * The analysis seems to rely on the assumption that a full taxonomy of tasks is available, and that demand curves for each task are independent. However, due to the general-purpose nature of generative models, tasks may change over time, and therefore identifying and estimating all demand curves may be infeasible.
* Assumed cost model (pay-per-query) doesn’t seem to be common - Current common pricing schemes for generative models are per-token (e.g, OpenAI’s GPT-4 API), or per-month (e.g, ChatGPT), possibly inducing different incentive structures.
* Code is not provided, making it harder to reproduce results and build upon them.
questions: questions: * What is the computational complexity of the optimization problem in Theorem 1?
* In the presented setting, can the firms benefit from colluding?
* What are the expected consequences of having more than two firms participating in the market?
* How would results change if the generative model was assumed to be able to "fail" in generating a satisfactory response? (either due to lack of ability, or users that churn before reaching satisfactory results) 
* How would results change if the two firms are allowed play simultaneously?
* Minor typos:
  * L223 - “Theorem holds..” - Missing theorem number?
  * L239 - “cn” - Can?
  * L356 - “assumes” - assume?
  * L357 - “fims” - firms?
limitations: limitations: Most limitations are discussed explicitly, and in sufficient detail. I feel that the paper can benefit from discussing the additional limitations that appear above.
paper_summary: paper_summary: The paper presents a theoretical analysis of pricing strategies for generative AI models in a competitive market. It investigates how companies can optimally set prices for AI models that are used across various tasks, considering the impact of competition from other firms. The study uses a game-theoretical approach to model the interactions between firms, exploring the implications of different pricing strategies on market dynamics and revenue optimization.
strength_weakness: strength_weakness: 1.	From the perspective of a stylized model, the work is solid and interesting. But I am not sure about the practice relevant. I am not sure whether the real world LLM company is thinking their pricing strategy in this way. I won’t push along this line, and would like leave the decision on this point to more senior reviewers and ACs.
2.	The main technical contribution to me is the modeling the GenAI’s prompt-based service. The analysis seems very standard.
3.	I am not fully convinced that in the current GenAI market, the companies are following the dynamics of model A and model B in the paper. All the companies should be keeping updated their accuracy and be almost homogenous in terms of pricing.
questions: questions: See previous comments.
limitations: limitations: See previous comments.

Comments: title: Thanks
Thanks
Thanks
Thanks for the feedback and additional references
Thanks
Score adjusted
comment: Thank you very much for the improved score and opinion on our paper. Thanks also for your valuable feedback and comments on this work.
Thank you very much for your valuable feedback and comments on this work.
Thank you very much for your valuable feedback and comments on this work.
Thank you for the detailed response. The responses increased my opinion of the paper both by adding new results and being simple enough that it increases my opinion of the original contributions. I am increasing my overall score from 6 to 8 and contribution score from 3 to 4.
Thank you for the thorough response, and for the helpful clarifications! I have no further questions.
Thank you for the response. I appreciate the paper as an early attempt towards pricing in generative AI. However, I am not convinced about the main technical challenges/distinctions specific to pricing in this context. Thus I will keep my original recommendation of borderline accept. I'd also encourage authors to include a more comprehensive related work and a clear delineation of the technical challenges specific to this setting in the next version. Thanks again and good luck!
I have read the rebuttal carefully. Thanks for your efforts and your responses.
Thanks for the detailed feedback and discussion. The related research literature on data pricing and fair model valuation is relevant and we are happy to revise our paper with this discussion.

---

> If we just change  to price per inference call, then I believe the whole story can be written for any other machine learning model.

Pricing per-prompt/per-token in generative AI and pricing per-instance in classical ML models are different because generative models uniquely permit multiple calls to obtain a good answer. For example in a reading QA task, if a promptable LLM gets a wrong output, we query again with a revised prompt, thus paying twice to get the right answer; if a classical ML model is incorrect, the user has no such recourse and must deal with the incorrect answer. This difference necessitates different model valuation strategies (for instance, [1] uses $V\times$ the amount a user is willing to pay for a marginal performance increase). **We believe that the user valuation of a generative AI model should be $p E[n(V)]$**. Note this valuation would not make sense for classical ML products.

Our valuation and corresponding Eqn 1 specializes all our downstream results, whereas classical ML models would require an alternative to Eqn 1 (e.g., [1]), which then would require a different revenue function and optimal solution structure. Most importantly, our valuation function naturally combines with the property of multiple tasks with different demands, thereby giving our differentiation property (Proposition 2), which does not have any place in classical ML products.

[1] https://dl.acm.org/doi/abs/10.1145/3328526.3329589

---

> The value of data is largely ignored. I believe a first mover has a significant advantage in obtaining the real data and improve their product and this cycle of rich gets richer continues i.e. better model --> more users --> more data --> better model.

We agree that the value of data is an important factor. We characterize data into two types, internal and external, that have different properties. We will revise our paper with the following discussion.

1. **Data that the model developer obtains independently:** This data directly impacts model performance, e.g., via a scaling law $V \propto a n^{-b}$ where $n$ is the amount of data. In our pricing problem, **the effect of this data can be directly incorporated by substituting the scaling law into the model performance parameter.** To this end, all our results that speak to requiring a minimium model performance also imply a minimum amount of training data required, e.g., Proposition 2 imposes a maximum value on $\kappa_2 / \kappa_1 := \frac{V_2 W_1}{V_1 W_2}$ as functions of $n$).

2. **Data collected via the AI flywheel:** We agree that this is important for model development, especially in the long-tail. We also agree that your suggestion is the natural next step for our work. However, this analysis presents the following concerns:

   - *Task-specific data may not always be available from the flywheel.* For example, ChatGPT offers data collection opt-out for customers and does not train on enterprise data [2]. Thus, the relationship between the AI flywheel and model performance on custom tasks is indirect (e.g., releasing a better model early to draw more users may improve overall/general model quality [3], but have limited effect on some specific tasks). 
   
   - *Analyzing this requires a more complex framework for which the current work is a necessary precursor.* For example, the extension requires a dynamic problem with time-variant variables for price and performance (or equivalently, data). This complexity, combined with the length and novelty of our current study, motivates us to leave the dynamic extension for follow-up work. 

[2] https://openai.com/enterprise-privacy/

[3] Huseyin Gurkan and Francis de Véricourt. Contracting, pricing, and data collection under the ai flywheel effect.
Thank you for the rebuttal. I have read it and my concerns about the connection with generative AI and "realisticness" of the setup/assumption remain the same. If we just change  $p$  to price per inference call, then I believe the whole story can be written for any other machine learning model. 

The paper does not do justice to the title/motivation. It is overall an oversimplified economic setup leading to some insights which seem unrealistic and of little value that are tied with the generative AI hype. I expected to learn more from this paper, particularly what makes the pricing problem interesting and challenging and some realistic solutions even in the 2 player setting. 

The value of data is largely ignored. I believe a first mover has a significant advantage in obtaining the real data and improve their product and this cycle of rich gets richer continues i.e. better model --> more users --> more data --> better model.  

Having said that, I do not expect a research paper will solve the problem in entirity, but I hope to learn more about the problem and why the solution makes sense. I suggest taking a look at the following references, ( some of them are non-academic blog posts ).

1. https://every.to/p/how-to-price-generative-ai-products 

2. https://sada.com/blog/generative-ai-pricing/ 

Pricing data and models

3. https://dl.acm.org/doi/abs/10.1145/3328526.3329589 

4. https://arxiv.org/pdf/1206.6443 

5. https://openreview.net/pdf?id=Y6IGTNMdLT

6. https://arxiv.org/abs/2312.04740 

7. https://arxiv.org/pdf/2108.07915

Based on the current state, I reserve my borderline score.
Thank you very much for the improved score. We appreciate your feedback and the overall review process. We are happy to continue discussing the paper and answering any further questions or concerns that come up.
I read the rebuttals thoroughly and appreciate the authors' detailed and thoughtful responses to my questions as well as similar concerns by other reviewers. I have thus adjusted my score accordingly.