Title: Fairshare Data Pricing via Data Valuation for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: data pricing

data market

large language model

data valuation

data attribution
Abstract: Training data is the backbone of large language models (LLMs), yet today’s data markets often operate under exploitative pricing -- sourcing data from marginalized groups with little pay or recognition. This paper introduces a theoretical framework for LLM data markets, modeling the strategic interactions between buyers (LLM builders) and sellers (human annotators). 
We begin with theoretical and empirical analysis showing how exploitative pricing drives high-quality sellers out of the market, degrading data quality and long-term model performance. 
Then we introduce fairshare, a pricing mechanism grounded in data valuation that quantifies each data’s contribution. It aligns incentives by sustaining seller participation and optimizing utility for both buyers and sellers. 
Theoretically, we show that fairshare yields mutually optimal outcomes: maximizing long-term buyer utility and seller profit while sustaining market participation. 
Empirically when training open-source LLMs on complex NLP tasks, including math problems, medical diagnosis, and physical reasoning,
fairshare boosts seller earnings and ensures a stable supply of high-quality data, while improving buyers’ performance-per-dollar and long-term welfare. Our findings offer a concrete path toward fair, transparent, and economically sustainable data markets for LLM. Our code will be open sourced.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theoretical framework to model the LLM data market, where buyers and sellers make decisions with transparent information. Theoretical and simulation results show that exploitative pricing leads to a lose-lose situation where data sellers exit the market, leaving buyers with data shortage. Furthermore, the authors propose a fairshare pricing mechanism that theoretically and empirically lead to a win-win situation, where data sellers obtain optimal profits and the buyers obtain long-run utility. Finally, across three LLMs, four tasks, and two budgets, the proposed fairshare pricing achieves the win-win situation as the theory suggests. These empirical results are consistent across several data valuation methods that are scalable to LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This work formalizes the LLM data market with a theoretical framework.\n- This work proposes a fairshare pricing mechanism based on the theoretical framework. This can be impactful for policymaking and the fairness domain.\n- The lose-lose outcome from exploitative pricing and the win-win outcome of the proposed fairshare pricing are well supported with theoretical and simulation results, as well as empirical results with actual LLM fine-tuning.\n\n**Weaknesses**\n- The seller-side pricing under fairshare pricing (Section 3.3) does not seem to take the cost $c_j$ into consideration, which is inconsistent with the overall market framework. In particular, $p^*_j$ may be outside $\\cup_k MWP_k$ if $c_j > MWP_k$ for all $k = 1, ..., M$. I think a minor assumption on $c_j$ is needed for Lemma 2 to hold. Intuitively, $c_j$ should not be larger than a buyer's budget and marginal utility.\n- Overall, Assumptions 1, 1.1, and 2 are reasonable. However, the authors should comment on how their results would change if these assumptions are broken.\n\nQUESTIONS:\n- Intuitively, Assumptions 1 and 1.1 suggest that the sellers are quite likely to exit the market once they are exploited. Can the authors comment on how realistic this is in the real world? What if the lower bound in Assumption 1.1 is replaced with an upper bound, and $P_T$ in Assumption 1 is not multiplicative, suggesting that sellers are more tolerant to exploitation?\n- In Figure 2(c), at Buyer 1's MWP, why is the profit (~2.0) larger than the price (<2.0)?\n- This is a minor comment. Should $R_{\\ge 0}$ in line 139 be replaced by $R$ for the range of the data valuation function?\n\nLIMITATIONS:\nThe limitations seem adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a theoretical fairshare data pricing mechanism that maximizes the seller's profits while aligning with the dataset's utility and the buyer's budget. \nThe mechanism relies on data valuation methods to determine the value of data, and the authors use data for training LLMs as a motivating example, theoretically and empirically.  \nThe authors also empirically test their framework in a semi-synthetic setting involving two buyers, three open-source LLMs, and 10 sellers.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The authors provide a nice background and motivation for their proposed framework of data pricing. \n- The theoretical model is well-motivated, technically sound, clearly written, and easy to parse. I especially like the authors' idea of incorporating both sellers' and buyers' utility and the long-term effects of (un)fair data pricing.\n- The motivating empirical results in Figures 2 and 3 are intuitive and complementary to their setup. \n- In the empirical experiments, the authors follow up with simulations specific to the LLM running example. \n\n**Weaknesses**\n\n- Mismatch between motivation and proposed framework. \n  - Although the proposed model is technically sound, it doesn't match the motivation of the work. The authors highlight the ethical concerns in LLM data acquisition, for example, the Sama company, Kenyan workers, and the OpenAI case ([11,12]). Accordingly to the theoretical framework, the sellers are actually companies like Sama, and not the Kenyan annotators. Additionally, the authors mention that the sellers would stop getting involved in the data market if not fairly treated, but as it was in this case, they simply move to new sellers because there is always someone willing to. So entry and exit of data buyers and sellers in the market should ideally be covered.\n  - There might be a decrease in data quality at time t, but a rise in quality at time t+x due to unfair data market or other factors outside of this. The model doesn't account for the changes in data quality at various time steps or for the entry and exit of different sellers and buyers in the data market, due to exploitative pricing and other factors. \n\n- Sections 3.1 and 3.2 and Appendix A, B and C\n  - I found Figure 1 confusing and unintuitive. For example, the data valuation methods require a model performance to compute the value of data, which is not clear from the figure. The relationship between the seller and buyer is oversimplified. For example, from the follow-up text, it seems to me there might be a back and forth between the two, sellers need potentially broadcasted buyers decision-making, and buyers need the data to compute value and set the price - to make decisions.\n  - The model of the seller's decision-making (Eqs. 3 and 4) has strong assumptions. For example, a lay seller, e.g., an annotator, might not have access to the seller utility or other data sources. As such, computing a net profit function is not trivial, and sellers might over- or underprice their data. Also how do we know the fixed price (line 158) is fair? Can buyers only get a fraction of seller data (Section 3.1 and Appendix B)?\n  - In lines 141 to 144, the authors show that utility (not the usual language of utility in dval literature) is a function of data values. It's unclear to me how well or accurately this formulation of utility captures gains such as commercial returns that buyers care about (line 196).   - The utility formulation has other limitations. In line 674, how high is high?  What's the threshold for determining that? All the utility formulations (C.1 to C.3) rely on marginal contribution-based data valuation methods defined in lines 141. I imagine it could be possible to combine various forms of valuating data to determine the utility.\n  - The setup for the results in Figure 3 is oversimplified. For example, why is utility u constant?\n\n- Experimental setups and analyses \n  - Given that authors consider varied tasks, why do they only use the affine mappings and not test the other formulations of utility? (c.2, c.3, c.4)? \n  - There is little justification or reasoning given for the choice of various parameters, e.g., how did the authors choose \\delta and T?\n\n- Limitations or discussions section \n  - Given that the authors make so many assumptions and the framework could be understood from various viewpoints, I think the paper should have a limitations section or discussions to include some of these perspectives and assumptions.\n\nQUESTIONS:\n- Mismatch between motivation and proposed framework. \n  - Although the proposed model is technically sound, it doesn't match the motivation of the work. The authors highlight the ethical concerns in LLM data acquisition, for example, the Sama company, Kenyan workers, and the OpenAI case ([11,12]). Accordingly to the theoretical framework, the sellers are actually companies like Sama, and not the Kenyan annotators. Additionally, the authors mention that the sellers would stop getting involved in the data market if not fairly treated, but as it was in this case, they simply move to new sellers because there is always someone willing to. So entry and exit of data buyers and sellers in the market should ideally be covered.\n  - There might be a decrease in data quality at time t, but a rise in quality at time t+x due to unfair data market or other factors outside of this. The model doesn't account for the changes in data quality at various time steps or for the entry and exit of different sellers and buyers in the data market, due to exploitative pricing and other factors. \n\n- Sections 3.1 and 3.2 and Appendix A, B and C\n  - I found Figure 1 confusing and unintuitive. For example, the data valuation methods require a model performance to compute the value of data, which is not clear from the figure. The relationship between the seller and buyer is oversimplified. For example, from the follow-up text, it seems to me there might be a back and forth between the two, sellers need potentially broadcasted buyers decision-making, and buyers need the data to compute value and set the price - to make decisions.\n  - The model of the seller's decision-making (Eqs. 3 and 4) has strong assumptions. For example, a lay seller, e.g., an annotator, might not have access to the seller utility or other data sources. As such, computing a net profit function is not trivial, and sellers might over- or underprice their data. Also how do we know the fixed price (line 158) is fair? Can buyers only get a fraction of seller data (Section 3.1 and Appendix B)?\n  - In lines 141 to 144, the authors show that utility (not the usual language of utility in dval literature) is a function of data values. It's unclear to me how well or accurately this formulation of utility captures gains such as commercial returns that buyers care about (line 196).   - The utility formulation has other limitations. In line 674, how high is high?  What's the threshold for determining that? All the utility formulations (C.1 to C.3) rely on marginal contribution-based data valuation methods defined in lines 141. I imagine it could be possible to combine various forms of valuating data to determine the utility.\n  - The setup for the results in Figure 3 is oversimplified. For example, why is utility u constant?\n\n- Experimental setups and analyses \n  - Given that authors consider varied tasks, why do they only use the affine mappings and not test the other formulations of utility? (c.2, c.3, c.4)? \n  - There is little justification or reasoning given for the choice of various parameters, e.g., how did the authors choose \\delta and T?\n\n- Limitations or discussions section \n  - Given that the authors make so many assumptions and the framework could be understood from various viewpoints, I think the paper should have a limitations section or discussions to include some of these perspectives and assumptions.\n\nLIMITATIONS:\nGiven that the authors make so many assumptions and the framework could be understood from various viewpoints, I think the paper should have a limitations section or discussions to include some of these perspectives and assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This work addresses the challenges posed by exploitative pricing (i.e. low pricing from buyers to data annotators where compensation does not reflect skill, effort, and downstream value of contributions) in LLM data markets. Specifically, the authors demonstrate theoretically and empirically that exploitative pricing results in a lose-lose scenario with regards to data sellers being underpaid and data buyers driving sellers away and thereby weakening the data pipeline and limiting model improvements. Based on this, the authors propose a fairshare pricing mechanism that they theoretically show results in win-win scenarios, where sellers maximize profits and buyers achieve higher model performance per dollar spent. Simulation experiments. are used to validate fairshare, and ablations demonstrate the generalizability across different models, tasks, and data valuation methods.

STRENGTHS AND WEAKNESSES:
Strengths:
- The work is well-written, theoretically sound, and addresses an important, timely topic.
- As the proposed fairshare pricing mechanism is model and method agnostic, it is flexible as new models and data valuation methods are developed, and flexible to real-world applications desired by potential data-market participants. 

Weaknesses:
- Shapley-based data valuation methods are not discussed in the work, nor compared against in the ablation study. Methods such as In-Run Data Shapley (https://openreview.net/pdf?id=HD6bWcj87Y) should make comparison computationally feasible.

QUESTIONS:
- The Related Work section (beginning line 106) should include a discussion of Shapley-based data valuation methods.

- 17: LLM --> LLMs

- 49: drive --> driving

-  98: thorugh --> through

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSetting:  \nThe paper studies data markets over repeated interactions between persistent sellers (data providers who create and sell data) and a buyer (ML model trainer) in a full information setting i.e. the buyer data valuation is known to all parties. They also define ${p^\\ast}_i$ be the maximum price the buyer is willing to pay seller $i$ for their data, taking their budget and marginal utility into account. This price is clearly the best price for the seller - any more and the buyer drops out and any less the seller leaves some money on the table.\n\nResults:  \nThe authors introduce a new assumption about seller behavior - that if their data is priced under the maximum price ${p^\\ast}_i$, the seller $i$ is significantly less likely to take part in future markets. The authors show that this implies that consistently pricing under this threshold means that sellers dropout, leading to market collapse. Thus, under the assumptions introduced, it is optimal for buyers as well to price at ${p^\\ast}_i$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Data valuations are a very important and fast-growing topic. However, past work has mostly ignored the game theoretic aspect. I really like how this work attempts to combine the ML / empirical data valuations (such as influence estimation or attribution techniques) with more classical game theoretic investigations. I see a scope for a lot more such follow up research.   \n- The repeated interactions model has also been understudied in data markets and, as this paper shows, yields interesting mechanisms and behaviors.  \n\nWeaknesses\n\n- The biggest flaw of the work is the assumption about seller behavior - that pricing less than ${p^\\ast}_i$ leads to increase in dropout likelihood. This usage of ${p^\\ast}_i$ as the threshold used is extremely artificial since ${p^\\ast}_i$ has to do with buyer purchase-power and has nothing to do with the seller. In particular, the seller behavior changes given a different set of buyers. A more realistic model would use a buyer-independent threshold based on cost of data creation to the seller. However, in such a model the conclusions shown here would not work.   \n- Similarly, I contest the naming of ${p^\\ast}_i$ to be the \"Fairshare\" data pricing -- by definition it is the seller-optimal prices. It is unclear how this price is \"fair\".  \n- There is also insufficient literature review. In particular, a lot of the work in incentivizing data sharing in federated learning might be highly relevant here. See for e.g. citations in this very recent (parallel) work [[Pang et al. 25](https://arxiv.org/abs/2506.16731)] or [this recent survey](https://www.sciencedirect.com/science/article/abs/pii/B9780443190377000260).  \n- Finally, the existing work assumes a full information setting where all the participants know the true valuation and budgets of everyone. While this is unlikely in practice, I expect this aspect can be improved upon in future research.  \n\n\nMy overall impression is that this is an intriguing but flawed first step along a new research direction.\n\nQUESTIONS:\nWould similar results hold under a more justifiable seller behavior model as described in weaknesses? Can the authors provide justification for their choice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-11"}