<figure><img src="image_1.png" /></figure>

HAL
open science

# Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic

Mélissa Tamine, Otmane Sakhi, Benjamin Heymann

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mélissa Tamine, Otmane Sakhi, Benjamin Heymann. Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic. 2026. hal-05476168

HAL Id: hal-05476168

https://hal.science/hal-05476168v1

Preprint submitted on 26 Jan 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic

MÉLISSA TAMINE*, Criteo AI Lab, Inria, Fairplay joint team, France

OTMANE SAKHI*, Criteo AI Lab, France

BENJAMIN HEYMANN*, Criteo AI Lab, Fairplay joint team, France

Data is a critical asset for training large language models (LLMs), alongside compute resources and skilled workers. While some training data is publicly available, substantial investment is required to generate proprietary datasets, such as human preference annotations or to curate new ones from existing sources. As larger datasets generally yield better model performance, two natural questions arise. First, how can data owners make informed decisions about curation strategies and data sources investment? Second, how can multiple data owners collaboratively pool their resources to train superior models while fairly distributing the benefits? This problem, data valuation, which is not specific to large language models, has been addressed by the machine learning community through the lens of cooperative game theory, with the Shapley value being the prevalent solution concept. However, computing Shapley values is notoriously expensive for data valuation, typically requiring numerous model retrainings, which can become prohibitive for large machine learning models. In this work, we demonstrate that this computational challenge is dramatically simplified for LLMs trained with Direct Preference Optimization (DPO). We show how the specific mathematical structure of DPO enables scalable Shapley value computation. We believe this observation unlocks many applications at the intersection of data valuation and large language models.

Additional Key Words and Phrases: Shapley value, Data valuation, Large-language models (LLMs), Fine-tuning, Direct preference optimization (DPO), Language Model Arithmetic

## 1 INTRODUCTION

Large language models (LLMs) are the result of collaborative training and alignment pipelines: a single deployed model may have been pre-trained on heterogeneous web-scale corpora, often mixed with proprietary data sources, and then adapted through instruction tuning [30] and several stages of preference-based alignment, such as *Reinforcement Learning from Human Preferences* (RLHF) [6, 18, 45] typically implemented with policy-gradient methods like *Proximal Policy Optimization* (PPO) [33], and more recent methods such as *Direct Preference Optimization* (DPO) [31] or weak-to-strong supervision schemes [4]. Recent scaling-law studies show that, beyond architecture, LLMs' performance is mainly driven by the amount, diversity, and quality of training data [15, 21]. This reflects the adage that *data is the new oil* [19]. The key scarce resource of LLMs is not the model design, but the *data* provided by companies, institutions, and user communities: people are paid to label and rank model outputs, LLMs themselves are used to generate additional training data, and companies engage in legal battles over access to valuable corpora. This raises a central *data valuation* question: *how should we attribute the contribution of each data source to the final behavior of an LLM?* Answering this question is not just of academic interest: data valuation (i.e., a systematic way to attribute value to a data source) is a prerequisite for data markets (*what is a fair price for a dataset?*) [1, 5], contractual guarantees (*what level of performance can we promise to a contributor?*), incentive design (*how should we reward agents whose data improves alignment?*) [22, 35], and even basic notions of responsibility (*which data source made the model toxic on an input?*) [12, 19, 20, 38].

Cooperative game theory, in particular the Shapley value [34], provides a formalism for such data valuation problems:

<footnote>*Equal contribution.</footnote>
interpret each data source as a player, define the utility of any coalition of players, and use the Shapley value to fairly split the utility among all players. However, directly instantiating this paradigm for LLM fine-tuning is computationally prohibitive. In Shapley-based data valuation, the core bottleneck is that the utility must be evaluated for every coalition of data sources, and the number of such coalitions grows exponentially with the number of sources [12, 34]. Even in classical supervised learning, this already leads to an impractical number of model retrainings. Still, the situation is worse for LLMs. Each utility evaluation now requires fine-tuning a large model. With PPO or DPO, this means running a complete preference optimization loop on each coalition of data sources, even if one relies on emulation or distillation techniques that use a smaller model to approximate the effect of fine-tuning the LLM [27] (such methods may lower the cost per run, but they do not remove the exponential number of coalition-specific runs).

The present proposal addresses this obstacle. We focus on preference-based fine-tuning, and in particular DPO, where the objective is defined directly over pairs of preferred and dispreferred responses [31]. Our key observation is that training sequentially across multiple datasets corresponds to summing the reward models learned from each dataset independently. Hence, in this setting, the utility of a coalition of datasets need not be defined by a dedicated fine-tuning, and we can take inspiration from recent work on language model arithmetic [9]: starting from one base model and a collection of models, each fine-tuned on a single dataset, one can construct at inference time composite models that approximately capture the effect of training on unions of datasets, by combining their output probabilities using a simple arithmetic rule over the fine-tuned models. Building on this observation, we propose a Shapley value approximation method for LLMs that reduces the number of required fine-tunings from exponential to linear in the number of data sources. We first use DPO to fine-tune one model per data source, then apply model arithmetic to construct, at inference time, an approximate model for any coalition of sources (a *coalition model*). The utility of a coalition is then defined as the performance of the coalition model on a fixed evaluation task. This enables estimating the Shapley value for all data sources while performing only one fine-tuning per source.

*Related works.* There are two streams of Shapley value applications for aligned LLMs: one focused on *model explainability* for token or feature-level attribution of predictions [16, 40, 42, 44], and another focused on *data valuation* for quantifying training data contributions [13, 28, 32, 41]. Our work is orthogonal to the first stream and contributes to the second. Its novelty, compared to prior data valuation studies for LLMs, is to exploit the specific mathematical structure of preference-based alignment (DPO combined with language model arithmetic) to reduce the cost of computing coalition utilities.

In this work, we show that combining DPO with the philosophy of language model arithmetic enables us to approximate coalition utilities without performing coalition-specific fine-tunings.

# 2 PRELIMINARY

## 2.1 Evaluation of LLMs

We consider an aligned LLM as a stochastic policy $\pi: \mathcal{X} \to \Delta(\mathcal{Y})$ mapping prompts $x$ to distributions over responses $y$, where $\Delta(\mathcal{Y})$ denotes the probability simplex over the response space $\mathcal{Y}$. Let $\mathcal{D}$ be the evaluation prompt distribution (e.g., from a held-out validation set) and $r: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ a reward function measuring response quality (e.g., from a
<figure><img src="image_4.png" /><figcaption>Fig. 1. We can equip the collections of datasets, the set of reward models, and the set of policies with binary operators to give them a group (or semi-group) structure. After doing so, we observe that some training methods, such as sequential DPO, preserve this structure and therefore induce morphisms between those semi-groups.</figcaption></figure>

reward model). We define the value of a policy $\pi$ as its expected reward:

$$v(\pi) = \mathbb{E}_{x \sim D, y \sim \pi(\cdot|x)} [r(x, y)]. \qquad (1)$$

Suppose we have a set $\mathcal{N} = \{1, \dots, n\}$ of data sources, where each source $i \in \mathcal{N}$ provides a dataset $\mathcal{D}_i$ of preference pairs (or, more generally, alignment examples). For any coalition $S \subseteq \mathcal{N}$, let $\pi_S$ denote the model obtained by applying a fixed alignment procedure (e.g., DPO) with fixed hyperparameters and a reference model, to the union of the corresponding datasets $\cup_{i \in S} \mathcal{D}_i$.

The utility of a coalition $S$ is then the value of its corresponding policy:

$$u(S) = v(\pi_S). \qquad (2)$$

In practice, $u(S) = v(\pi_S)$ is estimated empirically by averaging $r(x_j, y_j)$ over a finite validation set $\{x_j\}_{j=1}^m$ and sampled responses $y_j \sim \pi_S(\cdot | x_j)$.

## 2.2 Shapley-based data valuation

Given the utility $u(S) = v(\pi_S)$ defined in Equation (2), the Shapley value [12, 34] provides a game-theoretic method for splitting the total utility $u(\mathcal{N}) = v(\pi_\mathcal{N})$ (achieved when aligning on all data sources) among individual sources. Formally, for each data source $i \in \mathcal{N}$, its Shapley value is

$$\varphi_i = \sum_{S \subseteq \mathcal{N} \setminus \{i\}} \frac{|S|!(n - |S| - 1)!}{n!} [v(\pi_{S \cup \{i\}}) - v(\pi_S)]. \qquad (3)$$

The popularity of using the Shapley value to perform data valuation stems from the fact that it is the unique value notion satisfying four axioms (efficiency, symmetry, dummy, and linearity) that are economically desirable [34].

However, one challenge of performing Shapley-based data valuation is its computational complexity. Evaluating the exact Shapley value of each data source using Equation (3) involves computing the marginal utility $v(\pi_{S \cup \{i\}}) - v(\pi_S)$ of every source $i$ to every coalition $S$, which is $O(2^{|\mathcal{N}|})$. Such exponential computation is not feasible in any realistic setting, as computing exact Shapley values would entail an exponential number of full LLM finetunings, far beyond what is feasible at modern model scales.
## 2.3 LLM fine-tuning

LLM fine-tuning, and especially preference alignment [30, 45], is a crucial step that guides pre-trained models toward desired behaviors and underlies much of their success in chat applications [30]. The most successful and widely used preference alignment methods, for instance Reinforcement Learning from Human Feedback (RLHF) [45] and Direct Preference Optimization (DPO) [31], are motivated through KL-regularized expected-reward maximization [31, 45]. In these approaches, our objective is to align a pre-trained LLM, denoted by $\pi_0$, on a given coalition of datasets of preferences $\mathcal{D}_S = \cup_{\ell \in S} \mathcal{D}_\ell$ with $\mathcal{D}_\ell = \{x_{i,\ell}, y_{i,\ell}^+, y_{i,\ell}^-\}_{i \in [n_\ell]}$. We first describe the core alignment approaches on a single dataset of preferences $\mathcal{D}_\ell$, before discussing possible alignment approaches on coalitions. RLHF begins by modeling a reward signal $\hat{r}_\ell : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ on $\mathcal{D}_\ell$, where $\hat{r}_\ell(x, y)$ captures the quality of response $y \in \mathcal{Y}$ to the question or prompt $x \in \mathcal{X}$. This reward is modeled with Bradley-Terry and learned with maximum likelihood estimation by solving

$$ \hat{r}_\ell \in \arg\max_{r} \left\{ \sum_{i \in \mathcal{D}_\ell} \log \sigma(r(x_i, y_i^+) - r(x_i, y_i^-)) \right\}. \quad (4) $$

This reward is then used to align a pretrained LLM, denoted by $\pi_0$, and used as a reference, solving the following optimization problem:

$$ \pi_{\ell}^{*} \in \arg\max_{\pi} \left\{ \sum_{i \in \mathcal{D}_{\ell}} \mathbb{E}_{y \sim \pi(\cdot | x_i)} [\hat{r}_{\ell}(x_i, y)] - \beta \mathrm{KL}(\pi(\cdot | x_i), \pi_0(\cdot | x_i)) \right\}, $$

with $\beta > 0$ the KL regularization parameter. This optimization problem admits an analytical solution, giving the form of the optimal aligned policy. For each prompt $x \in \mathcal{X}$, the optimal distribution over responses $y \in \mathcal{Y}$ is

$$ \pi_{\ell}^{*}(y|x) \propto \exp(\hat{r}_{\ell}(x, y)/\beta)\pi_{0}(y|x). \quad (5) $$

Given its intractable normalization, this policy is approximated using policy learning algorithms such as REINFORCE [2] and PPO [30], to name a few. Direct Preference Optimization (DPO) [31] exploits the structure of the optimal policy to bypass the explicit reward modeling and RL steps, treating the LLM itself as the object of optimization. Plugging (5) into the Bradley-Terry likelihood of (4) leads to an equivalent preference-learning objective directly over $\pi$:

$$ \hat{\pi}_{\ell}^{*} \in \arg\max_{\pi} \left\{ \sum_{i \in \mathcal{D}_{\ell}} \log \sigma \left( \beta \left( \log \frac{\pi(y_{i}^{+}|x_{i})}{\pi_{0}(y_{i}^{+}|x_{i})} - \log \frac{\pi(y_{i}^{-}|x_{i})}{\pi_{0}(y_{i}^{-}|x_{i})} \right) \right) \right\}. $$

Thus, under the KL-regularized expected-reward framework, DPO targets the same family of optimal policies as RLHF on $\mathcal{D}_\ell$, but does so implicitly through a single maximum-likelihood stage without training an explicit reward model. In the following, without any loss of generality, we use DPO as our alignment algorithm due to its ease of implementation and practicality.

## 3 COALITION POLICIES AND LANGUAGE MODEL ARITHMETIC

After describing the core alignment algorithms on a single dataset, we now extend them, with DPO as a representative example, to subsets of data sources (which we will refer to as coalitions in the game theoretic sense). A straightforward approach is to apply DPO directly to the coalition dataset $\mathcal{D}_S = \cup_{\ell \in S} \mathcal{D}_\ell$, solving

$$ \hat{\pi}_{S}^{*} \in \arg\max_{\pi} \left\{ \sum_{\ell \in S} \sum_{i \in \mathcal{D}_{\ell}} \log \sigma \left( \beta \left( \log \frac{\pi(y_{i}^{+}|x_{i})}{\pi_{0}(y_{i}^{+}|x_{i})} - \log \frac{\pi(y_{i}^{-}|x_{i})}{\pi_{0}(y_{i}^{-}|x_{i})} \right) \right) \right\}. $$
While natural, this is not the only way to align a pretrained LLM using datasets {$\mathcal{D}_t$}$_{t \in S}$. Since our goal is to study data valuation, we seek alignment algorithms that expose useful algebraic structure across coalitions (see Figure 1), reducing computational cost and simplifying valuation. To this end, we propose a different coalition alignment procedure, which we call **Sequential DPO**, that is simple to implement and differs only slightly from naïve DPO on aggregations. Algorithm 1 describes this procedure. Despite its sequential implementation, Algorithm 1 should converge to the same

<caption>**Algorithm 1:** Sequential Direct Preference Optimization</caption>

1 **Input:** Reference $\pi_0$, Parameter $\beta > 0$, Coalition $S$.
2 **Initialize:** $k = 0$, $L$ list of indices in $S$.
3 **while** $k < |S|$ **do**
4 &nbsp;&nbsp;&nbsp;&nbsp;Set $\ell \leftarrow L[k]$.
5 &nbsp;&nbsp;&nbsp;&nbsp;Set $k \leftarrow k + 1$.
6 &nbsp;&nbsp;&nbsp;&nbsp;**Output:** $\hat{\pi}_S^* = \pi_{k+1}$.
$$ \pi_{k+1} \leftarrow \arg \max_{\pi} \sum_{i \in \mathcal{D}_\ell} \log \sigma \left( \beta \left( \log \frac{\pi(y_i^+ | x_i)}{\pi_k(y_i^+ | x_i)} - \log \frac{\pi(y_i^- | x_i)}{\pi_k(y_i^- | x_i)} \right) \right). $$

policy regardless of the ordering of datasets in $S$. Indeed, at convergence, the aligned coalition policy satisfies the closed form

$$ \pi_S^*(y|x) \propto \exp\left(\frac{1}{\beta}\sum_{\ell \in S} \hat{r}_\ell(x, y)\right) \pi_0(y|x), \quad (6) $$

which depends only on the set $S$ and not on its enumeration. **Sequential DPO** therefore recovers classical DPO when $|S| = 1$ and extends it naturally to coalitions. Our main question is whether we can bypass coalition-specific alignment entirely by exploiting the structure of the alignment objective. The answer is **yes**: assuming convergence of the alignment algorithms, we can express the coalition-aligned policy $\pi_S^*$ directly in terms of the individually aligned models {$\pi_\ell^*$}$_{\ell \in S}$. Starting from the identities

$$ \log \pi_0(y|x) = s_0(x, y) + C_0(x) \\ \log \pi_\ell^*(y|x) = s_\ell(x, y) + C_\ell(x) = \frac{1}{\beta} \hat{r}_\ell(x, y) + s_0(x, y) + \tilde{C}_\ell(x). $$

and discarding $y$-independent constants that vanish under normalization, we obtain

$$ \pi_S^*(y|x) \propto \exp\left(\frac{1}{\beta}\sum_{\ell \in S} \hat{r}_\ell(x, y)\right) \pi_0(y|x), \\ \propto \exp\left(\sum_{\ell \in S} s_\ell(x, y) + (1 - |S|) s_0(x, y)\right). $$

This simple result is powerful: for any coalition $S$, the coalition-aligned model $\pi_S^*$ can be recovered exactly using only the individually aligned models {$\pi_\ell^*$}$_{\ell \in S}$ and the reference model $\pi_0$. No additional optimization or training on the coalition dataset is required. In other words, the coalition policy can be obtained purely through algebraic operations on already-trained models. This provides a principled and training-free procedure for reconstructing $\pi_S^*$, and constitutes a formal instance of language model arithmetic, where new behaviors emerge from structured combinations of existing
LLMs rather than from further fine-tuning. This perspective closely aligns with the Language Model Arithmetic philosophy, originally introduced in [10].

# 4 IMPLEMENTATION

The goal of this section is to illustrate our approach in a small, easily reproducible setting. We take SmolLM-135M-Instruct [3] as the initial policy $\pi_0$ and fine-tune it with DPO on different subsets of the UltraFeedback dataset [8]. Specifically, we use 4 UltraFeedback sources as data providers: flan_v2_niv2, sharegpt, evol_instruct, and ultrachat. We use the Hugging Face TRL implementation of DPO [39] with the default inverse temperature $\beta = 0.1$. Fine-tuning is performed on a single A100 80GB GPU for 4 epochs, with batch size 32, 4 gradient-accumulation steps, and a learning rate of $2 \times 10^{-5}$. We apply LoRA [17] with rank $r = 8$ and scaling $\alpha = 16$ to obtain one adapter per data source.

In this experiment, we thus have $n = 4$ data sources and compute Shapley values for these $n$ sources using only $n$ DPO fine-tunings (instead of $2^4 = 16$), exploiting the language model arithmetic property derived in the previous section. Concretely, we keep the base model $\pi_0$ frozen and learn $n$ LoRA adapters, one per source. At evaluation time, we construct approximate coalition policies $\hat{\pi}_S$ by combining the corresponding LoRA adapters, and define utilities via $v(\hat{\pi}_S)$. This yields the following Shapley approximation:

$$ \hat{\phi}_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(n - |S| - 1)!}{n!} [v(\hat{\pi}_{S \cup \{i\}}) - v(\hat{\pi}_S)]. \quad (7) $$

For evaluation, we use two scalar reward models from the RLHF literature [43], one trained to measure helpfulness and the other to measure harmlessness. For each coalition model $\hat{\pi}_S$, we generate responses to 128 prompts drawn from UltraFeedback examples whose sources are disjoint from our 4 training subsets (the unused portion of UltraFeedback [8]), and define $v(\hat{\pi}_S)$ as the average reward over these prompts. Figure 2 shows the resulting approximate Shapley values, plotting each data source in the plane defined by its Shapley value under the helpfulness reward (x-axis) and under the harmlessness reward (y-axis).

**Remark.** In our experiments, we evaluate $v(\hat{\pi}_S)$ for all $2^n$ coalitions, so (7) is exact given the coalition models $\{\hat{\pi}_S\}_S$. At larger scales, our contribution can complement standard Shapley approximations. Once coalition utilities can be queried cheaply via our proposed instance of language model arithmetic, one can further subsample coalitions using Monte Carlo permutation sampling [12, 20, 25] or a regression trick [23] to reduce the number of inference calls from $2^n$ to a polynomial in the number of data sources.

The spatial signature [38] in Figure 2 represents the contribution of each source in a multi-objective value space and illustrates our aim at more interpretable LLM training. Each point corresponds to one UltraFeedback subset. We observe heterogeneous profiles: sharegpt has the most significant positive helpful value and a mildly positive harmless value, ultrachat contributes mainly to harmlessness with an almost neutral helpful value, flan_v2_niv2 yields a small helpful gain but slightly harms harmlessness, and evol_instruct provides helpfulness at the cost of the most significant negative harmless value. Given an LLM training, the data signature documents the effective influence of each dataset.

Negative Shapley values here are informative: they identify sources whose marginal contribution, averaged over coalitions, pushes the model in an undesirable direction for a given reward. Overall, the plot demonstrates that our
<figure><img src="image_5.png" /><figcaption>Fig. 2. Approximate Shapley values of 4 UltraFeedback data sources (flan_v2_niv2, sharegpt, evol_instruct, ultrachat) under two rewards. Each point corresponds to a data source. The x-axis shows its Shapley value for the *helpfulness* reward, and the y-axis for the *harmlessness* reward. The diagonal $y = x$ indicates perfect agreement between the two rewards on the relative importance of each source.</figcaption></figure>

method makes it practically feasible to characterize training datasets along multiple alignment dimensions, rather than treating them as undifferentiated additional data.

# 5 VISION

For LLM fine-tuning, sequential training across multiple datasets corresponds to summing the reward models learned from each dataset independently. We argue that this observation has practical implications and opens several research directions.

The Shapley value, and more generally semivalues, could be used to quantify, in an interpretable manner, the contribution of each dataset used during training. In SHAP [24], the Shapley value is used to identify the role of features in the output of the model, with a notable impact on the field of interpretable ML, in the context of LLM, the role of the features could be reinterpreted by the data. We notably foresee applications in contexts where copyright infringement and data ownership are becoming contentious.

Since data quality is key to LLM performance, one could envision new training algorithms that automatically curate or weight their input data, just as it is now typical in machine learning to reweight datasets or individual datapoints to improve model outcomes. Similarly, during training, there is room for Shapley values to improve synthetic data generation, as in NashMD or online DPO. Even further: new architectures could dynamically select model operations online, guided by Shapley values tailored to the specific context or task.
When datasets reflect individual preferences. The algebraic relation we highlight may support new methods at the intersection of social choice and LLM training. Indeed, it has been already observed that LLM are in fact preference aggregators [6, 11, 13, 25, 28, 35]. In what we described in Figure 1, it is the combination of a Bradley-Terry preference aggregation model, a KL-regularized reward objective and a sequential training resulted in a morphism between the dataset collections and the policies. An open research question is whether there exists other combinations of preference aggregation model and training rules such that such a morphism exists.

In parallel to those directions, our research program calls for a better theoretical and empirical understanding of **how sequential DPO relates to DPO**. First, checking how the commutativity property persists in practice, where convergence is not reached, and second, how does the sequential resolution of an optimization problem approximate the original problem.
ACKNOWLEDGMENTS

This work was partially supported by the French National Research Agency (ANR) through grants ANR-20-CE23-0007 and ANR-23-CE23-0002 and through the PEPR IA FOUNDRY project (ANR-23-PEIA-0003). Computational and storage resources were provided by GENCI at IDRIS through allocation 2025-A0191016862 on the Jean Zay supercomputer (V100/A100/H100 partitions).

REFERENCES

[1] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. 2019. A Marketplace for Data: An Algorithmic Solution. In *Proceedings of the 2019 ACM Conference on Economics and Computation* (Phoenix, AZ, USA) (EC '19). Association for Computing Machinery, New York, NY, USA, 701–726. https://doi.org/10.1145/3328526.3329589

[2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 12248–12267. https://doi.org/10.18653/v1/2024.acl-long.662

[3] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. SmollM - blazingly fast and remarkably powerful.

[4] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. 2024. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. In *Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235)*, Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 4971–5012. https://proceedings.mlr.press/v235/burns24b.html

[5] Yiling Chen, Stephen Chong, Ian A. Kash, Tal Moran, and Salil Vadhan. 2016. Truthful Mechanisms for Agents That Value Privacy. *ACM Trans. Econ. Comput.* 4, 3, Article 13 (March 2016), 30 pages. https://doi.org/10.1145/2892555

[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In *Advances in Neural Information Processing Systems*, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf

[7] Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. 2024. Position: social choice should guide AI alignment in dealing with diverse human feedback. In *Proceedings of the 41st International Conference on Machine Learning* (Vienna, Austria) (ICML '24). JMLR.org, Article 371, 15 pages.

[8] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. UltraFeedback: Boosting Language Models with High-quality Feedback. arXiv:2310.01377 [cs.CL]

[9] Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. 2024. Controlled Text Generation via Language Model Arithmetic. In *International Conference on Representation Learning*, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 35011–35038. https://proceedings.iclr.cc/paper_files/paper/2024/file/96aad3299d18497e2bea4fc20b949b81-Paper-Conference.pdf

[10] Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. 2024. Controlled Text Generation via Language Model Arithmetic. In *The Twelfth International Conference on Learning Representations*. https://openreview.net/forum?id=SLw9fp4yI6

[11] Luise Ge, Daniel Halpern, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. 2024. Axioms for ai alignment from human feedback. *Advances in Neural Information Processing Systems* 37 (2024), 80439–80465.

[12] Amirata Ghorbani and James Zou. 2019. Data Shapley: Equitable Valuation of Data for Machine Learning. In *Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97)*, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 2242–2251. https://proceedings.mlr.press/v97/ghorbani19c.html

[13] Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. 2024. SHED: shapley-based automated dataset refinement for instruction fine-tuning. In *Proceedings of the 38th International Conference on Neural Information Processing Systems* (Vancouver, BC, Canada) (NIPS '24). Curran Associates Inc., Red Hook, NY, USA, Article 3153, 22 pages.

[14] Benjamin Heymann. 2025. Adaptive Preference Aggregation. arXiv preprint arXiv:2503.10215 (2025).

[15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. 2022. Training compute-optimal large language models. In *Proceedings of the 36th International Conference on Neural Information Processing Systems* (New Orleans, LA, USA) (NIPS '22). Curran Associates Inc., Red Hook, NY, USA, Article 2176, 15 pages.

[16] Miriam Horovicz and Roni Goldshmidt. 2024. TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation. In *Proceedings of the 1st Workshop on NLP for Science (NLP4Science)*, Lotem Peled-Cohen, Nitay Calderon, Shir Lissak, and Roi Reichart (Eds.). Association for Computational Linguistics, Miami, FL, USA, 1–8. https://doi.org/10.18653/v1/2024.nlp4science-1.1
[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv 2021. arXiv preprint arXiv:2106.09685 10 (2021).

[18] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. 2017. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. In *Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70)*, Doina Precup and Yee Whye Teh (Eds.). PMLR, 1645–1654. https://proceedings.mlr.press/v70/jaques17a.html

[19] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. 2019. Efficient task-specific data valuation for nearest neighbor algorithms. Proc. VLDB Endow. 12, 11 (July 2019), 1610–1623. https://doi.org/10.14778/3342263.3342637

[20] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. 2019. Towards Efficient Data Valuation Based on the Shapley Value. In *Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 89)*, Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 1167–1176. https://proceedings.mlr.press/v89/jia19a.html

[21] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. CoRR abs/2001.08361 (2020). https://arxiv.org/pdf/2001.08361.pdf

[22] Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In *Proceedings of the 34th International Conference on Machine Learning - Volume 70* (Sydney, NSW, Australia) (ICML'17). JMLR.org, 1885–1894.

[23] Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In *Advances in Neural Information Processing Systems*, Vol. 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf

[24] Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. *Advances in neural information processing systems* 30 (2017).

[25] Sasan Maleki, Long Tran-Thanh, Greg Hines, Talal Rahwan, and Alex Rogers. 2013. Bounding the Estimation Error of Sampling-based Shapley Value Approximation With/Without Stratifying. CoRR abs/1306.4265 (2013). arXiv:1306.4265 http://arxiv.org/abs/1306.4265

[26] Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson. 2025. Jackpot! alignment as a maximal lottery. arXiv preprint arXiv:2501.19266 (2025).

[27] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher Manning. 2024. An Emulator for Fine-tuning Large Language Models using Small Language Models. In *International Conference on Representation Learning*, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 13229–13244. https://proceedings.iclr.cc/paper_files/paper/2024/file/389e161125965c0f0ba50420fee45774-Paper-Conference.pdf

[28] Hyeonseok Moon, Jaehyung Seo, Seonmin Koo, Jinsung Kim, Young-kyoung Ham, Jiwon Moon, and Heuiseok Lim. 2025. LimaCost: Data Valuation for Instruction Tuning of Large Language Models. In *Findings of the Association for Computational Linguistics: EMNLP 2025*, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 12841–12854. https://doi.org/10.18653/v1/2025.findings-emnlp.688

[29] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. 2024. Nash learning from human feedback. In *Forty-first International Conference on Machine Learning*.

[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 27730–27744. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf

[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In *Advances in Neural Information Processing Systems*, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 53728–53741. https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf

[32] Stephanie Schoch, Ritwick Mishra, and Yangfeng Ji. 2023. Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)*, Vishakh Padmakumar, Gisela Vallejo, and Yao Fu (Eds.). Association for Computational Linguistics, Toronto, Canada, 266–275. https://doi.org/10.18653/v1/2023.acl-srw.37

[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347

[34] Lloyd S Shapley. 1953. A Value for n-Person Games. In *Contributions to the Theory of Games II*, Harold W. Kuhn and Albert W. Tucker (Eds.). Princeton University Press, Princeton, 307–317.

[35] Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. 2020. Collaborative machine learning with incentive-aware model rewards. In *Proceedings of the 37th International Conference on Machine Learning (ICML'20)*. JMLR.org, Article 828, 10 pages.

[36] Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. 2024. Distributional preference learning: Understanding and accounting for hidden context in rhlf. ICLR (2024).

[37] Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. 2024. A minimaximalist approach to reinforcement learning from human feedback. In *Proceedings of the 41st International Conference on Machine Learning* (Vienna, Austria) (ICML'24). JMLR.org, Article 1929, 33 pages.
[38] Mélissa Tamine, Benjamin Heymann, Patrick Loiseau, and Maxime Vono. 2025. On the Impact of the Utility in Semivalue-based Data Valuation. arXiv:2502.06574 [cs.AI] https://arxiv.org/abs/2502.06574

[39] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl.

[40] Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. 2024. Helpful or harmful data? fine-tuning-free shapley attribution for explaining language model predictions. In *Proceedings of the 41st International Conference on Machine Learning* (Vienna, Austria) (ICML'24). JMLR.org, Article 2089, 32 pages.

[41] Jiachen (Tianhao) Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. 2025. Data Shapley in One Training Run. In *International Conference on Representation Learning*, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 12358-12395. https://proceedings.iclr.cc/paper_files/paper/2025/file/20fdaf67581e6d7157376d1ed584040a-Paper-Conference.pdf

[42] Yingtai Xiao, Yuqing Zhu, Sirat Samyoun, Wanrong Zhang, Jiachen T. Wang, and Jian Du. 2025. TokenShapley: Token Level Context Attribution with Shapley Value. In *Findings of the Association for Computational Linguistics: ACL 2025*, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 3882-3894. https://doi.org/10.18653/v1/2025.findings-acl.200

[43] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. *International Conference on Machine Learning* (2024).

[44] Zikun Ye and Hema Yoganarasimhan. 2025. Fair Document Valuation in LLM Summaries via Shapley Values. arXiv:2505.23842 [cs.CL] https://arxiv.org/abs/2505.23842

[45] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-Tuning Language Models from Human Preferences. arXiv preprint arXiv:1909.08593 (2019). https://arxiv.org/abs/1909.08593