On the Impact of the Utility in Semivalue-based Data Valuation

## Abstract

Abstract Semivalue–based data valuation uses cooperative-game theory intuitions to assign each data point a value reflecting its contribution to a downstream task. Still, those values depend on the practitioner’s choice of utility, raising the question: How robust is semivalue-based data valuation to changes in the utility? This issue is critical when the utility is set as a trade-off between several criteria and when practitioners must select among multiple equally valid utilities. We address it by introducing the notion of a dataset’s spatial signature : given a semivalue, we embed each data point into a lower-dimensional space where any utility becomes a linear functional, making the data valuation framework amenable to a simpler geometric picture. Building on this, we propose a practical methodology centered on an explicit robustness metric that informs practitioners whether and by how much their data valuation results will shift as the utility changes. We validate this approach across diverse datasets and semivalues, demonstrating strong agreement with rank-correlation analyses and offering analytical insight into how choosing a semivalue can amplify or diminish robustness.

## 1 Introduction

Supervised machine learning (ML) relies on data, but real-world datasets often suffer from noise and biases as they are collected from multiple sources and are subject to measurement and annotation errors [20]. Such variability can impact learning outcomes, highlighting the need for systematic methods to evaluate data quality. In response, data valuation has emerged as a growing research field that aims to quantify individual data points’ contribution to a downstream ML task, helping to identify informative samples and mitigate the impact of low-quality data. A popular way to tackle the data valuation problem is to adopt a cooperative game-theoretic viewpoint, where each data point is modeled as a player in a coalitional game, and the usefulness of any data subset is measured by a utility function. This approach leverages game theory solution concepts called semivalues [6], which input data and utility to assign an importance score to each data point, thereby inducing a ranking of points in the order of their contribution to the ML task [9, 16, 27, 11, 10].

#### Motivation.

When computing semivalues, the utility is typically selected by the practitioner to reflect the downstream task. In some contexts, this choice is obvious. For example, when fine-tuning a large language model (LLM), one might balance two competing objectives: helpfulness (how well the model follows user instructions) and harmlessness (its tendency to refuse or safely complete unsafe requests) [1, 2]. If the practitioner then asks, “Which training examples most contributed to my desired helpfulness–harmlessness balance?”, the only sensible utility for semivalue-based data valuation is this composite trade-off itself. By contrast, in more open-ended tasks, the utility can be genuinely ambiguous. Imagine training a dog vs. cat image classifier and asking, “Which data points contributed most to overall performance?” Then, accuracy, precision, recall, F1, AUROC, balanced accuracy, and many others are all defensible choices. However, none is uniquely dictated by the task.

These two example settings respectively motivate two general scenarios: (1) the *utility trade-off* scenario, where the utility is a convex combination of fixed criteria, with a tunable weight $\nu$, and (2) the *multiple-valid-utility* scenario, where the utility must be chosen among several equally defensible metrics, none of which being uniquely dictated by the task. In both cases, we argue that data valuation practitioners are well advised to ask themselves:

*How robust are my data valuation results to the utility choice?*

In what follows, we explain why.

*Utility trade-off* scenario: Anticipating costly re-training. In scenarios where the utility itself is a trade-off, e.g., when fine-tuning an LLM by combining helpfulness and harmlessness into a single objective parameterized by $\nu$, practitioners often rely on data values to identify the $k$ most valuable training examples and train next models on this smaller subset to reduce computational cost as is common practice when using data valuation [9, 12].
However, if the top-$k$ set shifts dramatically with changes in $\nu$, as priorities between harmlessness and helpfulness evolve, practitioners risk repeated, costly re-training. Quantifying robustness to utility choice makes this risk explicit, alerting practitioners up front to whether data valuation is a safe, one-time investment or if they must plan for ongoing computational overhead as their utility trade-off evolves.

*Multiple-valid-utility* scenario: Detecting when data valuation fails as a heuristic. In many real-world tasks, like the earlier dog vs. cat classifier example, practitioners must select a utility from several valid options, none uniquely dictated by the problem. Now, one would "morally" expect their induced orderings of points to be consistent. After all, each utility is a valid measure for the same task. It may be hard to think that a data point deemed highly important under accuracy would suddenly vanish from the top tier under F1-score, or vice versa. In practice, however, we observe precisely such discrepancies, depending on both the training dataset and the semivalue. We compute data values under both accuracy and F1-score on several public datasets, using three popular semivalues: Shapley [9], $(4,1)$-Beta Shapley [16], and Banzhaf [27] (see Appendix [A](https://arxiv.org/html/2502.06574v2#A1) for experimental details). Table [1](https://arxiv.org/html/2502.06574v2#S1.T1) reports the Kendall rank correlation between the two score sets for each combination of dataset and semivalue. Low correlations reveal cases where rankings change substantially depending on whether accuracy or F1-score is used as the utility (^1^11We extend these experiments to additional classification utilities and rank correlation metrics for completeness, and observe variability in rank correlations as well, depending on both the dataset and the semivalue. Corresponding results are provided in Appendix [A](https://arxiv.org/html/2502.06574v2#A1).). And because no utility is inherently better, a practitioner has no principled way to choose between the data values ranking produced under accuracy versus F1-score (or any other valid utility). In such settings, arbitrary utility choices can drive the ordering of data points in entirely different directions: the context (dataset + semivalue) is therefore not "data-valuationable", and semivalue-based data valuation *fails* as a reliable heuristic. By contrast, if rankings remain consistent across all valid utilities for the task, data values truly capture the underlying importance ordering of points. Therefore, knowing how robust the scores ranking is to the utility choice enables a practitioner to tell if semivalue-based data valuation can be trusted as a meaningful heuristic in that context, or if it is too sensitive to utility to provide reliable guidance.

**Table 1: Mean Kendall rank correlation (± standard error) between data values computed with accuracy versus F1-score. For each semivalue and dataset, we approximate data values $5$ times via Monte Carlo sampling. Standard errors reflect the variability across these $5$ trials.**
| Dataset | Semivalue |  |  |
| --- | --- | --- | --- |
|  | Shapley | $(4,1)$-Beta Shapley | Banzhaf |
| Breast | $0.95$ $(0.003)$ | $0.95$ $(0.003)$ | $0.97$ $(0.008)$ |
| Titanic | $-0.19$ $(0.007)$ | $-0.17$ $(0.01)$ | $0.94$ $(0.003)$ |
| Credit | $-0.47$ $(0.01)$ | $-0.44$ $(0.02)$ | $0.87$ $(0.01)$ |
| Heart | $0.64$ $(0.006)$ | $0.68$ $(0.004)$ | $0.96$ $(0.003)$ |
| Wind | $0.81$ $(0.008)$ | $0.82$ $(0.008)$ | $0.99$ $(0.002)$ |
| Cpu | $0.59$ $(0.02)$ | $0.62$ $(0.02)$ | $0.86$ $(0.007)$ |
| 2dplanes | $0.38$ $(0.01)$ | $0.44$ $(0.01)$ | $0.75$ $(0.03)$ |
| Pol | $0.67$ $(0.02)$ | $0.77$ $(0.01)$ | $0.40$ $(0.04)$ |

In this paper, we propose a methodology that enables data valuation practitioners to assess how robust their semivalue-based data valuation results are to the utility choice, in both scenarios. We summarize our main contributions as follows.

- 1.
Unified geometric modeling of the two scenarios. We observe that the same geometric representation can capture both scenarios. In this representation, we can, given a semivalue, embed each training data point into a lower-dimensional space (we call the set of embedded points the dataset *spatial signature*) where any utility becomes a linear functional, making the data valuation framework amenable to a simpler geometric interpretation.
- 2.
A robustness metric derived from the geometric representation. Building on the notion of *spatial signature*, we introduce a metric that practitioners can compute to quantify how robust the data values’ orderings are to the utility choice, providing a practical methodology for assessing the robustness of data valuation results.
- 3.
Empirical evaluation of robustness and insights. We compute the robustness metric across multiple public datasets and semivalues and find results consistent with our rank-correlation experiments: contexts with low rank correlation exhibit a low robustness score, and vice versa. Moreover, we observe that Banzhaf consistently achieves higher robustness scores than other semivalues, a phenomenon for which we provide analytical insights.

#### Related works.

Our focus on the robustness of semivalue-based data valuation to the utility choice departs from most prior work, which has concentrated on *defining* and *efficiently computing* data valuation scores. The Shapley value [23, 9], in particular, has been widely studied as a data valuation method because it uniquely satisfies four key axioms: linearity, dummy player, symmetry, and efficiency. Alternative approaches have emerged by relaxing some of these axioms. Relaxing efficiency gives rise to the semivalue family [6], which encompasses Leave-One-Out [13], Beta Shapley [16], and Data Banzhaf [27], while relaxing linearity leads to the Least Core [29]. Extensions such as Distributional Shapley [8, 15] further adapt the framework to handle underlying data distributions instead of a fixed dataset.
On the algorithmic front, exact semivalue computation is often intractable as each semivalue requires training models over all possible data subsets, which grows exponentially with the size of the dataset. Consequently, a rich literature on approximation methods has emerged to make data valuation practical at scale [19, 18, 11, 9, 10, 7]. By contrast, when and why data valuation scores remain consistent across different utilities has received far less attention. Prior work has explored related robustness questions in special cases: [27] examines how semivalue–based valuations fluctuate when the utility function is corrupted by inherent randomness in the learning algorithm, and [28] studies how different choices of utility affect the reliability of Data Shapley for data-subset selection. Our proposed methodology broadens this scope by quantifying when data valuation is robust to shifts in the utility function.

#### Notations.

We set $\mathbb{N}^{*}=\mathbb{N}\setminus\{0\}$. For $n\in\mathbb{N}^{*}$, we denote $[n]:=\{1,..,n\}$. For a dataset $\mathcal{D}$, we denote as $2^{\mathcal{D}}$ its powerset, i.e., the set of all possible subsets of $\mathcal{D}$, including the empty set $\emptyset$ and $\mathcal{D}$ itself. For $d\in\mathbb{N}^{*}$, we denote $\mathcal{X}\subseteq\mathbb{R}^{d}$ and $\mathcal{Y}\subseteq\mathbb{R}$ as an input space and an output space, respectively.

## 2 Background

The data valuation problem involves a dataset of interest $\mathcal{D}=\{z_{i}=(x_{i},y_{i})\}_{i\in[n]}$, where for any $i\in[n]$ each $x_{i}\in\mathcal{X}$ is a feature vector and $y_{i}\in\mathcal{Y}$ is the corresponding label. Data valuation aims to assign a scalar score to each data point in $\mathcal{D}$, quantifying its contribution to a downstream ML task. These scores will be referred to as data values.

#### Utility functions.

Most data valuation methods rely on utility functions to compute data values. A utility is a set function $u:2^{\mathcal{D}}\to\mathbb{R}$ that maps any subset $S$ of the training set $\mathcal{D}$ to a score indicating its usefulness for performing the considered task. Formally, this can be expressed as $u(S)=\texttt{perf}(\mathcal{A}(S))$, where $\mathcal{A}$ is a learning algorithm that takes a subset $S$ as input and returns a trained model, and perf is a metric used to evaluate the model’s ability to perform the task on a hold-out test set. For convenience, we interchangeably refer to the utility $u$ and the performance metric perf as $u$ inherently depends on perf.

#### Semivalues.

The most popular data valuation methods assign a value score to each data point in $\mathcal{D}$ using solution concepts from cooperative game theory, known as semivalues [6]. The collection of methods that fall under this category is referred to as semivalue-based data valuation. They rely on the notion of marginal contribution.
Formally, for any $i,j\in[n]$, let $\mathcal{D}_{j}^{\backslash z_{i}}$ denote the set of all subsets of $\mathcal{D}$ of size $j-1$ that exclude $z_{i}$. Then, the marginal contribution of $z_{i}$ with respect to samples of size $j-1$ is defined as

$$ $\displaystyle\Delta_{j}(z_{i};u):=\frac{1}{\binom{n-1}{j-1}}\sum_{S\subseteq \mathcal{D}_{j}^{\backslash z_{i}}}u\left(S\cup\{z_{i}\}\right)-u(S)\ .$ $$

The marginal contribution $\Delta_{j}(z_{i};u)$ considers all
possible subsets $S\in\mathcal{D}_{j}^{\backslash z_{i}}$ with the same cardinality $j-1$
and measures the average changes of $u$ when datum of interest $z_{i}$ is removed from $S\cup\{z_{i}\}$.

Each semivalue-based method is characterized by a weight vector $\omega:=(\omega_{1},\ldots,\omega_{n})$ and assigns a score $\phi(z_{i};\omega,u)$ to each data point $z_{i}\in\mathcal{D}$ by computing a weighted average of its marginal contributions $\{\Delta_{j}(z_{i};u)\}_{j\in[n]}$. Specifically,

$$ $\displaystyle\phi(z_{i};\omega,u):=\sum_{j=1}^{n}\omega_{j}\Delta_{j}(z_{i};u)\ .$ (1) $$

Below, we define the weights for three commonly used semivalue-based methods. Their differences in weighting schemes have geometric implications discussed in Section [4](https://arxiv.org/html/2502.06574v2#S4).

###### Definition 2.1 .

Data Shapley [9] is derived from the Shapley value [23], a solution concept from cooperative game theory that fairly allocates the total gains generated by a coalition of players based on their contributions. In the context of data valuation, Data Shapley takes a simple average of all the contributions so that its corresponding weight vector $\omega_{\text{shap}}=(\omega_{\text{beta},j})_{j\in[n]}$ is such that for all $j\in[n]$, $\omega_{\text{shap},j}=\frac{1}{n}$.

###### Definition 2.2 .

$(\alpha,\beta)$-Beta Shapley [16] extends Data Shapley by introducing tunable parameters $(\alpha,\beta)\in\mathbb{R}^{2}$, which control the emphasis placed on marginal contributions from smaller or larger subsets. The corresponding weight vector $\omega_{\text{beta}}$ = $(\omega_{\text{beta},j})_{j\in[n]}$ is such that for all $j\in[n]$, $\omega_{\text{beta},j}=\binom{n-1}{j-1}\cdot\frac{\texttt{Beta}(j+\beta-1,n-j+
\alpha)}{\texttt{Beta}(\alpha,\beta)}$, where $\texttt{Beta}(\alpha,\beta)=\Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta)$ and $\Gamma$ is the Gamma function.

###### Definition 2.3 .

Data Banzhaf [27] is derived from the Banzhaf value [3], a cooperative game theory concept originally introduced to measure a player’s influence in weighted voting systems. Data Banzhaf weight’s vector $\omega_{\text{banzhaf}}=(\omega_{\text{banzhaf},j})_{j\in[n]}$ is such that for all $j\in[n]$, $\omega_{\text{banzhaf},j}=\binom{n-1}{j-1}\cdot\frac{1}{2^{n-1}}$.

These semivalue-based methods satisfy fundamental axioms [6] that ensure desirable properties in data valuation. In particular, any semivalue $\phi(.;\omega,.)$ satisfy the *linearity* axiom which states that for any $\alpha_{1},\alpha_{2}\in\mathbb{R}$, and any utility $u$, $v$, $\phi(z_{i};\omega,\alpha_{1}u+\alpha_{2}v)=\alpha_{1}\phi(z_{i};\omega,u)+
\alpha_{2}\phi(z_{i};\omega,v)$.

###### Definition 2.1 .

###### Definition 2.2 .

###### Definition 2.3 .

## 3 A methodology to assess data valuation robustness to the utility choice

We now turn to the two scenarios introduced in Section [1](https://arxiv.org/html/2502.06574v2#S1), namely the *utility trade-off* scenario and the *multiple-valid-utility* scenario, and show how both admit a common geometric formalization.
In what follows, we let $\mathcal{D}=\{z_{i}\}_{i\in[n]}$ be the dataset that the practitioner seeks to score and rank by order of importance, and we let
$\omega$ be the chosen semivalue weight vector, so that each datum score is given by $\phi(z_{i};\omega,u)$ as defined in ([1](https://arxiv.org/html/2502.06574v2#S2.E1)).
We start by giving a formal definition of each scenario.

#### Utility trade-off scenario.

In this scenario, the practitioner defines utility as a convex combination of multiple fixed criteria.
In the simplest case where one considers only two fixed criteria $u^{A}$ and $u^{B}$ (e.g. helpfulness vs. harmlessness when fine-tuning an LLM), the utility is

$$ $\displaystyle u_{\nu}=\nu u^{A}+(1-\nu)u^{B},\qquad\nu\in[0,1],$ $$

where the scalar $\nu$ is explicitly chosen by the practitioner (based on operational priorities) to set the desired trade-off between $u^{A}$ and $u^{B}$.
Note that the previous utility choice naturally extends to $K$ fixed criteria $u^{1},\dots,u^{K}$ by taking $u_{\nu}=\sum_{k=1}^{K}\nu_{k}u^{k}$.

By semivalue linearity, each data point’s score under $u_{\nu}$ is

$$ $\displaystyle\phi(z_{i};\omega,u_{\nu})=\nu\phi(z_{i};\omega,u^{A})+(1-\nu) \phi(z_{i};\omega,u^{B})\ .$ $$

#### Multiple-valid-utility scenario.

In this scenario, there is no single *correct* utility: practitioners must choose among several equally defensible performance metrics. In the common case of binary classification, one might measure model quality with accuracy, F1-score, or negative log-loss: each of which is valid but potentially yields different data valuation results; see Table [1](https://arxiv.org/html/2502.06574v2#S1.T1).
Almost all of these utilities admit a *linear-fractional* form [14], that is, they can be written as a ratio of affine functions of the model’s true-positive rate and positive-prediction rate. Formally, given a training dataset $S\in(\mathcal{X}\times\{0,1\})^{n}$, a test dataset $\mathcal{D}_{\text{test }}=\left\{\left(x_{j},y_{j}\right)\right\}_{j\in[m]}
\in(\mathcal{X}\times\{0,1\})^{m}$ and a learning algorithm $\mathcal{A}$, we denote $g_{S}=\mathcal{A}(S)$ a classifier trained on $S$ which maps input features to predicted labels, i.e., $g_{S}:\mathcal{X}\rightarrow\{0,1\}$. A *linear-fractional* utility $u$ evaluates the predictive performance of $g_{S}$ on $\mathcal{D}_{\text{test }}$ as

$$ $\displaystyle u(S)=\frac{c_{0}+c_{1}\lambda\left(S,\mathcal{D}_{\text{test }} \right)+c_{2}\gamma\left(S,\mathcal{D}_{\text{test }}\right)}{d_{0}+d_{1} \lambda\left(S,\mathcal{D}_{\text{test }}\right)+d_{2}\gamma\left(S,\mathcal{D }_{\text{test }}\right)}\ ,$ (2) $$

where $\{c_{0},c_{1},c_{2},d_{0},d_{1},d_{2}\}$ specify the particular utility; see Table [2](https://arxiv.org/html/2502.06574v2#A3.T2) for concrete examples.
The quantity $\lambda\left(S,\mathcal{D}_{\text{test}}\right)=\widehat{\mathbb{P}}_{\mathcal
{D}_{\text{test }}}\left(g_{S}(x)=1,y=1\right)=$ $\frac{1}{m}\sum_{j=1}^{m}\mathbf{1}\left[g_{S}\left(x_{j}\right)=1,y_{j}=1\right]$ is the empirical probability of true positives and $\gamma\left(S,\mathcal{D}_{\text{test }}\right)=\widehat{\mathbb{P}}_{\mathcal
{D}_{\text{test }}}\left(g_{S}(x)=1\right)=$ $\frac{1}{m}\sum_{j=1}^{m}\mathbf{1}\left[g_{S}\left(x_{j}\right)=1\right]$ is the empirical probability of positive predictions. When the test dataset $\mathcal{D}_{\text{test}}$ is clear from context, we use the shorthand $\lambda(S)$ to denote $\lambda\left(S,\mathcal{D}_{\text{test }}\right)$ and $\gamma(S)$ to denote $\gamma\left(S,\mathcal{D}_{\text{test }}\right)$.

Any linear-fractional utility $u$ of the form ([2](https://arxiv.org/html/2502.06574v2#S3.E2)) with $d_{0}\neq 0$, admits a first-order (Taylor–Young) expansion around $(\lambda,\gamma)=(0,0)$ of the form

$$ $\displaystyle u(S)=\frac{c_{0}}{d_{0}}+\frac{c_{1}d_{0}-c_{0}d_{1}}{d_{0}^{2}} \lambda(S)+\frac{c_{2}\,d_{0}-c_{0}\,d_{2}}{d_{0}^{2}}\gamma(S)+o\bigl{(}\max \{|\lambda(S)|,|\gamma(S)|\}\bigr{)}.$ $$

In other words, to first order, $u$ is simply an affine function of $\lambda$ and $\gamma$. Full details of this derivation are given in Appendix [B](https://arxiv.org/html/2502.06574v2#A2).

Empirically, we validate this affine surrogate on two linear fractional utilities for which the denominator is not constant, namely the F1 score and the Jaccard similarity coefficient, by comparing the semivalue scores ranking obtained from the exact ratio against those from the first-order approximation. Across multiple datasets and semivalue methods, the proportion of discordant pairs between the two rankings does not exceed $2.3\%$ with standard error precision (the results are displayed in Table [3](https://arxiv.org/html/2502.06574v2#A2.T3)). This low discordance rate suggests that the higher-order terms exert only a minor effect on the induced ordering. Further details on this validation are provided in Appendix [B](https://arxiv.org/html/2502.06574v2#A2).

Consequently, it is reasonable to substitute any linear-fractional utility $u$ with its affine surrogate, and by linearity of the semivalue, we get for each data point $z_{i}$:

$$ $\displaystyle\phi(z_{i};\omega,u)=\frac{c_{1}d_{0}-c_{0}d_{1}}{d_{0}^{2}}\phi( z_{i};\omega,\lambda)+\frac{c_{2}\,d_{0}-c_{0}\,d_{2}}{d_{0}^{2}}\phi(z_{i}; \omega,\gamma).$ $$

### 3.1 A unified geometric modeling of the two scenarios

Both scenarios can be unified by observing that the practitioner’s utility lives in a two-dimensional family spanned by two fixed base utilities $u_{1}$ and $u_{2}$. Concretely, we consider

$$ $\displaystyle u_{\alpha}(S)\;=\;\alpha_{1}\,u_{1}(S)\;+\;\alpha_{2}\,u_{2}(S), \quad\alpha=(\alpha_{1},\alpha_{2})\in\mathbb{R}^{2},$ $$

so that varying the utility means moving $\alpha$ in this two-dimensional parameter space.
In the *utility trade-off* scenario restricted to two fixed criteria, $(u_{1},u_{2})=(u^{A},u^{B})$, and $(\alpha_{1},\alpha_{2})=(\nu,1-\nu)$ ranges over $[0,1]^{2}$. In the *multiple-valid-utility* scenario for binary classification, $(u_{1},u_{2})=(\lambda,\gamma)$, and $(\alpha_{1},\alpha_{2})\in\mathbb{R}^{2}$.
In either case, the objective is the same: to quantify robustness, i.e., how stable the ranking of semivalue scores $\{\phi\bigl{(}z_{i};\omega,\,u_{\alpha}\bigr{)}\}$ is as we change $\alpha$.

With this unified view in hand, we have the following proposition, which can be extended to the general case $u_{\alpha}=\sum_{k=1}^{K}\alpha_{k}u_{k}$. A detailed extension is provided in Appendix [B](https://arxiv.org/html/2502.06574v2#A2).

###### Proposition 3.1 .

Let $\mathcal{D}$ be any dataset of size $n$ and let $\omega\in\mathbb{R}^{n}$ be a semivalue weight vector. Then there exists a map $\psi_{\omega,\mathcal{D}}:\mathcal{D}\longrightarrow\mathbb{R}^{2}$
such that for every utility $u_{\alpha}=\alpha_{1}u_{1}+\alpha_{2}u_{2}$, $\phi\bigl{(}z;\omega,u_{\alpha}\bigr{)}=\bigl{\langle}\psi_{\omega,\mathcal{D}
}(z),\alpha\bigr{\rangle}$, for any $z\in\mathcal{D}$.
We call
$\mathcal{S}_{\omega,\mathcal{D}}=\{\psi_{\omega,\mathcal{D}}(z)\mid z\in
\mathcal{D}\}$ the *spatial signature* of $\mathcal{D}$ under semivalue $\omega$.

Consequently, ranking the data points in $\mathcal{D}$ by $u_{\alpha}$ is equivalent to sorting their projections onto the vector $\alpha$:

$$ $\phi(z_{i};\omega,u_{\alpha})>\phi(z_{j};\omega,u_{\alpha})\quad \Longleftrightarrow\quad\langle\psi_{\omega,\mathcal{D}}(z_{i}),\alpha\rangle> \langle\psi_{\omega,\mathcal{D}}(z_{j}),\alpha\rangle.$ $$

Moreover, since scaling $\alpha$ by any positive constant does not change the sign of $\langle\psi_{\omega,\mathcal{D}}(z_{i}),\alpha\rangle-\langle\psi_{\omega,
\mathcal{D}}(z_{j}),\alpha\rangle$, any two utilities $u_{\alpha}$ and $u_{\alpha^{\prime}}$ whose coefficient vectors point in the same direction induce identical rankings. Thus, each utility in the parametric family can be uniquely identified by its normalized vector $\bar{\alpha}=\frac{\alpha}{\|\alpha\|}\in\mathcal{S}^{1}$, with $\bar{\alpha}$ ranging over the unit circle $\mathcal{S}^{1}$. Consequently, ranking stability to the utility choice reduces to analyzing how the projections order of $\{\langle\psi_{\omega,\mathcal{D}}(z),\alpha\rangle\mid z\in\mathcal{D}\}
\subset\mathbb{R}^{2}$ changes as we rotate the unit-vector $\bar{\alpha}$ around $\mathcal{S}^{1}$.
Figure [1](https://arxiv.org/html/2502.06574v2#S3.F1) illustrate the geometric mapping at hand.

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/shapley.png

Figure [1](https://arxiv.org/html/2502.06574v2#S3.F1) shows that, under Banzhaf, the points lie almost exactly on a single line through the origin, much more so than under Shapley or $(4,1)$-Beta Shapley. This near-collinearity persists across all the datasets used for experiments (see Appendix [D](https://arxiv.org/html/2502.06574v2#A4)). In Section [4](https://arxiv.org/html/2502.06574v2#S4), we formalize how this geometric property directly leads to Banzhaf’s higher robustness.

###### Proposition 3.1 .

### 3.2 A robustness metric derived from the geometric representation

Building on the geometric mapping of semivalue-based data valuation proposed in Section [3.1](https://arxiv.org/html/2502.06574v2#S3.SS1), a natural way to quantify how robust a semivalue scores ranking is to changes in the utility is to ask *how far on the unit circle one must rotate from a given utility direction before the induced ordering undergoes a significant change?*

Formally, let $\bar{\alpha}_{0}$ be the starting utility direction, whose semivalue scores induce a reference ranking of the data points. We say that two points $z_{i}$ and $z_{j}$ experience a *pairwise swap* when their order under a new direction $\bar{\alpha}$ is opposite to their order under $\bar{\alpha}_{0}$. We then aim to define robustness based on the smallest geodesic distance on $\mathcal{S}^{1}$ one must travel from $\bar{\alpha}_{0}$ before $p$ such pairwise swaps have occurred.

To make this concrete, we now express the required geodesic distance in closed form by characterizing the critical angles on $\mathcal{S}^{1}$ at which pairwise swaps occur. For each unordered pair $(i,j)$, let $v_{ij}=\psi_{\omega,\mathcal{D}}(z_{i})-\psi_{\omega,\mathcal{D}}(z_{j})$ and observe that the condition $\langle\alpha,v_{ij}\rangle=0$ defines two antipodal “cut” points on the unit circle: $H_{ij}=\bigl{\{}\alpha\in\mathcal{S}^{1}:\langle\alpha,v_{ij}\rangle=0\bigr{\}}.$ Across all $\binom{n}{2}=N$ pairs, these give $2N$ cuts, whose polar angles we list in ascending order as

$$ $\displaystyle 0\leq\theta_{1}\leq\theta_{2}\leq\dots\leq\theta_{2N}<2\pi,$ $$

and then wrap around by setting $\theta_{2N+1}=\theta_{1}+2\pi$. The open arcs between successive cuts are $A_{k}=(\theta_{k},\theta_{k+1})$ of length $\lambda_{k}=\theta_{k+1}-\theta_{k},\quad k=1,\dots,2N$ so that $\sum_{k=1}^{2N}\lambda_{k}=2\pi$. These open arcs partition $\mathcal{S}^{1}$ into ranking regions, meaning that the induced semivalue ordering is identical for every utility direction $\bar{\alpha}\in A_{k}$. Figure [2](https://arxiv.org/html/2502.06574v2#S3.F2) illustrates two example spatial signatures and their induced ranking regions. We view these arcs cyclically by taking indices modulo $2N$. Now let our reference direction $\bar{\alpha}_{0}$ have polar angle $\varphi_{0}\in(\theta_{k},\theta_{k+1})$. To induce $p$ swaps, one must cross $p$ distinct arcs: counterclockwise this is $S_{k}^{+}(p)=\sum_{i=1}^{p}\lambda_{(k+i)\bmod 2N}$ while clockwise it is $S_{k}^{-}(p)=\sum_{i=1}^{p}\lambda_{(k-i)\bmod 2N}$. Writing $t=\varphi_{0}-\theta_{k}\in(0,\lambda_{k})$, the minimal geodesic distance from $\bar{\alpha}_{0}$ to achieve $p$ swaps is(^2^22All cut-angles, arc-lengths, and resulting geodesic distance $\rho_{p}$ are entirely determined by the spatial signature $\mathcal{S}_{\omega,\mathcal{D}}$. For brevity, we omit the explicit dependence on it from our notations.)

$$ $\displaystyle\rho_{p}(\bar{\alpha}_{0})=\min\bigl{\{}S_{k}^{+}(p)-t,\quad S_{k }^{-}(p)+t\bigr{\}}.$ $$

Figure: Figure 2: Ranking regions induced by utilities on the unit circle $\mathcal{S}^{1}$ for two example spatial signatures. Each colored arc on the unit circle corresponds to one of the open arcs $A_{k}$. Within any single arc, the projection order (and hence the data-point ranking) remains unchanged.
Refer to caption: extracted/6471790/figures/ranking_regions.png

We now define our robustness metric based on $\rho_{p}$.

###### Definition 3.2 (Robustness metric $R_{p}$ ) .

Let $\mathcal{S}_{\omega,\mathcal{D}}=\{\psi_{\omega,\mathcal{D}}(z_{i})\}_{i\in[n]}$ be the spatial signature for dataset $\mathcal{D}$ under semivalue weights $\omega$. For $\bar{\alpha}\in\mathcal{S}^{1}$, let $\rho_{p}(\bar{\alpha})$ denote the minimal geodesic distance on $\mathcal{S}^{1}$ one must travel from $\bar{\alpha}$ to incur $p<\binom{n}{2}$ pairwise swaps in the induced ranking. Define the average $p$–swaps distance
$\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]=\frac{1}{2\pi}\int_{0}^{2\pi
}\rho_{p}(t)dt$. Then the *robustness metric* $R_{p}\in[0,1]$ is

$$ $R_{p}(S_{\omega,\mathcal{D}})=\frac{\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{ \alpha})]}{\displaystyle\max_{S_{\omega,\mathcal{D}}}\mathbb{E}_{\bar{\alpha}} [\rho_{p}(\bar{\alpha)}]}=\frac{\mathbb{E}_{\bar{\alpha}}[\rho_{p}({\bar{ \alpha}})]}{\pi/4},$ $$

where the denominator $\pi/4$ is the maximum possible value of $\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]$ which occurs precisely when all embedded points $\psi_{\omega,\mathcal{D}}(z_{i})$ are collinear (^3^33Proof of this proposition is given in Appendix [B](https://arxiv.org/html/2502.06574v2#A2).).

Concretely, given a spatial signature, the $p$-robustness metric $R_{p}$ of this signature is the normalized average minimal angular distance one must rotate on the unit circle to force exactly $p$ pairs of points to swap in order in the induced ranking. $R_{p}$ close to 1 means that one can rotate $\bar{\alpha}$ significantly without flipping more than $p$ pairs, so the ranking is stable. $R_{p}$ close to 0 0 means that even a tiny rotation will likely flip $p$ pairs.

In Appendix [B](https://arxiv.org/html/2502.06574v2#A2), we derive a closed-form for $\mathbb{E}[\rho_{p}]$ and show that it can be computed exactly in $\mathcal{O}(n^{2}\log n)$ time with only standard arithmetic and sorting operations. In contrast, semivalue approximation methods based on Monte Carlo sampling require $\mathcal{O}(n^{2}\log n)$ model trainings to estimate the data values [11]. Therefore, in practice, once the semivalue scores are in hand, the additional cost of computing $R_{p}$ is negligible compared to the heavy model-training overhead, making this robustness metric an affordable add-on to any data valuation pipeline.

###### Definition 3.2 (Robustness metric R p subscript 𝑅 𝑝 R_{p} italic_R start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) .

### 3.3 Link between robustness metric and rank correlation

If there are no tied ranks, the Kendall rank correlation between two orderings of $n$ points is defined as $\tau=1-\frac{2D}{N}$, where $D$ is the number of discordant pairs and $N=\binom{n}{2}$ is the total number of pairs. Since crossing one ranking region swaps exactly one pair, each such swap increases $D$ by one and thus decreases $\tau$ by $2/N$. Consequently, $p$ swaps lower the correlation from 1 to $1-\frac{2p}{N}$. Therefore, $R_{p}$ captures how far in expectation one must move from a utility direction before the Kendall rank correlation degrades by at least $2p/N$. It degrades more than $\frac{2p}{N}$ if ties occur (see Appendix [B](https://arxiv.org/html/2502.06574v2#A2) for details).

## 4 Empirical evaluation of robustness and discussion

In this section, we empirically validate the $p$-robustness metric $R_{p}$ in the *multiple-valid-utility* scenario.

We evaluate $R_{p}$ for three semivalues, Shapley, $(4,1)$-Beta Shapley, and Banzhaf, on several public binary classification datasets. The results in Figure [3](https://arxiv.org/html/2502.06574v2#S4.F3) (detailed in Table [4](https://arxiv.org/html/2502.06574v2#A1.T4)) closely track Section [1](https://arxiv.org/html/2502.06574v2#S1)’s correlation experiments reported in Table [1](https://arxiv.org/html/2502.06574v2#S1.T1): datasets and semivalues that exhibit low rank correlations between different utilities also show low $R_{p}$, and vice versa.

Figure: Figure 3: Mean $p$-robustness $R_{p}$ (error bars = standard errors over $5$ Monte Carlo approximations) plotted against $p\in\{500,1000,1500\}$ for each dataset and semivalue. Each plot corresponds to one dataset, with Shapley (blue), $(4,1)$-Beta Shapley (pink), and Banzhaf (green) curves. Higher $R_{p}$ indicates greater ranking stability under utility shifts.
Refer to caption: extracted/6471790/figures/r_p.png

We also observe that across practically every dataset and choice of $p$, using the Banzhaf weights achieves the highest $R_{p}$. This makes sense geometrically: Figure [1](https://arxiv.org/html/2502.06574v2#S3.F1) and the analogous plots for the other datasets in Appendix [D](https://arxiv.org/html/2502.06574v2#A4) show that the Banzhaf weighting scheme tends to *collinearize* the spatial signature, i.e., push the points closer to a common line through the origin. And since the maximum possible average swap-distance
$\displaystyle\max_{\mathcal{S}_{\omega,\mathcal{D}}}\mathbb{E}_{\bar{\alpha}}[
\rho_{p}(\bar{\alpha})]=\pi/4$
occurs only when all embedded points are exactly collinear; this near-collinearity explains why Banzhaf yields the greatest robustness to utility shifts. This observation aligns with prior empirical findings [27, 17], which reported that Banzhaf scores tend to vary less than other semivalues under changing conditions.

To gain analytic insight into why the Banzhaf value produces more collinear embeddings, we turn to an exact decomposition of the global correlation (i.e. collinearity) between two semivalue score vectors $\phi(\lambda)=(\phi(z_{1},\omega,\lambda)\dots\phi(z_{n},\omega,\lambda))$ and $\phi(\gamma)=(\phi(z_{1},\omega,\gamma)\dots\phi(z_{n},\omega,\gamma))$. Under the mild assumption that marginal contributions for $\lambda$ and $\gamma$ at different coalition sizes are uncorrelated, we show that(^4^44Details are provided in Appendix [B](https://arxiv.org/html/2502.06574v2#A2).)

$$ $\displaystyle\operatorname{Corr}\bigl{(}\varphi(\lambda),\varphi(\gamma)\bigr{ )}=\sum_{j=1}^{n}\omega_{j}^{2}\,r_{j}\quad\rm{and}\quad r_{j}\propto \operatorname{Corr}\bigl{(}\Delta_{j}(\lambda),\Delta_{j}(\gamma)\bigr{)}\sqrt {\operatorname{Var}(\Delta_{j}(\lambda))\,\operatorname{Var}(\Delta_{j}(\gamma ))}$ $$

with $\Delta_{j}(u)=(\Delta_{j}(z_{1},u),\dots,\Delta_{j}(z_{n},u))$ for any given utility $u$ and with $\operatorname{Var}$ and $\operatorname{Corr}$ being empirical statistics taken over the $n$ data points (precise definitions are given in Appendix [C](https://arxiv.org/html/2502.06574v2#A3)). Thus, the global correlation is a weighted average of $r_{j}$ with weights $\omega_{j}^{2}$ inherited from the semivalue definition. In Figure [4](https://arxiv.org/html/2502.06574v2#S4.F4), we plot the (normalized) sequence $r_{j}$ against $j$, overlaying the semivalue weight profiles $\omega$ of Shapley, $(4,1)$-Beta Shapley and Banzhaf.

Figure: Figure 4: Mean $r_{j}$ (error bars = standard errors over $5$ semivalue approximations) for breast (blue) and titanic (red) vs. coalition size $j$, with semivalue weights $\omega$ overlaid.
Refer to caption: extracted/6471790/figures/r_j_vs_j.png

On breast, $r_{j}$ stays uniformly high, so all three semivalues achieve similar collinearity as reflected by their overlapping robustness curves in Figure [4](https://arxiv.org/html/2502.06574v2#S4.F4). On titanic, however, $r_{j}$ peaks at intermediate $j$ and falls off at the extremes; because Banzhaf puts more weight on those middle-$j$ contributions, it attains a higher weighted average and thus greater overall correlation (and thus, collinearity) exactly as seen in Figure [4](https://arxiv.org/html/2502.06574v2#S4.F4), where Banzhaf’s robustness curve sits well above those of Shapley and $(4,1)$-Beta Shapley.

## 5 Conclusion

In this work, we identify two scenarios where the robustness of semivalue-based data valuation under shifts in the utility is critical. We introduce a unified geometric modeling of these scenarios based on the notion of *spatial signature* that enables the definition of a parametric robustness measure $R_{p}$. Its computation gives practitioners a methodology to assess how stable their data values’ orderings remain under changes in the utility.

#### Limitations.

The closed-form expression for $R_{p}$ applies directly when utilities lie in a 2D space but does not extend to mixtures of $K>2$ utilities, where one must instead approximate $R_{p}$ numerically (see Appendix [B](https://arxiv.org/html/2502.06574v2#A2)). Furthermore, our analysis of the *multiple-valid-utility* scenario focuses on binary classification metrics in the linear-fractional family; other utility forms (e.g., negative log-loss), multiclass classification, and regression remain outside the current scope.

#### Future works.

Natural next steps involve (1) generalizing $R_{p}$ to mixtures of more than two utilities by developing efficient approximations with provable guarantees; and (2) extending the spatial signature approach to handle non-linear fractional utilities, as well as multiclass and regression tasks. Moreover, by revealing cases where semivalue-based data valuation fails to produce reliable scores, we aim to encourage future research to assess whether these methods genuinely solve the problem they promise to address.

## Acknowledgments and Disclosure of Funding

This work was partially supported by the French National Research Agency (ANR) through grants ANR-20-CE23-0007 and ANR-23-CE23-0002 and through the PEPR IA FOUNDRY project (ANR-23-PEIA-0003).

## References

- [1]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
- [2]
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.
Constitutional ai: Harmlessness from ai feedback, 2022.
- [3]
J.F. Banzhaf.
Weighted voting doesn’t work: A mathematical analysis.
Rutgers Law Review, 19(2):317–343, 1965.
- [4]
Louise Bloch and Christoph Friedrich.
Data analysis with shapley values for automatic subject selection in alzheimer’s disease data sets using interpretable machine learning.
Alzheimer’s Research & Therapy, 13, 09 2021.
- [5]
SHC Choi, Sung-Hyuk Cha, and Charles Tappert.
A survey of binary similarity and distance measures.
J. Syst. Cybern. Inf., 8, 11 2009.
- [6]
Pradeep Dubey, Abraham Neyman, and Robert James Weber.
Value theory without efficiency.
Mathematics of Operations Research, 6(1):122–128, 1981.
- [7]
Felipe Garrido-Lucero, Benjamin Heymann, Maxime Vono, Patrick Loiseau, and Vianney Perchet.
Du-shapley: A shapley value proxy for efficient dataset valuation, 2024.
- [8]
Amirata Ghorbani, Michael P. Kim, and James Zou.
A distributional framework for data valuation, 2020.
- [9]
Amirata Ghorbani and James Zou.
Data shapley: Equitable valuation of data for machine learning, 2019.
- [10]
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J. Spanos, and Dawn Song.
Efficient task-specific data valuation for nearest neighbor algorithms, 2020.
- [11]
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gurel, Bo Li, Ce Zhang, Dawn Song, and Costas Spanos.
Towards efficient data valuation based on the shapley value, 2023.
- [12]
Kevin Fu Jiang, Weixin Liang, James Zou, and Yongchan Kwon.
Opendataval: a unified benchmark for data valuation, 2023.
- [13]
Pang Wei Koh and Percy Liang.
Understanding black-box predictions via influence functions, 2020.
- [14]
Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon.
Consistent binary classification with generalized performance metrics.
In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
- [15]
Yongchan Kwon, Manuel A. Rivas, and James Zou.
Efficient computation and analysis of distributional shapley values.
In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 793–801. PMLR, 13–15 Apr 2021.
- [16]
Yongchan Kwon and James Zou.
Beta shapley: a unified and noise-reduced data valuation framework for machine learning, 2022.
- [17]
Weida Li and Yaoliang Yu.
Robust data valuation with weighted banzhaf values.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 60349–60383. Curran Associates, Inc., 2023.
- [18]
Sasan Maleki.
Addressing the computational issues of the shapley value with applications in the smart grid.
In University of Southampton, Physical Sciences and Engineering, Doctoral Thesis, 115pp, 2015.
- [19]
Irwin Mann and Lloyd S. Shapley.
Values of Large Games, IV: Evaluating the Electoral College by Montecarlo Techniques.
Rand Corporation, 1960.
- [20]
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller.
Pervasive label errors in test sets destabilize machine learning benchmarks, 2021.
- [21]
Konstantin D. Pandl, Fabian Feiland, Scott Thiebes, and Ali Sunyaev.
Trustworthy machine learning for health care: scalable data valuation with the shapley value.
In Proceedings of the Conference on Health, Inference, and Learning, CHIL ’21, page 47–57, New York, NY, USA, 2021. Association for Computing Machinery.
- [22]
Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson, and Gianluca Bontempi.
Calibrating probability with undersampling for unbalanced classification.
In 2015 IEEE Symposium Series on Computational Intelligence, pages 159–166, 2015.
- [23]
Lloyd S Shapley.
A value for n-person games.
In Harold W. Kuhn and Albert W. Tucker, editors, Contributions to the Theory of Games II, pages 307–317. Princeton University Press, Princeton, 1953.
- [24]
Richard Stanley.
An introduction to hyperplane arrangements, pages 389–496.
MIT, 10 2007.
- [25]
Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A. Dunnmon, James Zou, and Daniel L. Rubin.
Data valuation for medical imaging using shapley value and application to a large-scale chest x-ray dataset.
Scientific Reports, 11(1), April 2021.
- [26]
Dootika Vats and Christina Knudson.
Revisiting the gelman-rubin diagnostic, 2020.
- [27]
Jiachen T. Wang and Ruoxi Jia.
Data banzhaf: A robust data valuation framework for machine learning, 2023.
- [28]
Jiachen T. Wang, Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia.
Rethinking data shapley for data selection tasks: Misleads and merits, 2024.
- [29]
Tom Yan and Ariel D. Procaccia.
If you like shapley then you’ll love the core.
In AAAI Conference on Artificial Intelligence, 2021.
- [30]
T. Zaslavsky.
Facing up to Arrangements: Face-Count Formulas for Partitions of Space by Hyperplanes: Face-count Formulas for Partitions of Space by Hyperplanes.
American Mathematical Society: Memoirs of the American Mathematical Society. American Mathematical Society, 1975.
- [31]
Kaiping Zheng, Horng-Ruey Chua, Melanie Herschel, H. V. Jagadish, Beng Chin Ooi, and James Wei Luen Yip.
Exploiting negative samples: a catalyst for cohort discovery in healthcare analytics.
In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.

## Appendix A Additional settings & experiments

For the reader’s convenience, we first outline the main points covered in this section.

- –
Appendix [A.1](https://arxiv.org/html/2502.06574v2#A1.SS1): Experiment settings for empirical results in the main text.
- –
Appendix [A.2](https://arxiv.org/html/2502.06574v2#A1.SS2): Additional results on rank correlation for more binary classification metrics.
- –
Appendix [A.3](https://arxiv.org/html/2502.06574v2#A1.SS3): Additional results on rank correlation using the Spearman rank correlation.
- –
Appendix [A.4](https://arxiv.org/html/2502.06574v2#A1.SS4): Table for $R_{p}$ results in Section [4](https://arxiv.org/html/2502.06574v2#S4).

### A.1 Experiment settings for empirical results in the main text

In this section, we describe our experimental protocol for estimating semivalue scores, which serve to obtain all the tables and figures included in this paper.

#### Datasets.

Table [5](https://arxiv.org/html/2502.06574v2#A1.T5) summarizes the datasets used in our experiments, all of which are standard benchmarks in the data valuation literature [9, 16, 11, 27, 12]. Due to the computational cost of repeated model retraining in our experiments, we select a subset of $100$ instances for training and $50$ instances for testing from each dataset.

**Table 5: A summary of datasets used in experiments.**
| Dataset | Source |
| --- | --- |
| Breast | [https://www.openml.org/d/13](https://www.openml.org/d/13) |
| Titanic | [https://www.openml.org/d/40945](https://www.openml.org/d/40945) |
| Credit | [22] |
| Heart | [https://www.openml.org/d/43398](https://www.openml.org/d/43398) |
| Wind | [https://www.openml.org/d/847](https://www.openml.org/d/847) |
| CPU | [https://www.openml.org/d/761](https://www.openml.org/d/761) |
| 2DPlanes | [https://www.openml.org/d/727](https://www.openml.org/d/727) |
| Pol | [https://www.openml.org/d/722](https://www.openml.org/d/722) |

Because our primary objective is to measure how changing the utility alone affects semivalue rankings, we must eliminate any other sources of variation, such as different train/test splits, model initialization, or Monte Carlo sampling noise, that could confound our results. To this end, we enforce two strict controls for semivalue scores computation across utilities:

- 1.
A fixed learning context $(\mathcal{A},\mathcal{D}_{\text{test}})$,
- 2.
Aligned sampling for semivalue approximations.

##### Fixed learning context.

As outlined in Section [2](https://arxiv.org/html/2502.06574v2#S2), a utility function $u$ is defined as:

$$ $\displaystyle u(S)=\texttt{perf}(\mathcal{A}(S),\mathcal{D}_{\text{test}}),$ $$

where $\mathcal{A}$ is a learning algorithm that outputs a model trained on a dataset $S$, and perf evaluates the model on a test set $\mathcal{D}_{\text{test}}$. The learning algorithm $\mathcal{A}$ specifies the model class, objective function, optimization procedure, and hyperparameters (e.g., learning rate, weight initialization).

By fixing ($\mathcal{A},\mathcal{D}_{\rm{test}}$), we ensure that swapping between two utilities, say, accuracy versus F1-score, amounts solely to changing the performance metric perf. Consequently, any shift in the semivalue scores’ ranking (measured by rank correlation metrics) can only be attributed to the utility choice.

#### Controlling for sampling noise in semivalue estimates.

The above discussion assumes access to exact semivalue scores, but in practice, we approximate them via Monte Carlo permutation sampling, which injects random noise into each run. Without accounting for this sampling variability, differences in semivalue scores’ rankings could reflect estimator noise rather than genuine sensitivity to the utility.

To enforce this, we introduce *aligned sampling* alongside the fixed learning context $(\mathcal{A},\mathcal{D}_{\rm{test}})$. Aligned sampling consists of pre-generating a single pool of random permutations (or sampling seeds) and reusing those same permutations when estimating semivalues for each utility. By sharing both the model-training environment and the permutation draws, we ensure that any differences in resulting rankings are driven solely by the change in utility.

#### Fixed set of permutations.

Let $\mathcal{P}=\left\{\pi_{1},\pi_{2},\ldots,\pi_{m}\right\}$ denote a fixed set of $m$ random permutations of the data points in $\mathcal{D}$. We apply this exact set of permutations across multiple utilities $\left\{u_{1},u_{2},\ldots,u_{K}\right\}$ such that $u_{k}(\cdot)=\texttt{perf}_{k}[(\mathcal{A})(\cdot),\mathcal{D}_{\text{test}}]$ with fixed $(\mathcal{A},\mathcal{D}_{\text{test}})$ for all $k\in[K]$.

For a given performance metric $\texttt{perf}_{k}$ and the set of permutations $\mathcal{P}$, we estimate the marginal contributions $\{\hat{\Delta}_{j}\left(z_{i};u_{k}\right)\}_{j=1}^{n}$ for each data point $z_{i}\in\mathcal{D}$ with respect to the utility $u_{k}$ such as

$$ $\displaystyle\hat{\Delta}_{j}\left(z_{i};u_{k}\right):=\frac{1}{m}\sum_{s=1}^{ m}\left(u_{k}\left(S_{j}^{\pi_{s}}\cup\{z_{i}\}\right)-u_{k}\left(S_{j}^{\pi_{ s}}\right)\right),$ $$

where $m$ is the number of permutations used, $\pi_{s}$ denotes the $s$-th permutation and $S_{j}^{\pi_{s}}$ represents the subset of data points of size $j-1$ that precedes $z_{i}$ in the order defined by permutation $\pi_{s}$.

#### Determining the number of permutations m 𝑚 m italic_m .

The number of permutations $m$ used in the marginal contribution estimator is determined based on a maximum limit and a convergence criterion applied across all utilities $u_{1},\dots,u_{K}$. Formally,

$$ $\displaystyle m=\max\left(m_{\text{min}},\min\left(m_{\text{max}},m_{\text{ conv}}\right)\right),$ $$

where $m_{\text{min}}$ is a predefined minimum number of permutations to avoid starting convergence checks prematurely, $m_{\text{max}}$ is a predefined maximum number of permutations set to control computational feasibility, $m_{\text{conv}}$ is the smallest number of permutations required for the Gelman-Rubin (GR) [26] statistic to converge across all utility functions $u_{1},\dots,u_{K}$. Using the Gelman-Rubin statistic as a convergence criterion follows established practices in the literature [12, 16].

For each data point $z_{i}$, the GR statistic $R_{i}$ is computed for every $100$ permutations across all utilities. The sampling process halts when the maximum GR statistic across all data points and all utilities falls below a threshold, indicating convergence. We adopt the conventional threshold of $1.05$ for GR convergence, consistent with prior studies in data valuation [12].

In this framework, the GR statistic, $R_{i}^{k}$, is used to assess the convergence of marginal contribution estimates for each data point $z_{i}$ across $C$ independent chains of $s$ sampled permutations under each utility $u_{k}$. The GR statistic evaluates the agreement between chains by comparing the variability within each chain to the variability across the chains, with convergence indicated when $R_{i}^{k}$ approaches 1. Specifically, to compute the GR statistic for each data point $z_{i}$ under utility $u_{k}$, we determine

- 1.
The within-chain variance $W_{i}^{k}$ which captures the variability of marginal contributions for $z_{i}$ within each chain. Specifically, if there are $c$ independent chains, $W_{i}^{k}$ is calculated as the average of the sample variances within each chain
$W_{i}^{k}=\frac{1}{C}\sum_{c=1}^{C}s_{i,c}^{2},$
where $s_{i,c}^{2}$ is the sample variance of marginal contributions for $z_{i}$ within chain $c$. This term reflects the dispersion of estimates within each chain,
- 2.
And the between-chain variance $B_{i}^{k}$, which measures the variability between the mean marginal contributions across the chains. It indicates how much the chains differ from each other. The between-chain variance is defined as
$\displaystyle B_{i}^{k}=\frac{s}{C-1}\sum_{c=1}^{C}\left(\bar{\Delta}_{c}(z_{i
};u_{k})-\bar{\Delta}(z_{i};u_{k})\right)^{2},$
where $\bar{\Delta}_{c}(z_{i};u_{k})$ is the mean marginal contribution for $z_{i}$ in chain $c$, and $\bar{\Delta}(z_{i};u_{k})$ is the overall mean across all chains
$\displaystyle\bar{\Delta}(z_{i};u_{k})=\frac{1}{C}\sum_{c=1}^{C}\bar{\Delta}_{
c}(z_{i};u_{k}).$
The term $B_{i}^{k}$ quantifies the extent of disagreement among the chain means.

Combining both $W_{i}^{k}$ and $B_{i}^{k}$, the GR statistic $R_{i}^{k}$ for data point $z_{i}$ under utility $u_{k}$ is defined as:

$$ $\displaystyle R_{i}^{k}=\sqrt{\frac{(s-1)}{s}+\frac{B_{i}^{k}}{W_{i}^{k}\cdot s }}.$ $$

#### Intra-permutation truncation.

Building on existing literature [9, 12], we further improve computational efficiency by implementing an intra-permutation truncation criterion that restricts coalition growth once contributions stabilize. Given a random permutation $\pi_{s}\in\mathcal{P}$, the marginal contribution for each data point $z_{\pi_{s,j}}$ (the $j$-th point in the permutation $\pi_{s}$) is calculated incrementally as the coalition size $j$ increases from $1$ up to $n$. However, instead of expanding the coalition size through all $n$ elements, the algorithm stops increasing $j$ when the marginal contributions become stable based on a relative change threshold.

For each step $l\in[n]$ within a permutation, the relative change $V_{l}^{k}$ in the utility $u_{k}$ is calculated as:

$$ $\displaystyle V_{l}^{k}:=\frac{\left|u_{k}\left(\{z_{\pi_{s,j}}\}_{j=1}^{l} \cup\{z_{\pi_{s,l+1}}\}\right)-u_{k}\left(\{z_{\pi_{s,j}}\}_{j=1}^{l}\right) \right|}{u_{k}\left(\{z_{\pi_{s,j}}\}_{j=1}^{l}\right)}.$ $$

where $\{z_{\pi_{s,j}}\}_{j=1}^{l}$ represents the coalition formed by the first $l$ data points in $\pi_{s}$. This measures the relative change in the utility $u_{k}$ when adding the next data point to the coalition. The truncation criterion stops increasing the coalition size at the smallest value $j$ satisfying the following condition:

$$ $\displaystyle j^{*}=\arg\min\left\{j\in[n]:\left|\{l\leq j:V_{l}\leq 10^{-8}\} \right|\geq 10\right\}.$ $$

This means that the coalition size $j^{*}$ is fixed at the smallest $j$ for which there are at least $10$ prior values of $V_{l}$ (for $l\leq j$) that are smaller than a threshold of $10^{-8}$. This condition ensures that the utility $u_{k}$ has stabilized, indicating convergence within the permutation. This intra-permutation truncation reduces computational cost by avoiding unnecessary calculations once marginal contributions stabilize.

#### Aggregating marginal contributions for semivalues estimation.

Once the marginal contributions have been estimated consistently across all permutations and utilities, they are aggregated to compute various semivalues, such as the Shapley, Banzhaf, and $(4,1)$-Beta Shapley values. Each semivalue method applies a specific weighting scheme (see Definition [2.1](https://arxiv.org/html/2502.06574v2#S2.Thmtheorem1), [2.2](https://arxiv.org/html/2502.06574v2#S2.Thmtheorem2), [2.3](https://arxiv.org/html/2502.06574v2#S2.Thmtheorem3)) to the marginal contributions to reflect the intended measure of data point importance.

For a data point $z_{i}$ under utility $u_{k}$, its approximated data value $\hat{\phi}(z_{i};\omega,u_{k})$ is computed by applying a weighting scheme $\omega$ to the marginal contributions across coalition sizes

$$ $\displaystyle\hat{\phi}(z_{i};\omega,u_{k})=\sum_{j=1}^{n}\omega_{j}\,\hat{ \Delta}_{j}(z_{i};u_{k}),$ $$

where $\hat{\Delta}_{j}(z_{i};u_{k})$ is the estimated marginal contribution for coalition size $j-1$, and $\omega_{j}$ is the weight assigned to coalition size $j-1$.

#### Learning algorithm 𝒜 𝒜 \mathcal{A} caligraphic_A .

Across all experiments, $\mathcal{A}$ is a logistic-regression classifier (binary cross-entropy loss) trained via L-BFGS with $\ell_{2}$ regularization ($\lambda=1.0$). We initialize all weights from $\mathcal{N}(0,1)$ with a fixed random seed, disable early stopping, and fix the maximum number of training epochs to 100. The optimizer’s step size is $1.0$.

#### Decision-threshold calibration.

Because we compare multiple binary-classification utilities (accuracy, F1-score, etc.), using a fixed probability cutoff (e.g., $0.5$) can unfairly favor some metrics over others, especially under class imbalance. To ensure that differences in semivalue scores’ rankings arise from the utility definition (and not an arbitrary threshold), we calibrate the decision boundary to the empirical class prevalence. Concretely, if $p$ is the fraction of positive labels in the training set, we set the cutoff at the $(1-p)$-quantile of the model’s predicted probabilities. This way, each trained model makes exactly $p$% positive predictions, aligning base-rate assumptions across utilities and isolating the effect of the performance metric itself.

#### Computational resources and runtime.

All experiments ran on a single machine (Apple M1 (8-core CPU) with 16 GB RAM) without parallelization. A full semivalue estimation, consisting of $5$ independent Monte Carlo approximations, for one dataset of $100$ data points takes approximately $15$ minutes.

### A.2 Additional results on rank correlation for more binary classification metrics

In Table [1](https://arxiv.org/html/2502.06574v2#S1.T1), we compare semivalue score rankings under accuracy versus F1-score. Here, we broaden this analysis to include other widely used binary classification utilities (recall, negative log-loss, and arithmetic mean). Tables [6](https://arxiv.org/html/2502.06574v2#A1.T6) and [7](https://arxiv.org/html/2502.06574v2#A1.T7) show that ranking variability persists across datasets and semivalue choices when using these additional metrics.

**Table 6: Mean Kendall rank correlations (standard error in parentheses rounded to one significant figure for clarity) between accuracy (acc) and negative log-loss (nll), and between F1-score (f1) and negative log loss, for three semivalues (Shapley, Beta (4,1), Banzhaf). Values are averaged over $5$ estimations.**
| Dataset | Shapley | (4,1)-Beta Shapley | Banzhaf |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | acc-nll | f1-nll | acc-nll | f1-nll | acc-nll | f1-nll |
| Breast | -0.59 (0.02) | -0.60 (0.02) | -0.65 (0.01) | -0.66 (0.01) | 0.18 (0.01) | 0.18 (0.01) |
| Titanic | -0.53 (0.01) | 0.54 (0.01) | -0.60 (0.01) | -0.61 (0.01) | 0.14 (0.02) | -0.07 (0.01) |
| Credit | -0.59 (0.02) | -0.43 (0.01) | -0.66 (0.01) | -0.49 (0.01) | 0.38 (0.01) | 0.28 (0.03) |
| Heart | -0.04 (0.02) | 0.01 (0.02) | -0.20 (0.02) | -0.17 (0.03) | -0.07 (0.01) | -0.05 (0.01) |
| Wind | 0.67 (0.02) | 0.69 (0.01) | 0.74 (0.02) | 0.73 (0.01) | 0.26 (0.01) | 0.44 (0.01) |
| Cpu | 0.55 (0.01) | 0.68 (0.01) | 0.59 (0.01) | 0.69 (0.01) | -0.53 (0.01) | 0.52 (0.01) |
| 2dplanes | 0.22 (0.02) | 0.98 (0.01) | 0.41 (0.01) | 0.98 (0.01) | -0.03 (0.01) | 0.18 (0.01) |
| Pol | 0.58 (0.01) | 0.79 (0.01) | 0.74 (0.01) | 0.81 (0.01) | -0.01 (0.02) | 0.13 (0.02) |

**Table 7: Mean Kendall rank correlations (standard error in parentheses rounded to one significant figure for clarity) between recall (rec) and accuracy (acc) for three semivalues (Shapley, Beta (4,1), Banzhaf). Values are averaged over $5$ estimations.**
| Dataset | Shapley | (4,1)-Beta Shapley | Banzhaf |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | acc-am | acc-rec | am-rec | acc-am | acc-rec | am-rec | acc-am | acc-rec | am-rec |
| Breast | 0.93 (0.01) | 0.98 (0.01) | 0.92 (0.01) | 0.94 (0.01) | 0.98 (0.01) | 0.92 (0.01) | 0.82 (0.03) | 0.99 (0.01) | 0.81 (0.03) |
| Titanic | -0.25 (0.04) | 0.77 (0.02) | -0.05 (0.05) | -0.27 (0.03) | 0.62 (0.04) | 0.08 (0.05) | 0.46 (0.02) | 0.81 (0.01) | 0.65 (0.01) |
| Credit | -0.31 (0.01) | 0.07 (0.01) | 0.60 (0.02) | -0.31 (0.02) | 0.12 (0.04) | 0.62 (0.01) | 0.35 (0.01) | 0.58 (0.01) | 0.76 (0.01) |
| Heart | 0.19 (0.02) | 0.98 (0.01) | 0.18 (0.02) | 0.22 (0.01) | 0.98 (0.01) | 0.19 (0.01) | 0.61 (0.01) | 0.98 (0.01) | 0.59 (0.02) |
| Wind | 0.08 (0.03) | 0.98 (0.01) | 0.07 (0.03) | 0.10 (0.02) | 0.98 (0.01) | 0.08 (0.04) | 0.77 (0.01) | 0.98 (0.01) | 0.75 (0.01) |
| Cpu | 0.19 (0.04) | 0.75 (0.02) | 0.18 (0.01) | 0.22 (0.03) | 0.78 (0.02) | 0.22 (0.02) | 0.79 (0.01) | 0.93 (0.01) | 0.86 (0.01) |
| 2dplanes | 0.31 (0.02) | 0.99 (0.01) | 0.31 (0.02) | 0.33 (0.02) | 0.99 (0.01) | 0.33 (0.02) | 0.037 (0.01) | 0.99 (0.01) | 0.37 (0.01) |
| Pol | 0.56 (0.01) | 0.73 (0.01) | 0.29 (0.01) | 0.56 (0.01) | 0.79 (0.01) | 0.34 (0.01) | 0.67 (0.01) | 0.69 (0.01) | 0.36 (0.01) |

### A.3 Additional results on rank correlation using the Spearman rank correlation

For completeness, we re-evaluate all of our pairwise semivalue ranking comparisons using Spearman rank correlation instead of Kendall rank correlation. As shown in Tables [8](https://arxiv.org/html/2502.06574v2#A1.T8) and [9](https://arxiv.org/html/2502.06574v2#A1.T9), datasets and semivalues that exhibit low Kendall correlations between different utilities also yield low Spearman correlations, and vice versa.

**Table 8: Mean Spearman rank correlations (standard error in parentheses rounded to one significant figure for clarity) between accuracy (acc) and negative log-loss (nll), and between F1-score (f1) and negative log loss, for three semivalues (Shapley, Beta (4,1), Banzhaf). Values are averaged over $5$ estimations.**
| Dataset | Shapley | (4,1)-Beta Shapley | Banzhaf |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | acc-f1 | acc-nll | f1-nll | acc-f1 | acc-nll | f1-nll | acc-f1 | acc-nll | f1-nll |
| Breast | 0.99 (0.01) | -0.76 (0.02) | -0.78 (0.02) | 0.99 (0.01) | -0.82 (0.01) | -0.83 (0.01) | 0.98 (0.01) | 0.22 (0.01) | 0.23 (0.01) |
| Titanic | -0.20 (0.01) | -0.71 (0.01) | 0.74 (0.01) | -0.18 (0.01) | -0.79 (0.01) | -0.80 (0.01) | 0.95 (0.01) | 0.18 (0.02) | -0.20 (0.01) |
| Credit | -0.50 (0.02) | -0.76 (0.02) | -0.61 (0.02) | -0.52 (0.01) | -0.83 (0.01) | -0.68 (0.02) | 0.90 (0.01) | 0.53 (0.01) | 0.40 (0.03) |
| Heart | 0.71 (0.01) | -0.04 (0.02) | 0.03 (0.03) | 0.67 (0.01) | -0.28 (0.04) | -0.23 (0.04) | 0.96 (0.01) | -0.10 (0.02) | -0.08 (0.02) |
| Wind | 0.85 (0.01) | 0.84 (0.01) | 0.86 (0.01) | 0.85 (0.01) | 0.90 (0.01) | 0.89 (0.01) | 0.97 (0.01) | 0.34 (0.01) | 0.62 (0.01) |
| Cpu | 0.47 (0.02) | 0.73 (0.01) | 0.85 (0.01) | 0.45 (0.01) | 0.77 (0.01) | 0.86 (0.01) | 0.87 (0.01) | -0.71 (0.01) | 0.70 (0.01) |
| 2dplanes | 0.24 (0.01) | 0.33 (0.02) | 0.99 (0.01) | 0.28 (0.02) | 0.58 (0.01) | 0.99 (0.01) | 0.75 (0.01) | -0.04 (0.02) | 0.24 (0.05) |
| Pol | 0.70 (0.01) | 0.77 (0.01) | 0.92 (0.01) | 0.69 (0.01) | 0.90 (0.01) | 0.93 (0.01) | 0.53 (0.01) | -0.01 (0.03) | 0.21 (0.02) |

**Table 9: Mean Spearman rank correlations (standard error in parentheses rounded to one significant figure for clarity) between recall (rec) and accuracy (acc) for three semivalues (Shapley, Beta (4,1), Banzhaf). Values are averaged over $5$ estimations.**
| Dataset | Shapley | (4,1)-Beta Shapley | Banzhaf |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | acc-am | acc-rec | am-rec | acc-am | acc-rec | am-rec | acc-am | acc-rec | am-rec |
| Breast | 0.99 (0.01) | 0.99 (0.01) | 0.99 (0.01) | 0.99 (0.01) | 0.99 (0.01) | 0.99 (0.01) | 0.90 (0.02) | 0.99 (0.01) | 0.89 (0.03) |
| Titanic | -0.37 (0.05) | 0.91 (0.02) | -0.08 (0.08) | -0.37 (0.04) | 0.89 (0.02) | 0.10 (0.08) | 0.62 (0.03) | 0.93 (0.01) | 0.84 (0.01) |
| Credit | -0.45 (0.01) | 0.09 (0.02) | 0.79 (0.01) | -0.40 (0.03) | 0.11 (0.02) | 0.83 (0.01) | 0.5 (0.01) | 0.75 (0.01) | 0.92 (0.01) |
| Heart | 0.29 (0.02) | 0.99 (0.01) | 0.27 (0.02) | 0.28 (0.02) | 0.89 (0.02) | 0.27 (0.01) | 0.80 (0.02) | 0.99 (0.01) | 0.78 (0.02) |
| Wind | 0.12 (0.04) | 0.99 (0.01) | 0.11 (0.04) | 0.12 (0.03) | 0.97 (0.02) | 0.10 (0.01) | 0.92 (0.01) | 0.99 (0.01) | 0.92 (0.01) |
| Cpu | 0.27 (0.01) | 0.90 (0.01) | 0.27 (0.01) | 0.27 (0.02) | 0.92 (0.03) | 0.31 (0.03) | 0.93 (0.01) | 0.99 (0.01) | 0.97 (0.01) |
| 2dplanes | 0.44 (0.03) | 0.99 (0.01) | 0.44 (0.03) | 0.47 (0.03) | 0.99 (0.01) | 0.47 (0.03) | 0.52 (0.01) | 0.99 (0.01) | 0.52 (0.01) |
| Pol | 0.75 (0.01) | 0.90 (0.01) | 0.42 (0.01) | 0.74 (0.01) | 0.93 (0.01) | 0.48 (0.02) | 0.85 (0.01) | 0.87 (0.01) | 0.52 (0.01) |

### A.4 Table for R p subscript 𝑅 𝑝 R_{p} italic_R start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT results in Section 4

In support of Figure [3](https://arxiv.org/html/2502.06574v2#S4.F3) displayed in Section [4](https://arxiv.org/html/2502.06574v2#S4), Table [4](https://arxiv.org/html/2502.06574v2#A1.T4) below reports the mean and standard error of the $p$-robustness metric $R_{p}$ for $p\in\{500,1000,1500\}$ on each dataset and semivalue.

**Table 4: Mean $p$-robustness $R_{p}$ (standard error in parentheses) for $p\in\{500,1000,1500\}$ computed on datasets of size $n=50$ and estimated over $5$ Monte Carlo trials (each trial corresponding to approximating the semivalue scores). Boldface marks the semivalue with the highest $R_{p}$ for each dataset and $p$. Higher $R_{p}$ indicates greater stability of the induced ranking under utility shifts.**
| Dataset | $R_{500}$ | $R_{1000}$ | $R_{1500}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Shapley | (4,1)-Beta Shapley | Banzhaf | Shapley | (4,1)-Beta Shapley | Banzhaf | Shapley | (4,1)-Beta Shapley | Banzhaf |
| Breast | 0.22 (0.004) | 0.23 (0.004) | 0.34 (0.06) | 0.63 (0.004) | 0.64 (0.003) | 0.59 (0.02) | 0.79 (0.003) | 0.80 (0.002) | 0.71 (0.06) |
| Titanic | 0.058 (0.001) | 0.058 (0.001) | 0.44 (0.03) | 0.27 (0.004) | 0.28 (0.004) | 0.81 (0.01) | 0.48 (0.004) | 0.48 (0.004) | 0.89 (0.007) |
| Credit | 0.084 (0.005) | 0.091 (0.005) | 0.82 (0.07) | 0.36 (0.01) | 0.38 (0.01) | 0.97 (0.01) | 0.58 (0.01) | 0.60 (0.01) | 0.99 (0.002) |
| Heart | 0.16 (0.003) | 0.16 (0.003) | 0.53 (0.01) | 0.54 (0.008) | 0.54 (0.009) | 0.87 (0.007) | 0.73 (0.006) | 0.73 (0.006) | 0.93 (0.003) |
| Wind | 0.23 (0.009) | 0.21 (0.01) | 0.58 (0.01) | 0.64 (0.01) | 0.62 (0.009) | 0.88 (0.005) | 0.79 (0.005) | 0.77 (0.007) | 0.94 (0.004) |
| Cpu | 0.11 (0.003) | 0.12 (0.003) | 0.38 (0.02) | 0.45 (0.009) | 0.45 (0.009) | 0.77 (0.009) | 0.66 (0.009) | 0.67 (0.009) | 0.88 (0.004) |
| 2dplanes | 0.090 (0.001) | 0.084 (0.002) | 0.19 (0.012) | 0.38 (0.004) | 0.36 (0.006) | 0.59 (0.02) | 0.59 (0.004) | 0.57 (0.006) | 0.75 (0.01) |
| Pol | 0.090 (0.003) | 0.09 (0.003) | 0.19 (0.01) | 0.39 (0.008) | 0.42 (0.007) | 0.50 (0.01) | 0.60 (0.006) | 0.62 (0.006) | 0.70 (0.01) |

## Appendix B Additional proofs & derivations

For the reader’s convenience, we first outline the main points covered in this section.

- –
Appendix [B.1](https://arxiv.org/html/2502.06574v2#A2.SS1): First-order approximation of the utility in the *multiple-valid-utility* scenario for binary classification and empirical validation.
- –
Appendix [B.2](https://arxiv.org/html/2502.06574v2#A2.SS2): Proof of Proposition [3.1](https://arxiv.org/html/2502.06574v2#S3.Thmtheorem1) and its extension to $K\geq 2$ base utilities.
- –
Appendix [B.3](https://arxiv.org/html/2502.06574v2#A2.SS3): Ranking region counts for specific cases of spatial signature.
- –
Appendix [B.4](https://arxiv.org/html/2502.06574v2#A2.SS4): Closed-form for $\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]$.
- –
Appendix [B.5](https://arxiv.org/html/2502.06574v2#A2.SS5): Maximum average $p$-swaps distance occurs under collinearity of the spatial signature.
- –
Appendix [B.6](https://arxiv.org/html/2502.06574v2#A2.SS6): Why can the Kendall rank correlation degrade by more than $\frac{2p}{N}$ if ties occur?
- –
Appendix [B.7](https://arxiv.org/html/2502.06574v2#A2.SS7): Analytic derivation of the correlation decomposition in Section [4](https://arxiv.org/html/2502.06574v2#S4).

### B.1 First-order approximation of the utility in the multiple-valid-utility scenario for binary classification and empirical validation

This section justifies the approximation used in Section [3](https://arxiv.org/html/2502.06574v2#S3), where a linear-fractional utility function $u$ is replaced by its affine surrogate.

Formally, we state in Section [3](https://arxiv.org/html/2502.06574v2#S3) that any linear-fractional utility $u$ of the form ([2](https://arxiv.org/html/2502.06574v2#S3.E2)) with $d_{0}\neq 0$, admits a first-order (Taylor–Young) expansion around $(\lambda,\gamma)=(0,0)$ of the form

$$ $\displaystyle u(S)=\frac{c_{0}}{d_{0}}+\frac{c_{1}d_{0}-c_{0}d_{1}}{d_{0}^{2}} \lambda(S)+\frac{c_{2}\,d_{0}-c_{0}\,d_{2}}{d_{0}^{2}}\gamma(S)+o\bigl{(}\max \{|\lambda(S)|,|\gamma(S)|\}\bigr{)}.$ $$

where $\{c_{0},c_{1},c_{2},d_{0},d_{1},d_{2}\}$ are real coefficients which specify the particular utility.

The proof is a direct Taylor expansion of
$u$ at $(\lambda,\gamma)=(0,0)$, followed by an empirical validation of the affine surrogate by inspecting discordance rates.

###### Proof.

Define $N(\lambda,\gamma)=c_{0}+c_{1}\,\lambda+c_{2}\,\gamma$ and $D(\lambda,\gamma)=d_{0}+d_{1}\,\lambda+d_{2}\,\gamma$ so that $u(S)=f\bigl{(}\lambda(S),\gamma(S)\bigr{)}$ with

$$ $\displaystyle f(\lambda,\gamma)=\frac{N(\lambda,\gamma)}{D(\lambda,\gamma)}.$ $$

Assuming $d_{0}\neq 0$ (i.e., the denominator does not vanish at $(0,0)$), the first-order Taylor expansion of $f$ around $(0,0)$ is

$$ $\displaystyle f(\lambda,\gamma)=f(0,0)+\left.\frac{\partial f}{\partial\lambda }\right|_{(0,0)}\lambda+\left.\frac{\partial f}{\partial\gamma}\right|_{(0,0)} \gamma+o(\lVert(\lambda,\gamma)\rVert).$ $$

Concretely,

$$ $\displaystyle f(0,0)=\frac{c_{0}}{d_{0}},\left.\quad\frac{\partial f}{\partial \lambda}\right|_{(0,0)}=\frac{c_{1}d_{0}-c_{0}d_{1}}{d_{0}^{2}},\left.\quad \frac{\partial f}{\partial\gamma}\right|_{(0,0)}=\frac{c_{2}d_{0}-c_{0}d_{2}}{ d_{0}^{2}}.$ $$

Moreover, since all norms are equivalent in $\mathbb{R}^{2}$, the Euclidean norm $\lVert(\lambda,\gamma)\rVert$ is equivalent to the infinity norm $\max\{\lambda,\gamma\}$. This concludes the proof.
∎

To verify that the affine surrogate faithfully preserves the true utility’s induced ordering, we compare rankings under $u$ and under its first-order proxy $\hat{u}=\frac{c_{0}}{d_{0}}+\frac{c_{1}d_{0}-c_{0}d_{1}}{d_{0}^{2}}\lambda+
\frac{c_{2}d_{0}-c_{0}d_{2}}{d_{0}^{2}}\gamma$. For each of the eight public binary-classification datasets introduced in Section [A.1](https://arxiv.org/html/2502.06574v2#A1.SS1), and for each of the three semivalues (Shapley, $(4,1)$-Beta Shapley, and Banzhaf), we proceed as follows:

- 1.
*Exact ranking*. Compute semivalue scores by using the exact linear-fractional utility $u$, then sort the resulting scores to obtain a reference ranking $r$ of the $n$ data points.
- 2.
*Affine surrogate ranking*. Replace the utility $u$ with its first-order affine approximation around $(\lambda,\gamma)=(0,0)$ denoted as $\hat{u}$, compute semivalue scores, and sort to obtain an approximate ranking $\hat{r}$.
- 3.
*Discordance measurement*. For each pair of rankings $(r,\hat{r})$, count the number $d$ of discordant pairs (i.e., pairs of points ordered differently between the two rankings), and record the proportion $d/N$, where $N=\binom{n}{2}$.
- 4.
*Repetition and averaging* Repeat steps 1–3 several times, each time using an independent Monte Carlo approximation of the semivalue scores, to capture sampling variability.

Table [3](https://arxiv.org/html/2502.06574v2#A2.T3) reports, for each dataset and semivalue, the average proportion of discordant pairs ($\pm$ standard error) between rankings obtained with the exact linear-fractional utility and its first-order affine proxy, for both F1-score and Jaccard coefficient (see Table [2](https://arxiv.org/html/2502.06574v2#A3.T2) for their definitions). In all experiments, the sum of the mean discordance rate and its standard error never exceeds 2.3%.

These discordance rates, at most a few percent of all $\binom{n}{2}$ pairs, confirm that, in practice, the omitted higher-order terms of the utility have only a minor effect on the induced semivalue ranking. Consequently, using the affine surrogate instead of the exact linear-fractional form is reasonably justified whenever one’s primary interest lies in the *ordering* of data values rather than their precise numerical magnitudes.

**Table 3: Proportion of discordant pairs (± standard error) between rankings induced by the exact linear-fractional utility $u$ and its first-order affine surrogate $\hat{u}$, for F1-score and Jaccard utilities. Values are computed over $N=\binom{50}{2}=1225$ pairs and averaged over $5$ Monte Carlo trials.**
| Dataset | F1-score | Jaccard |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Shapley | (4,1)-Beta Shapley | Banzhaf | Shapley | (4,1)-Beta Shapley | Banzhaf |
| Breast | $0.8\%\ (0.1\%)$ | $0.8\%\ (0.2\%)$ | $0.9\%\ (0.1\%)$ | $0.7\%\ (0.1\%)$ | $0.9\%\ (0.1\%)$ | $0.9\%\ (0.2\%)$ |
| Titanic | $1.3\%\ (0.3\%)$ | $1.3\%\ (0.3\%)$ | $0.8\%\ (0.3\%)$ | $1.6\%\ (0.4\%)$ | $1.3\%\ (0.3\%)$ | $0.7\%\ (0.1\%)$ |
| Credit | $1.5\%\ (0.5\%)$ | $1.6\%\ (0.2\%)$ | $1.0\%\ (0.1\%)$ | $1.5\%\ (0.3\%)$ | $1.7\%(0.1\%)$ | $0.7\%\ (0.3\%)$ |
| Heart | $1.0\%\ (0.1\%)$ | $0.8\%\ (0.1\%)$ | $0.8\%\ (0.1\%)$ | $1.2\%\ (0.2\%)$ | $1.1\%\ (0.3\%)$ | $0.7\%\ (0.2\%)$ |
| Wind | $1.0\%\ (0.2\%)$ | $0.8\%\ (0.1\%)$ | $0.8\%\ (0.2\%)$ | $0.9\%(0.1\%)$ | $1.2\%\ (0.4\%)$ | $1.0\%\ (0.4\%)$ |
| Cpu | $1.6\%\ (0.5\%)$ | $1.2\%\ (0.2\%)$ | $0.7\%\ (0.1\%)$ | $1.3\%\ (0.2\%)$ | $1.3\%\ (0.2\%)$ | $0.9\%\ (0.1\%)$ |
| 2dplanes | $1.7\%\ (0.1\%)$ | $1.9\%(0.1\%)$ | $0.8\%\ (0.1\%)$ | $1.3\%(0.1\%)$ | $1.6\%(0.2\%)$ | $1.1\%\ (0.4\%)$ |
| Pol | $1.8\%\ (0.4\%)$ | $2.0\%\ (0.2\%)$ | $1.5\%\ (0.5\%)$ | $2.1\%\ (0.2\%)$ | $2.0\%\ (0.2\%)$ | $1.6\%\ (0.5\%)$ |

###### Proof.

### B.2 Proof of Proposition 3.1 and its extension to K ≥ 2 𝐾 2 K\geq 2 italic_K ≥ 2 base utilities

This section provides the formal proof of Proposition [B.1](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem1), which generalizes Proposition [3.1](https://arxiv.org/html/2502.06574v2#S3.Thmtheorem1). It states that the semivalue score of any data point under a utility that is a linear combination of $K$ base utilities can be written as an inner product in $\mathbb{R}^{K}$. This result forms the backbone of the geometric perspective developed in Section [3](https://arxiv.org/html/2502.06574v2#S3).

###### Proposition B.1 (Extension of Proposition 3.1 to $K\geq 2$ base utilities) .

Let $\mathcal{D}$ be any dataset of size $n$ and let $\omega\in\mathbb{R}^{n}$ be a semivalue weight vector. Then there exists a map $\psi_{\omega,\mathcal{D}}:\mathcal{D}\longrightarrow\mathbb{R}^{K}$
such that for every utility $u_{\alpha}=\sum_{k=1}^{K}\alpha_{k}u_{k}$, $\phi\bigl{(}z;\omega,u_{\alpha}\bigr{)}=\bigl{\langle}\psi_{\omega,\mathcal{D}
}(z),\alpha\bigr{\rangle}$, for any $z\in\mathcal{D}$.
We call
$\mathcal{S}_{\omega,\mathcal{D}}=\{\psi_{\omega,\mathcal{D}}(z)\mid z\in
\mathcal{D}\}$ the *spatial signature* of $\mathcal{D}$ under semivalue $\omega$.

The proof is a straightforward application of semivalue linearity. The main contribution is the geometric interpretation of semivalue vectors as projections.

###### Proof.

For each data point $z\in\mathcal{D}$, let its semivalue characterized by $\omega$ be denoted by $\varphi(z;\omega,u_{\alpha})$ when the utility is $u_{\alpha}$. Under the standard linearity property of semivalues, the following linear decomposition holds:

$$ $\displaystyle\phi\Bigl{(}z;\omega,u_{\alpha}\Bigr{)}=\phi\Bigl{(}z;\omega,\sum _{k=1}^{K}\alpha_{k}u_{k}\Bigr{)}=\sum_{k=1}^{K}\alpha_{k}\phi(z;\omega,u_{k}).$ $$

So if we define for each $z$,

$$ $\displaystyle\psi_{\omega,\mathcal{D}}(z)=\Bigl{(}\phi(z;\omega,u_{1}),\dots, \phi(z;\omega,u_{K})\Bigr{)}\in\mathbb{R}^{K},$ $$

then by definition of the scalar (inner) product in $\mathbb{R}^{K}$,

$$ $\displaystyle\phi(z;\omega,u_{\alpha})=\langle\psi_{\omega,\mathcal{D}}(z), \alpha\rangle.$ $$

∎

In the main text, we focus on the case where $K=2$, i.e., utilities correspond to directions on the unit circle $\mathcal{S}^{1}$. Proposition [B.1](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem1) shows that the same reasoning carries over to any finite family of $K$ base utilities: data points embed as $\psi_{\omega,\mathcal{D}}(z)\in\mathbb{R}^{K}$, and ranking by a convex combination $u_{\alpha}=\sum_{k=1}^{K}\alpha_{k}u_{k}$ is equivalent to sorting the inner products $\langle\psi_{\omega,\mathcal{D}}(z),\alpha\rangle$. Since only the direction of $\alpha$ matters, each utility is identified with a point $\bar{\alpha}=\alpha/\|\alpha\|$ on the unit sphere $\mathcal{S}^{K-1}$. Thus, for general $K$, robustness to utility choice reduces to studying how the ordering of these projections varies as $\bar{\alpha}$ moves over $\mathcal{S}^{K-1}$.

###### Proposition B.1 (Extension of Proposition 3.1 to K ≥ 2 𝐾 2 K\geq 2 italic_K ≥ 2 base utilities) .

###### Proof.

### B.3 Ranking regions counts for specific cases of spatial signatures

This section formalizes the notion of *ranking regions*, which play a central role in the robustness analysis developed in Section [3](https://arxiv.org/html/2502.06574v2#S3). We begin by considering the hyperplane arrangement induced by all pairwise differences between embedded data points in the spatial signature. This arrangement partitions space into connected components, referred to as *regions* in the theory of hyperplane arrangements (see Definition [B.2](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem2)). In our context, each such region corresponds to a set of utility directions under which the ordering of data points remains constant. We refer to these as *ranking regions*.

###### Definition B.2 (Region of a hyperplane arrangement) .

Let $\mathcal{A}\subset V$ be a finite arrangement of hyperplanes in a real vector space $V$. The *regions* of $\mathcal{A}$ are the connected components of

$$ $\displaystyle V\setminus\bigcup_{H\in\mathcal{A}}H.$ $$

Each region is the interior of a (possibly unbounded) polyhedral cone and is homeomorphic to
$V$. We denote the number of such regions by $r(\mathcal{A})$.

We now specialize Definition [B.2](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem2) to our data valuation setting. Let $\mathcal{D}=\{z_{1},\dots,z_{n}\}$ be a dataset and let $\psi_{\omega,\mathcal{D}}(z_{i})\in\mathbb{R}^{K}$ denote the embedding of each point under semivalue weighting $\omega$.
For each pair $i<j$, we define

$$ $\displaystyle H_{ij}=\left\{\alpha\in\mathbb{R}^{K}:\left\langle\alpha,\psi_{ \omega,\mathcal{D}}(z_{i})-\psi_{\omega,\mathcal{D}}(z_{j})\right\rangle=0 \right\}.$ $$

Each set $H_{ij}$ is defined as the kernel of the linear functional $\alpha\mapsto\left\langle\alpha,\psi_{\omega,\mathcal{D}}(z_{i})-\psi_{\omega,
\mathcal{D}}(z_{j})\right\rangle$.
Since $\psi_{\omega,\mathcal{D}}(z_{i})\neq\psi_{\omega,\mathcal{D}}(z_{j})$
for $i\neq j$ (unless the data points are embedded identically), the difference vector $\psi_{\omega,\mathcal{D}}(z_{i})-\psi_{\omega,\mathcal{D}}(z_{j})\in\mathbb{R}
^{K}$ is nonzero. Therefore, this kernel is a linear subspace of codimension one in $\mathbb{R}^{K}$, which, by definition, is a hyperplane. Moreover, each $H_{ij}$ contains the origin $\alpha=0_{K}$; it is thus a central hyperplane by definition.

Each hyperplane $H_{ij}$ is the set of utility directions that assign equal projection scores to points $z_{i}$ and $z_{j}$. The finite arrangement $\mathcal{A}_{\omega,\mathcal{D}}=\{H_{ij}:1\leq i<j\leq n\}$ then induces a collection of regions in the sense of Definition [B.2](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem2), partitioning $\mathbb{R}^{K}$ into open cones such that, in each region, the relative ordering of projected values $\langle\alpha,\psi_{\omega,\mathcal{D}}(z_{i})\rangle$ and $\langle\alpha,\psi_{\omega,\mathcal{D}}(z_{j})\rangle$ remains the same for all $i<j$. Therefore, each region determines a unique ordering of the embedded points, corresponding to a distinct way of ranking the data points of $\mathcal{D}$ based on utility direction. To study robustness with respect to directional changes, we project this arrangement onto the unit sphere $\mathcal{S}^{K-1}$. Since all hyperplanes are central, their intersection with the sphere produces great spheres, and the resulting decomposition of $\mathcal{S}^{K-1}$ consists of spherical connected regions over which the ranking of the data points remains invariant. We refer to these regions as *ranking regions*. Formally, a *ranking region* is a connected component of $\mathcal{S}^{K-1}\setminus\bigcup_{i<j}\big{(}H_{ij}\bigcap\mathcal{S}^{K-1}
\big{)}$.

We now study how the number of such ranking regions depends on the geometry of the spatial signature. In particular, using Proposition [B.3](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem3), we provide an explicit count of ranking regions in two specific geometric configurations of the embedded points.

###### Proposition B.3 (Regions counts) .

Let $\mathcal{A}=\{H_{1},\dots,H_{m}\}$ be an arrangement of $m$ central (i.e., origin-passing) hyperplanes in a real vector space $V$ of dimension $K$.

- 1.
If no $K$ hyperplanes in the arrangement intersect in a common subspace of dimension greater than zero (in particular, not in a line), then the number of regions into which $\mathcal{A}$ partitions $V$ is
$\displaystyle r(\mathcal{A})=2\sum_{k=0}^{K-1}\binom{m-1}{j}.$
- 2.
If all hyperplanes coincide (i.e., $H_{1}=\cdots=H_{m}$), then the number of regions is:
$\displaystyle r(\mathcal{A})=2.$

###### Proof.

Let $\mathcal{A}=\left\{H_{1},\ldots,H_{m}\right\}$ be an arrangement of $m$ hyperplanes in a real vector space $V$ of dimension $K$.

- 1.
Suppose no $K$ hyperplanes in the arrangement intersect in a common subspace of dimension greater than zero (in particular, not in a line).
Choose any hyperplane $H\in\mathcal{A}$, and define two affine hyperplanes $H^{+}$and $H^{-}$, parallel to $H$ and on opposite sides of the origin, such that the origin lies strictly between them.
Each of the remaining $m-1$ hyperplanes of $\mathcal{A}$ intersects $H^{+}$ in a hyperplane of dimension $K-2$, and these intersections form an arrangement of $m-1$ hyperplanes in $H^{+}$ (which is a space of dimension $K-1$). By Proposition 2.4 in [24] (derived from Zaslavsky’s work [30]), the number of regions induced by this non-central(^5^55Since the $m-1$ hyperplanes do not all pass through a same point on $\mathrm{H}^{+}$.) arrangement in $\mathrm{H}^{+}$ is:
$\displaystyle\sum_{j=0}^{n-1}\binom{m-1}{j}$
These regions correspond exactly to the regions of $V\backslash\bigcup_{H\in\mathcal{A}}H$ that lie entirely on one side of $H$. By symmetry, the same number of regions lies on the opposite side (on $\mathrm{H}^{-}$). Therefore, the total number of regions for the whole arrangement is:
$\displaystyle r(\mathcal{A})=2\sum_{j=0}^{n-1}\binom{m-1}{j}$
- 2.
Suppose that all hyperplanes in the arrangement coincide, i.e., $H_{1}=\cdots=H_{m}=H$ for some hyperplane $H\subset V$. Then
$\displaystyle\bigcup_{H\in\mathcal{A}}H=H,$
and the complement $V\setminus\bigcup_{H\in\mathcal{A}}H$ consists of exactly two connected open sets: the two half-spaces determined by $H$. Therefore, the number of regions is
$\displaystyle r(\mathcal{A})=2.$

∎

We now apply Proposition [B.3](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem3) to the arrangement $\mathcal{A}_{\omega,\mathcal{D}}$ formed by the hyperplanes $H_{ij}$
defined from pairwise differences of embedded points in the spatial signature $S_{\omega,\mathcal{D}}$. Since each $H_{ij}$ is a central hyperplane in $\mathbb{R}^{K}$, the arrangement $\mathcal{A}_{\omega,\mathcal{D}}$ partitions the space into open polyhedral cones, whose connected components are the regions of the arrangement. Each of these cones intersects the unit sphere $\mathcal{S}^{K-1}$ in a unique open subset, yielding a spherical partition of $\mathcal{S}^{K-1}$. Therefore, the number of ranking regions on $\mathcal{S}^{K-1}$
is equal to the number of regions of the central hyperplane arrangement in $\mathbb{R}^{K}$, and can be computed directly using Proposition [B.3](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem3).

Corollary [B.4](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem4) provides the number of ranking regions for two specific geometric configurations of the spatial signature.

###### Corollary B.4 (Ranking regions counts) .

Let $\mathcal{D}=\{z_{1},\dots,z_{n}\}$ be a dataset and let $\psi_{\omega,\mathcal{D}}(z_{i})\in\mathbb{R}^{K}$ denote the spatial signature of point $z_{i}$
under semivalue weighting $\omega$. For each pair $i<j$, define the hyperplane

$$ $\displaystyle H_{ij}=\left\{\alpha\in\mathbb{R}^{K}:\left\langle\alpha,\psi_{ \omega,\mathcal{D}}(z_{i})-\psi_{\omega,\mathcal{D}}(z_{j})\right\rangle=0 \right\}.$ $$

Since there are $\binom{n}{2}$ pairs $(i,j)$, the arrangement $\mathcal{A}_{\omega,\mathcal{D}}=\{H_{ij}:1\leq i<j\leq n\}$
consists of $N=\binom{n}{2}$ central hyperplanes in $\mathbb{R}^{K}$. Let $r(\mathcal{A}_{\omega,\mathcal{D}})$ denote the number of connected regions in the complement of this arrangement. Then

- 1.
If no $K$ hyperplanes $H_{ij}$ intersect in a common subspace of dimension greater than zero, the number of ranking regions is
$\displaystyle r(\mathcal{A}_{\omega,\mathcal{D}})=2\sum_{k=0}^{K-1}\binom{N-1}
{k},\quad\text{where }N=\binom{n}{2}.$
- 2.
If all embedded points $\psi_{\omega,\mathcal{D}}(z_{i})$ lie on a line in $\mathbb{R}^{K}$, then all hyperplanes $H_{ij}$ coincide and
$\displaystyle r(\mathcal{A}_{\omega,\mathcal{D}})=2.$

Figure [2](https://arxiv.org/html/2502.06574v2#S3.F2) illustrates these two specific geometric configurations on the circle $\mathcal{S}^{1}$ (corresponding to the case $K=2$). In both cases, the observed number of ranking regions coincides with the counts given by Corollary [B.4](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem4).

###### Definition B.2 (Region of a hyperplane arrangement) .

###### Proposition B.3 (Regions counts) .

###### Proof.

###### Corollary B.4 (Ranking regions counts) .

### B.4 Closed-form for 𝔼 α ¯ ⁢ [ ρ p ⁢ ( α ¯ ) ] subscript 𝔼 ¯ 𝛼 delimited-[] subscript 𝜌 𝑝 ¯ 𝛼 \mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})] blackboard_E start_POSTSUBSCRIPT over¯ start_ARG italic_α end_ARG end_POSTSUBSCRIPT [ italic_ρ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( over¯ start_ARG italic_α end_ARG ) ]

This section provides the derivation of a closed-form expression for $\mathbb{E}[\rho_{p}]$, introduced in Section [3](https://arxiv.org/html/2502.06574v2#S3), which quantifies how far, on average, one must rotate the utility direction on the sphere before $p$ pairwise ranking swaps occur. In Section [3](https://arxiv.org/html/2502.06574v2#S3), we describe how this quantity captures the local stability of the ranking induced by the spatial signature. Here, we formally compute this quantity in the case where $K=2$ (i.e., in the case where the utilities we consider can be written as a linear combination of two base-utilities). We also show that the closed-form expression derived in the $K=2$ case can be computed in $\mathcal{O}(n^{2}\log n)$ time. Finally, we briefly discuss the higher-dimensional case $K>2$, for which no closed-form is available, and describe how $\mathbb{E}[\rho_{p}]$ can be approximated via Monte Carlo sampling.

Recall that $\rho_{p}(\bar{\alpha})$ measures the minimal geodesic distance one must rotate a utility direction $\bar{\alpha}\in\mathcal{S}^{K-1}$ before the ranking of points in $\mathcal{D}=\{z_{i}\}_{i\in[n]}$ changes by $p$ pairwise swaps. Each pair of points $(z_{i},z_{j})$ defines *cuts* on $\mathcal{S}^{K-1}$ which are utility directions along which the scores of $z_{i}$ and $z_{j}$ are equal. These cuts partition $\mathcal{S}^{K-1}$ into (ranking) regions where the ranking of points remains fixed. In what follows, we focus on the case $K=2$, where utility directions lie on the unit circle $\mathcal{S}^{1}$, and $\rho_{p}(\bar{\alpha})$ can be treated as a function of the angle associated with $\bar{\alpha}\in\mathcal{S}^{1}$.

We parametrize the unit circle $\mathcal{S}^{1}$
by the angle $\varphi\in[0,2\pi[$, writing $\bar{\alpha}(\varphi)=(\cos{\varphi},\sin{\varphi})\in\mathcal{S}^{1}$.
Since $\rho_{p}$ depends only on this angle, we abbreviate $\rho_{p}\bigl{(}\bar{\alpha}(\varphi)\bigr{)}$
by $\rho_{p}(\varphi)$. Equivalently,

$$ $\displaystyle\mathbb{E}_{\bar{\alpha}\sim\mathrm{Unif}(\mathcal{S}^{1})}[\rho_ {p}(\bar{\alpha})]=\mathbb{E}_{\varphi\sim\mathrm{Unif}[0,2\pi[}[\rho_{p}( \varphi)]=\frac{1}{2\pi}\int_{0}^{2\pi}\rho_{p}(\varphi)d\varphi.$ $$

Recall from Section [3](https://arxiv.org/html/2502.06574v2#S3) that the $2N$ emphcut angles $\theta_{1}\leq\theta_{2}\leq\cdots<\theta_{2N}<2\pi$
partition the interval $[0,2\pi[$
into arcs of lengths $\lambda_{k}=\theta_{k+1}-\theta_{k}$
(with $\theta_{2N+1}=\theta_{1}+2\pi$). Within each arc, the ranking remains fixed, and crossing into the next arc incurs exactly one additional swap.

For $\varphi\in(\theta_{k},\theta_{k+1})$, the function $\rho_{p}(\varphi)$ equals the minimum of the clockwise and counterclockwise distances to the
$p$-th next cut:

$$ $\displaystyle\rho_{p}(\varphi)=\min\bigl{\{}S_{k}^{+}(p)-(\varphi-\theta_{k}), S_{k}^{-}(p)+(\varphi-\theta_{k})\bigr{\}},$ $$

where the quantities $S_{k}^{+}(p)$ and $S_{k}^{-}(p)$, recalled from Section [3](https://arxiv.org/html/2502.06574v2#S3), are defined as

$$ $\displaystyle S_{k}^{+}(p)=\sum_{i=1}^{p}\lambda_{(k+i)\bmod 2N},\quad S_{k}^{ -}(p)=\sum_{i=1}^{p}\lambda_{(k-i)\bmod 2N}.$ $$

Hence, the average value of $\rho_{p}$
can be written as

$$ $\displaystyle\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]=\frac{1}{2\pi} \int_{0}^{2\pi}\rho_{p}(\varphi)d\varphi=\frac{1}{2\pi}\sum_{k=1}^{2N}\int_{ \theta_{k}}^{\theta_{k+1}}\rho_{p}(\varphi)d\varphi=\frac{1}{2\pi}\sum_{k=1}^{ 2N}\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t,S_{k}^{-}(p)+t\}dt,$ (3) $$

where we set $t=\varphi-\theta_{k}\in[0,\lambda_{k}]$ as a local coordinate that measures the angular distance from the left endpoint of the $k$-th arc.

The expression inside the integral reflects the shortest of two angular paths along the circle from the start of the $k$-th arc: one going clockwise (of length $S_{k}^{+}(p)-t$) and one counterclockwise (of length $S_{k}^{-}(p)+t$). These two expressions intersect at

$$ $\displaystyle t_{k}^{*}=\frac{S_{k}^{+}(p)-S_{k}^{-}(p)}{2}.$ $$

Intuitively,

- •
If $t_{k}^{*}\leq 0$, even at $t=0$, the clockwise path is already shorter, so $\rho_{p}(t)=S_{k}^{+}(p)-t$ for all $t\in[0,\lambda_{k}]$.
- •
If $t_{k}^{*}\geq\lambda_{k}$, the counterclockwise path is shorter throughout the entire arc, $\rho_{p}(t)=S_{k}^{-}(p)+t$ for all $t\in[0,\lambda_{k}]$.
- •
If $0<t_{k}^{*}<\lambda_{k}$, then for $t<t_{k}^{*}$ the counterclockwise path is shorter, and for $t>t_{k}^{*}$, the clockwise path is shorter.

We therefore split $\int_{0}^{\lambda_{k}}\rho_{p}(t)dt=\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t
,S_{k}^{-}(p)+t\}dt$ into the three cases:

- 1.
If $t_{k}^{\star}\leq 0$, we have
$\displaystyle\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t,S_{k}^{-}(p)+t\}dt=
\int_{0}^{\lambda_{k}}(S_{k}^{+}(p)-t)dt=S_{k}^{+}(p)\lambda_{k}-\frac{1}{2}
\lambda_{k}^{2}$
- 2.
If $t_{k}^{\star}\geq\lambda_{k}$, we have
$\displaystyle\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t,S_{k}^{-}(p)+t\}dt=
\int_{0}^{\lambda_{k}}(S_{k}^{-}(p)+t)dt=S_{k}^{-}(p)\lambda_{k}+\frac{1}{2}
\lambda_{k}^{2}$
- 3.
If $0<t_{k}^{\star}<\lambda_{k}$, we have
$\displaystyle\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t,S_{k}^{-}(p)+t\}dt$
$\displaystyle=\int_{0}^{t_{k}^{\star}}(S_{k}^{-}(p)+t)dt+\int_{t_{k}^{\star}}^
{\lambda_{k}}(S_{k}^{+}(p)-t)dt$
$\displaystyle=S_{k}^{-}(p)t_{k}^{\star}+\frac{1}{2}(t_{k}^{\star})^{2}+S_{k}^{
+}(p)(\lambda_{k}-t_{k}^{\star})$
$\displaystyle\quad-\frac{1}{2}(\lambda_{k}^{2}-(t_{k}^{\star})^{2}).$

Putting these three cases together and summing over $k$ yields a piecewise-defined expression for the integral on each arc. Precisely, plugging these into the expression for $\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha}]$ in Eq. ([3](https://arxiv.org/html/2502.06574v2#A2.E3)), we finally obtain the closed-form

$$ $\displaystyle\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]=\frac{1}{2\pi} \sum_{k=1}^{2N}I_{k},$ $$

where $I_{k}$ denotes the value of the integral over the $k$-th arc, defined as

$$ $\displaystyle I_{k}:=\int_{0}^{\lambda_{k}}\min\{S_{k}^{+}(p)-t,S_{k}^{-}(p)+t \}dt,$ $$

and can be computed using the following case distinction

$$ $\displaystyle I_{k}=\begin{cases}S_{k}^{+}(p)\lambda_{k}-\frac{1}{2}\lambda_{k }^{2}&\text{if }t_{k}^{*}\leq 0,\\ S_{k}^{-}(p)\lambda_{k}+\frac{1}{2}\lambda_{k}^{2}&\text{if }t_{k}^{*}\geq \lambda_{k},\\ S_{k}^{-}(p)t_{k}^{*}+\frac{1}{2}(t_{k}^{*})^{2}+S_{k}^{+}(p)(\lambda_{k}-t_{k }^{*})-\frac{1}{2}(\lambda_{k}^{2}-(t_{k}^{*})^{2})&\text{if }0<t_{k}^{*}< \lambda_{k},\end{cases}$ $$

with $t_{k}^{*}=\frac{S_{k}^{+}(p)-S_{k}^{-}(p)}{2}$ as previously defined.

###### Remark B.5 (Computational cost) .

The closed-form expression for $\mathbb{E}[\rho_{p}]$ can be computed in $\mathcal{O}(n^{2}\log n)$ time. First, computing the $2N=\mathcal{O}(n^{2})$ cut angles defined by all unordered pairs of points $(z_{i},z_{j})$ requires $\mathcal{O}(n^{2})$ time, since each involves a simple trigonometric operation in $\mathbb{R}^{2}$. Sorting these angles to define the arc intervals costs $\mathcal{O}(n^{2}\log n)$. Once sorted, the distances $S_{k}^{+}(p)$ and $S_{k}^{-}(p)$ to the $p$-th next and previous cuts can be computed efficiently for all $k$ using sliding windows indexing in $\mathcal{O}(n^{2})$ time. The final step, i.e., evaluating the integral over each of the $2N$ arcs, also takes $\mathcal{O}(n^{2})$ time. Thus, the total computational cost is $\mathcal{O}(n^{2}\log n)$, dominated by the sorting step.

###### Remark B.6 (Case $K>2$ ) .

The above closed-form derivation relies on the fact that utilities correspond to angles on $\mathcal{S}^{1}$ (i.e., $K=2$). When $K>2$, utilities lie on the higher-dimensional sphere $\mathcal{S}^{K-1}$. In that setting, one can still define $\rho_{p}(\bar{\alpha})$ as the minimal geodesic distance on $\mathcal{S}^{K-1}$ to incur $p$ swaps, but the integral $\mathbb{E}_{\bar{\alpha}\sim\mathrm{Unif}(\mathcal{S}^{K-1})}\bigl{[}\rho_{p}(
\bar{\alpha})\bigr{]}$
admits no simple closed-form expression. In practice, one must approximate it numerically by Monte Carlo sampling. Specifically, for any unit vector $\bar{\alpha}\in\mathcal{S}^{K-1}$, each pair $(i,j)$ defines a *cut* great–sphere

$$ $\displaystyle H_{ij}=\{\beta\in\mathcal{S}^{K-1}:\langle\beta,v_{ij}\rangle=0 \}\quad v_{ij}=\psi(z_{i})-\psi(z_{j}).$ $$

The shortest geodesic distance from $\bar{\alpha}$ to that cut is given in closed form by

$$ $\displaystyle d_{ij}(\bar{\alpha})=\arcsin{\bigl{|}\langle\bar{\alpha},v_{ij}/ \|v_{ij}\|\rangle\bigr{|}}.$ $$

Thus, by getting all $N=\binom{n}{2}$ distances $\{d_{ij}\}$, sorting them, and picking the $p$-th smallest, we obtain $\rho_{p}(\bar{\alpha})$. Repeating for many independent $\bar{\alpha}\sim\mathrm{Unif}(\mathcal{S}^{K-1})$ gives a Monte Carlo estimate of the average.

###### Remark B.5 (Computational cost) .

###### Remark B.6 (Case K > 2 𝐾 2 K>2 italic_K > 2 ) .

### B.5 Maximum average p 𝑝 p italic_p -swaps distance occurs under collinearity of the spatial signature

This section provides the theoretical justification for the claim made in Section [3](https://arxiv.org/html/2502.06574v2#S3) that the average distance $\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]$ is maximized when the spatial signature is collinear, and equals $\pi/4$ in this case.

Recall that the spatial signature is the set of embedded vectors

$$ $\displaystyle\mathcal{S}_{\omega,\mathcal{D}}=\{\psi_{\omega,\mathcal{D}}(z) \in\mathbb{R}^{K}:z\in\mathcal{D}\},$ $$

where $\psi_{\omega,\mathcal{D}}(z)$ reflects the contribution of each data point $z\in\mathcal{D}$ to a family of $K$ base utilities, weighted by the semivalue coefficients $\omega$. This embedding allows utility directions $\bar{\alpha}\in\mathcal{S}^{K-1}$ to induce rankings via projection. The quantity $\rho_{p}(\bar{\alpha})$ measures the minimal geodesic distance on the sphere $\mathcal{S}^{K-1}$ one must rotate $\bar{\alpha}$ before the ranking changes by $p$ pairwise swaps. We focus here on the case $K=2$, where utility directions lie on the unit circle $\mathcal{S}^{1}$, and show that the maximum of $\mathbb{E}[\rho_{p}]$ is achieved when all embedded points lie on a common line through the origin. This derivation provides an upper bound for $\mathbb{E}[\rho_{p}]$ and motivates the normalization in the robustness metric $R_{p}$ (see Definition [3.2](https://arxiv.org/html/2502.06574v2#S3.Thmtheorem2)).

Each pair of points $(z_{i},z_{j})$ induces a *cut* on the circle $\mathcal{S}^{1}$, namely the two antipodal points where $\langle\alpha,\psi_{\omega,\mathcal{D}}(z_{i})-\psi_{\omega,\mathcal{D}}(z_{j}
)\rangle=0$. When all embedded points $\psi_{\omega,\mathcal{D}}(z_{i})$ lie on a single line through the origin, Corollary [B.4](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem4) states that there is exactly one cut (of multiplicity $N=\binom{n}{2}$) which splits $\mathcal{S}^{1}$ into two open arcs, each of length $\pi$. Within either arc, no swaps occur until one crosses that cut, at which point all $N$ pairs flip simultaneously. Concretely, for any direction angle $\theta\in[0,\pi[$, $\rho_{p}(\theta)$ corresponds to the shortest angular distance to this cut, either clockwise or counterclockwise, and is thus given by

$$ $\displaystyle\rho_{p}(\theta)=\min\{\theta,\pi-\theta\},$ $$

Hence,

$$ $\displaystyle\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]=\frac{1}{2\pi} \int_{0}^{2\pi}\rho_{p}(\varphi)d\varphi=\frac{1}{2\pi}\cdot 2\int_{0}^{\pi} \min\{\theta,\pi-\theta\}d\theta=\frac{\pi}{4}.$ $$

Now, any deviation from perfect collinearity introduces distinct cuts, which can only further subdivide those two $\pi$-length arcs into shorter pieces. Shorter maximal arc-lengths imply a smaller average distance to the nearest swap, so for every spatial signature and every $1\leq p<N$,

$$ $\displaystyle\mathbb{E}[\rho_{p}]\leq\frac{\pi}{4},$ $$

with equality if and only if the signature is exactly collinear.

###### Remark B.7 (Case $K>2$ ) .

For $K>2$, perfect collinearity of the spatial signature still maximizes the average $p$-swap distance, but the value $\displaystyle\max_{\mathcal{S}_{\omega,\mathcal{D}}}\mathbb{E}_{\bar{\alpha}}[
\rho_{p}(\bar{\alpha})]$ must be evaluated numerically since for $K>2$ the distribution of angular distances from a uniformly random point on the sphere to a fixed great sub-sphere no longer admits a simple elementary integral like for $K=2$.

###### Remark B.8 (Lower bounds for $\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]$ ) .

Trivially, since $\rho_{p}(\bar{\alpha})\geq 0$ for all $\bar{\alpha}$,

$$ $\displaystyle\mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})]\geq 0,\quad \forall p<\binom{n}{2}.$ $$

If instead we assume the spatial signature to be such that all $N=\binom{n}{2}$ cuts on $\mathcal{S}^{1}$ are distinct, then Proposition [B.3](https://arxiv.org/html/2502.06574v2#A2.Thmtheorem3) states that there are exactly $2N$ positive-length arcs of total length $2\pi$. In this case, it is easy to see that considering all ways to choose $\{\lambda_{k}\}$ summing to $2\pi$, the configuration $\lambda_{k}=\pi/N$ for all $k$ minimizes the average $\rho_{p}$. Concretely, for $\lambda_{k}=\pi/N$ we find

$$ $\displaystyle\mathbb{E}[\rho_{p}]=\frac{1}{2\pi}\sum_{k=1}^{2N}\int_{0}^{\pi/N }\Bigl{(}p\frac{\pi}{N}-t\Bigr{)}dt=(p-\tfrac{1}{2})\frac{\pi}{N}.$ $$

Hence, under the distinct cuts assumption,

$$ $\displaystyle\mathbb{E}[\rho_{p}(\varphi)]\geq\Bigl{(}p-\tfrac{1}{2}\Bigr{)} \frac{\pi}{N},$ $$

with equality exactly when the $2N$ cuts are perfectly equally spaced.

In this special setting, where all $\binom{n}{2}$ cuts on $\mathcal{S}^{1}$ are distinct, one could alternatively define the robustness metric as

$$ $\displaystyle R_{p}(S_{\omega,\mathcal{D}})=\frac{\mathbb{E}_{\bar{\alpha}}[ \rho_{p}(\bar{\alpha})]-(p-\frac{1}{2})\frac{\pi}{N}}{\pi/4-(p-\frac{1}{2}) \frac{\pi}{N}}\in[0,1].$ $$

However, since in practice we cannot detect a priori that this condition on cuts holds, we instead use the general robustness metric $R_{p}$
as given in Definition [3.2](https://arxiv.org/html/2502.06574v2#S3.Thmtheorem2).

###### Remark B.7 (Case K > 2 𝐾 2 K>2 italic_K > 2 ) .

###### Remark B.8 (Lower bounds for 𝔼 α ¯ ⁢ [ ρ p ⁢ ( α ¯ ) ] subscript 𝔼 ¯ 𝛼 delimited-[] subscript 𝜌 𝑝 ¯ 𝛼 \mathbb{E}_{\bar{\alpha}}[\rho_{p}(\bar{\alpha})] blackboard_E start_POSTSUBSCRIPT over¯ start_ARG italic_α end_ARG end_POSTSUBSCRIPT [ italic_ρ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( over¯ start_ARG italic_α end_ARG ) ] ) .

### B.6 Why can the Kendall rank correlation degrade by more than 2 ⁢ p N 2 𝑝 𝑁 \frac{2p}{N} divide start_ARG 2 italic_p end_ARG start_ARG italic_N end_ARG if ties occur?

This section supports the statement made in Section [3](https://arxiv.org/html/2502.06574v2#S3) that performing
$p$ pairwise swaps among ranked data points can result in a degradation of the Kendall rank correlation $\tau$ that exceeds $2p/N$ in magnitude, where $N=\binom{n}{2}$ is the total number of item pairs. While the robustness score $R_{p}$ is a well-defined geometric quantity that remains meaningful even in the presence of ties, the ambiguity lies in translating it into a change in Kendall rank correlation. The statement that performing $p$ pairwise swaps leads to a degradation of $\tau$ equal to $-2p/N$, which links $R_{p}$ to a drop in Kendall correlation, only holds in the idealized setting where no ties occur. In practical scenarios involving ties, the degradation in $\tau$ can be either smaller or larger than what $R_{p}$ would suggest. The purpose of this subsection is to explain why *worse-than-expected degradation* is possible, which is the main risk when interpreting $R_{p}$ through the lens of Kendall correlation in practice.

The Kendall rank correlation between rankings $A$ and $B$ is defined as

$$ $\displaystyle\tau=\frac{c-d}{\sqrt{\bigl{(}N-t_{A}\bigr{)}\,\bigl{(}N-t_{B} \bigr{)}}},$ $$

where $c$ is the number of concordant pairs, $d$ is the number of discordant pairs ($c$ and $d$ count only untied pairs), $N=\binom{n}{2}$ is the total number of pairs, $t_{A}$ (resp. $t_{B}$) is the number of tied pairs in ranking $A$ (resp. $B$).

Performing $p$ pairwise swaps among tied items can amplify the degradation of $\tau$ beyond the idealized $-2p/N$ amount (derived under the no-ties assumption) due to two effects:

- –
Resolving ties i.e., decreasing $t_{A}$ or $t_{B}$, increases the factors $N-t_{A}$ or $N-t_{B}$ and thus the denominator. For a fixed numerator $c-d$, this directly reduces the magnitude of $\tau_{b}$. Critically, even as $c-d$ decreases (due to increased discordance), the growing denominator further exacerbates the decline.
- –
Swapping two items within a block of $k$ tied points can order up to $\binom{k}{2}$ formerly tied pairs at once. If these newly ordered pairs are discordant, a single swap increases $d$ by up to $\binom{k}{2}$, rather than just 1.

Consequently, when many tied groups exist, one might observe after $p$ swaps,

$$ $\displaystyle\Delta\tau_{b}<-\frac{2p}{N},$ $$

i.e., a larger drop in rank correlation than in the no-ties case.

### B.7 Analytic derivation of the correlation decomposition in Section 4

In Section [4](https://arxiv.org/html/2502.06574v2#S4), we assert that the Pearson correlation between the two semivalue score vectors

$$ $\displaystyle\phi(\lambda)=\bigl{(}\phi(z_{1};\omega,\lambda),\dots,\phi(z_{n} ;\omega,\lambda)\bigr{)},\quad\phi(\gamma)=\bigl{(}\phi(z_{1};\omega,\gamma), \dots,\phi(z_{n};\omega,\gamma)\bigr{)}$ $$

admits the following decomposition

$$ $\displaystyle\operatorname{Corr}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}= \sum_{j=1}^{n}\omega_{j}^{2}r_{j},$ $$

where each term

$$ $\displaystyle r_{j}\propto\operatorname{Corr}\bigl{(}\Delta_{j}(\lambda), \Delta_{j}(\gamma)\bigr{)}\sqrt{\operatorname{Var}(\Delta_{j}(\lambda))\, \operatorname{Var}(\Delta_{j}(\gamma))}$ $$

captures the effective alignment of marginal contributions at coalition size $j-1$.

This decomposition holds under the mild assumption that, for all $j\neq k$, the empirical covariance between the marginal contribution vectors $\Delta_{j}(\lambda)$ and $\Delta_{k}(\gamma)$ is zero:

$$ $\displaystyle\operatorname{Cov}\bigl{(}\Delta_{j}(\lambda),\Delta_{k}(\gamma) \bigr{)}=0.$ $$

We now provide the detailed derivation of this result. See Section [C.5](https://arxiv.org/html/2502.06574v2#A3.SS5) for the definitions of the operators $\operatorname{Var}$, $\operatorname{Cov}$, and $\operatorname{Corr}$ in this context.

Recall that for any utility $u$,

$$ $\displaystyle\phi(z_{i};\omega,u)=\sum_{j=1}^{n}\omega_{j}\;\Delta_{j}\bigl{(} z_{i},u\bigr{)},$ $$

where
$\Delta_{j}(z_{i},u)$ is the marginal contribution of $z_{i}$ with respect to coalitions of size $j-1$.

By definition of the covariance,

$$ $\displaystyle\operatorname{Cov}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}= \frac{1}{n}\sum_{i=1}^{n}\bigl{(}\phi(z_{i};\omega,\lambda)-\bar{\phi}(\lambda )\bigr{)}\bigl{(}\phi(z_{i};\omega,\gamma)-\bar{\phi}(\gamma)\bigr{)},$ $$

where $\bar{\phi}(\cdot)$ denotes the mean over $i$. Using billinearity of covariance, we get

$$ $\displaystyle\operatorname{Cov}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}=\sum _{j=1}^{n}\sum_{k=1}^{n}\omega_{j}\omega_{k}\,\operatorname{Cov}\bigl{(}\Delta _{j}(\lambda),\,\Delta_{k}(\gamma)\bigr{)}.$ $$

where

$$ $\displaystyle\Delta_{j}(\lambda)=\Bigl{(}\Delta_{j}(z_{1};\omega,\lambda), \dots,\Delta_{j}(z_{n};\omega,\lambda)\Bigr{)}\quad\text{and}\quad\Delta_{j}( \gamma)=\Bigl{(}\Delta_{j}(z_{1};\omega,\gamma),\dots,\Delta_{j}(z_{n};\omega, \gamma)\Bigr{)}$ $$

Under the assumption
$\operatorname{Cov}(\Delta_{j}(\lambda),\Delta_{k}(\gamma))=0$
for all $j\neq k$, only the $j=k$ terms remain, giving

$$ $\displaystyle\operatorname{Cov}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}=\sum _{j=1}^{n}\omega_{j}^{2}\operatorname{Cov}\bigl{(}\Delta_{j}(\lambda),\Delta_{ j}(\gamma)\bigr{)}.$ $$

Similarly,

$$ $\displaystyle\operatorname{Var}\bigl{(}\phi(\lambda)\bigr{)}=\sum_{j=1}^{n} \omega_{j}^{2}\operatorname{Var}\bigl{(}\Delta_{j}(\lambda)\bigr{)},\quad \operatorname{Var}\bigl{(}\phi(\gamma)\bigr{)}=\sum_{j=1}^{n}\omega_{j}^{2} \operatorname{Var}\bigl{(}\Delta_{j}(\gamma)\bigr{)}.$ $$

By the definition of Pearson correlation,

$$ $\displaystyle\operatorname{Corr}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}= \frac{\displaystyle\sum_{j=1}^{n}\omega_{j}^{2}\,\operatorname{Cov}(\Delta_{j} (\lambda),\Delta_{j}(\gamma))}{\sqrt{\displaystyle\sum_{j=1}^{n}\omega_{j}^{2} \,\operatorname{Var}(\Delta_{j}(\lambda))}\sqrt{\displaystyle\sum_{j=1}^{n} \omega_{j}^{2}\,\operatorname{Var}(\Delta_{j}(\gamma))}}.$ $$

with $\operatorname{Cov}(\Delta_{j}(\lambda),\Delta_{j}(\gamma))=\operatorname{Corr}
(\Delta_{j}(\lambda),\Delta_{j}(\gamma))\sqrt{\operatorname{Var}(\Delta_{j}(
\lambda))\operatorname{Var}(\Delta_{j}(\gamma)}$
Then, the correlation becomes

$$ $\displaystyle\operatorname{Corr}\bigl{(}\phi(\lambda),\phi(\gamma)\bigr{)}= \displaystyle\sum_{j=1}^{n}\omega_{j}^{2}r_{j}$ $$

with

$$ $\displaystyle r_{j}=\frac{\displaystyle\operatorname{Corr}(\Delta_{j}(\lambda) ,\Delta_{j}(\gamma))\sqrt{\operatorname{Var}(\Delta_{j}(\lambda))\operatorname {Var}(\Delta_{j}(\gamma))}}{\sqrt{\displaystyle\sum_{j=1}^{n}\omega_{j}^{2}\, \operatorname{Var}(\Delta_{j}(\lambda))}\sqrt{\displaystyle\sum_{j=1}^{n} \omega_{j}^{2}\,\operatorname{Var}(\Delta_{j}(\gamma))}}\propto\displaystyle \operatorname{Corr}(\Delta_{j}(\lambda),\Delta_{j}(\gamma))\sqrt{\operatorname {Var}(\Delta_{j}(\lambda))\operatorname{Var}(\Delta_{j}(\gamma))}$ $$

as claimed.

Each term $r_{j}\propto\operatorname{Corr}\bigl{(}\Delta_{j}(\lambda),\Delta_{j}(\gamma)
\bigr{)}\sqrt{\operatorname{Var}\bigl{(}\Delta_{j}(\lambda)\bigr{)}
\operatorname{Var}\bigl{(}\Delta_{j}(\gamma)\bigr{)}}$ can be understood as the *effective alignment* of marginal contributions at coalition size $j-1$ across the two utilities $\lambda$ and $\gamma$. Specifically, $\operatorname{Corr}(\Delta_{j}(\lambda),\Delta_{j}(\gamma))$ measures how similarly data points’ marginal contributions at size $j-1$ move under $\lambda$ compared to under $\gamma$ and $\sqrt{\operatorname{Var}(\Delta_{j}(\lambda))\operatorname{Var}(\Delta_{j}(
\gamma))}$ down-weights sizes $j-1$ where marginal contributions are nearly constant (and thus uninformative) for either utility.

## Appendix C Additional definitions

For the reader’s convenience, we first outline the main points covered in this section.

- –
Appendix [C.1](https://arxiv.org/html/2502.06574v2#A3.SS1): Some definitions of linear fractional utilities.
- –
Appendix [C.2](https://arxiv.org/html/2502.06574v2#A3.SS2): Rank correlation metrics (Kendall & Spearman).
- –
Appendix [C.3](https://arxiv.org/html/2502.06574v2#A3.SS3): Axioms satisfied by semivalues.
- –
Appendix [C.4](https://arxiv.org/html/2502.06574v2#A3.SS4): Applications of semivalue-based data valuation methods.
- –
Appendix [C.5](https://arxiv.org/html/2502.06574v2#A3.SS5): Additional definitions for the correlation decomposition in Section [4](https://arxiv.org/html/2502.06574v2#S4).

### C.1 Some definitions of linear fractional utilities

Below, we give the concrete coefficients $(c_{0},c_{1},c_{2})$ and $(d_{0},d_{1},d_{2})$ for several commonly used linear-fractional performance metrics. Each of these metrics can be expressed in the form

$$ $\displaystyle u(S)=\frac{c_{0}+c_{1}\lambda(S)+c_{2}\gamma(S)}{d_{0}+d_{1} \lambda(S)+d_{2}\gamma(S)}.$ $$

as recalled from Eq.([2](https://arxiv.org/html/2502.06574v2#S3.E2)).

**Table 2: Some examples of *linear fractional* utilities. For more examples, see [5]. We set $\pi=\frac{1}{m}\sum_{j=1}^{m}\mathbf{1}[y_{j}=1]$, the proportion of positive labels in $\mathcal{D}_{\text{test}}$.**
| Utility | $(c_{0},c_{1},c_{2})$ | $(d_{0},d_{1},d_{2})$ |
| --- | --- | --- |
| Accuracy | $\bigl{(}1-\pi,2,-1\bigr{)}$ | $(1,0,,0)$ |
| Fβ-score | $\bigl{(}0,1+\beta^{2},0\bigr{)}$ | $\bigl{(}\beta^{2}\pi,0,1\bigr{)}$ |
| Jaccard | $(0,1,0)$ | $(\pi,\,-1,\,1)$ |
| AM-measure | $\bigl{(}\frac{1}{2},\frac{2}{\pi}+\frac{2}{1-\pi},-\frac{2}{1\pi}\bigr{)}$ | $(1,0,0)$ |

### C.2 Rank correlation metrics (Kendall & Spearman)

Let $X=(x_{1},x_{2},\dots,x_{n})$ and $Y=(y_{1},\dots,y_{n})$ be two real-valued score vectors on the same $n$ items. And let $\pi_{X}$ and $\pi_{Y}$ be their induced rankings. Rank correlations measure monotonic relationships between relative ordering $\pi_{X}$ and $\pi_{Y}$.

###### Definition C.1 (Kendall rank correlation) .

Define the set of all pairs of distinct indices $\mathcal{P}=\bigl{\{}(i,j):1\leq i<j\leq n\bigr{\}}$. For each $(i,j)\in\mathcal{P}$, call the pair *concordant* if $(x_{i}-x_{j})(y_{i}-y_{j})>0$, *discordant* if $(x_{i}-x_{j})(y_{i}-y_{j})<0$, and a *tie* in $X$ (resp. $Y$) if $x_{i}=x_{j}$ (resp. $y_{i}=y_{j}$).

Let $c$ the number of concordant pairs, $d$ the number of discordant pairs, and $t_{X}$ (resp. $t_{Y}$) the number of ties in $X$ (resp. $Y$). Then, the Kendall rank correlation $\tau$ is

$$ $\displaystyle\tau=\frac{c-d}{\sqrt{\bigl{[}\binom{n}{2}-t_{X}\bigr{]}\bigl{[} \binom{n}{2}-t_{Y}\bigr{]}}},$ $$

which simplify to $\tau=\frac{c-d}{\binom{n}{2}}$ if there are no ties ($t_{X}=t_{Y}=0$).

###### Definition C.2 (Spearman rank correlation) .

Let $\pi_{X}(i)$ be the rank of $x_{i}$ in $X$ and likewise $\pi_{Y}(i)$ for $Y$. Define the rank-differences $d_{i}=\pi_{X}(i)-\pi_{Y}(i)$. The Spearman rank correlation $s$ is the Pearson correlation of the ranked vectors:

$$ $\displaystyle s=\frac{\displaystyle\sum_{i=1}^{n}(\pi_{X}(i)-\bar{\pi}_{X})\,( \pi_{Y}(i)-\bar{\pi}_{Y})}{\sqrt{\sum_{i=1}^{n}(\pi_{X}(i)-\bar{\pi}_{X})^{2}} \,\sqrt{\sum_{i=1}^{n}(\pi_{Y}(i)-\bar{\pi}_{Y})^{2}}}$ $$

where $\bar{\pi}_{X}=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\pi_{X}(i)$ and $\bar{\pi}_{Y}=\frac{1}{n}\displaystyle\sum_{i=1}^{n}\pi_{Y}(i)$.
If there are no ties, it simplifies to

$$ $\displaystyle s=1-\frac{6\sum_{i=1}^{n}d_{i}^{2}}{n(n^{2}-1)}.$ $$

Both metrics lie in $[-1,1]$, with $+1$ indicating perfect agreement and $-1$ perfect reversal.

###### Definition C.1 (Kendall rank correlation) .

###### Definition C.2 (Spearman rank correlation) .

### C.3 Axioms satisfied by semivalues

Semivalues as defined in ([1](https://arxiv.org/html/2502.06574v2#S2.E1)) satisfy fundamental axioms that ensure desirable properties in data valuation. We formally state these axioms in the following.
Let $\phi(.,\omega;.)$ be a semivalue-based data valuation method defined by a weight vector $\omega$ and let $u$ and $v$ be utility functions. Then, $\phi$ satisfies the following axioms:

- 1.
*Dummy*. If $u(S\cup\{z_{i}\})=u(S)+c$ for all $S\subseteq\mathcal{D}\backslash\{z_{i}\}$ and some $c\in\mathbb{R}$, then $\phi(z_{i};\omega,u)=c$.
- 2.
*Symmetry*. If $u(S\cup\{z_{j}\})=u(S\cup\{z_{j}\})$ for all $S\subseteq\mathcal{D}\backslash\{z_{i},z_{j}\}$, then $\phi(z_{i};\omega,u)=\phi(z_{j};\omega,u)$.
- 3.
*Linearity*. For any $\alpha_{1},\alpha_{2}\in\mathbb{R}$, $\phi(z_{i};\omega,\alpha_{1}u+\alpha_{2}v)=\alpha_{1}\phi(z_{i};\omega,u)+
\alpha_{2}\phi(z_{i};\omega,v)$.

While all semivalues satisfy the above axioms, Data Shapley uniquely also guarantees *efficiency*: $\sum_{z\in\mathcal{D}}\phi(z,\omega,u)=u(\mathcal{D})$.

### C.4 Applications of semivalue-based data valuation methods

In practice, semivalue-based methods are mostly applied to perform data cleaning or data subset selection [25, 21, 4, 31]. Both tasks involve ranking data points according to their assigned values.

#### Data cleaning.

Data cleaning aims to improve dataset quality by identifying and removing noisy or low-quality data points. Since semivalue-based methods quantify each point’s contribution to a downstream task, low-valued points are natural candidates for removal. Specifically, a common approach is to remove points that fall into the set $\mathcal{N}_{\tau}$, defined as the subset of data points with the lowest values [9]. Formally, $\mathcal{N}_{\tau}=\{z_{i}\in\mathcal{D}\mid\phi(z_{i};u,\omega)\leq\tau\}$, where $\tau$ is a threshold determined through domain knowledge or empirical evaluation.

#### Data subset selection.

Data subset selection involves choosing the optimal training set from available samples to maximize final model performance. Since semivalues measure data quality, prioritizing data points with the highest values is a natural approach. Consequently, a common practice in the literature [27, 12, 28] is selecting, given a size budget $k$, the subset $\mathcal{S}^{(k)}_{\phi(u,\omega)}$ of data points with top-$k$ data values, i.e., $\mathcal{S}^{(k)}_{\phi(u,\omega)}=\arg\max_{\mathcal{S}\subseteq\mathcal{D},|
\mathcal{S}|=k}\sum_{z_{i}\in\mathcal{S}}\phi(z_{i};u,\omega)$.

### C.5 Additional definitions for the correlation decomposition in Section 4

For a vector $v=(v_{1},\dots,v_{n})$, the variance operator $\operatorname{Var}$ is defined as

$$ $\displaystyle\operatorname{Var}(v)=\frac{1}{n}\sum_{i=1}^{n}(v_{i}-\bar{v})^{2}$ $$

and for two vectors $v,w$, the covariance operator is defined as

$$ $\displaystyle\operatorname{Cov}(v,w)=\frac{1}{n}\sum_{i}(v_{i}-\bar{v})(w_{i}- \bar{w}).$ $$

This naturally gives,

$$ $\displaystyle\operatorname{Corr}(v,w)=\frac{\frac{1}{n}\sum_{i}(v_{i}-\bar{v}) (w_{i}-\bar{w})}{\sqrt{\operatorname{Var}(v)\operatorname{Var}(w)}}.$ $$

## Appendix D Additional figures

In Section [3](https://arxiv.org/html/2502.06574v2#S3), we plot the spatial signatures for the wind dataset (Figure [1](https://arxiv.org/html/2502.06574v2#S3.F1)) to illustrate the geometric mapping at the heart of our framework. Figures [5](https://arxiv.org/html/2502.06574v2#A4.F5), [6](https://arxiv.org/html/2502.06574v2#A4.F6), [7](https://arxiv.org/html/2502.06574v2#A4.F7), [8](https://arxiv.org/html/2502.06574v2#A4.F8), [9](https://arxiv.org/html/2502.06574v2#A4.F9), [10](https://arxiv.org/html/2502.06574v2#A4.F10) and [11](https://arxiv.org/html/2502.06574v2#A4.F11) present the analogous plots for all remaining datasets used in experiments (which are introduced in Table [5](https://arxiv.org/html/2502.06574v2#A1.T5)).

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/breast_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/titanic_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/credit_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/heart_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/cpu_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/2dplanes_shapley.png

Figure: (a) Shapley
Refer to caption: extracted/6471790/figures/spatial_signatures/pol_shapley.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 2
questions: 1. I’d be interested in the authors' perspective on whether the robustness metric reflects an inherent property of the data valuation framework, or if it is more influenced by the dataset and the choice of utility function. For example, in imbalanced binary classification tasks, the rankings generated by different utility functions (e.g., F1 score vs. accuracy) can vary substantially. How does this variation impact the stability of data rankings and the robustness metric? And if it is affected by dataset and utility functions, then in what setting do the authors advice using this robustness metric in practice ?

2. How do the authors propose choosing p (especially as a factor of the size of the training dataset) for a given dataset?
soundness: 3
strengths: 1. The introduction of a novel geometric perspective on and a formal robustness metric for semi-value based data valuation offers a way to benchmark data valuation frameworks and test their stability. In real-world datasets, the utility function can change anytime, reinforcing the importance of this metric. 
2.  The authors provide a rigorous evaluation of their metric on existing data valuation methods and utility functions. 
3. The paper also offers insights into why some valuation methods tend to collinearize the spatial signature and in doing so achieve higher robustness to utility shifts. 
4. The robustness metric proposed is shown to be computationally efficient via the closed-form expression for the expected minimal geodesic distance needed to induce p pairwise swaps in ranking.
summary: This paper provides an in-depth study of stability of semi-value based data valuation methods when there are shifts in utility. The authors  introduce the concept of spatial signature, where a training data point is projected on to a low-dimensional space. The authors assess the robustness of semi-value using a geometric representation of the utility space, specifically a unit circle, such that the metric quantifies how much utility direction must be rotated before the ranking undergoes a significant change. The authors use a geodesic angular distance over the unit circle, and the robustness metric Rp is measured as normalized minimal geodesic distance of rotation needed to achieve p swaps in ranking. The paper offers an evaluation of their metric against data valuation methods Data Shapley, Data Banzhaf and Beta Shapley.
weaknesses: 1. The paper focuses on utility functions specific to binary classification and raises the question of whether the findings will translate to other utility functions / learning tasks. Similarly the evaluations focus on two utility systems, and multi-utility systems are yet to be explored.
2. Benchmarking the stability of data valuation methods is not new. The key contribution of the paper is the geometric perspective - and this paper would be strengthened by additional experiments that show the true value of this geometric nature of the stability problem. While this paper discusses collinearity of spatial signatures leading to better robustness, the experimental validations focus on the robustness metric and not the geometric properties (eg. how the spatial signatures are distributed for a given valuation framework / dataset or the practical impact of this collinearity in geometric structure).

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Currently, empirical studies of multiple-valid-utility systems are mainly conducted in binary classification. It is also recommended to conduct multiple-valid comparisons of "multiple equally-valid indicators" in multi-class classification, rather than just trade-offs (e.g., macro-F1 vs macro-Recall vs Accuracy). 2. While $R_p$ provides a theoretically elegant proxy for stability under utility shifts, the paper would be significantly strengthened by connecting $R_p$ to downstream selection stability. It is recommended to report the relationship between top-k overlap@k/Jaccard@k and $R_p$.
soundness: 3
strengths: The paper is clearly written and easy to follow.

It presents an elegant and intuitive geometric representation that translates variations in utility functions into simple linear projections.

The proposed method is validated across a wide range of datasets and semivalues (Shapley, Beta Shapley, and Banzhaf).

The authors have thoroughly addressed the major limitations of the previous version by adding regression and multiclass tasks, introducing a formal robustness metric, and conducting experiments to assess algorithmic modifications.
summary: This paper investigates how sensitive semivalue-based data valuation scores are to the practitioner’s choice of utility function. It introduces the dataset spatial signature, an embedding in ℝ² where any candidate utility function corresponds to a linear functional. In this space, ranking data points with a semivalue reduces to projecting the dataset signature onto a direction that represents the chosen utility.
weaknesses: I have reviewed this paper before, and the authors have resolved most of my confusion, but I still have a remaining question.

The experiments on multiple-valid-utility scenarios remain somewhat limited and would benefit from further expansion.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: How does embedding the data point to a low dimensional space affects its value and ranking?

Is there a specific reason why authors typically restrict the use of regression problems to utility trade-off scenarios, rather than including them in multiple-utility analyses? 

Why do Tables 9–14 only consider p=500?

How distinct are the observed patterns with the robustness metric from those obtained using Kendall or Spearman correlations? In what ways is the robustness metric a great addition to the data valuation practitioner’s toolbox?

In what ways do dataset characteristics, data preprocessing choices, and algorithmic choices, e.g., chosen classification/regression algorithms, affect the results? 

Why restrict the multiple-valid-utility scenario to binary classification problems?
soundness: 3
strengths: Authors present another perspective on how the choice of the utility affects the resulting data values and rankings during data valuation.  

To mitigate the noise and randomness introduced by Monte Carlo sampling, they propose the use of aligned sampling.

The theory and empirical setups are well-written and easy to follow. 

Authors conduct multiple experiments to assess the efficacy of their metric, and also make their code available.
summary: The authors propose a unified geometric framework based on the spatial signature and also the robustness metric, designed to help data valuation practitioners assess the sensitivity of semivalue-based data valuation results to the choice of utility in both the multiple-valid utility and utility trade-off settings. They evaluate the effectiveness of the proposed metric through experiments on multiple real-world datasets, considering both the multiple-valid-utility and utility trade-off scenarios.
weaknesses: Restricting the utility to a 2D space spanned by two fixed base utilities u1 and u2 (and extension to class-wise utilities ) makes the robustness metric hard to scale.

It's possible that some information is lost when embedding data to a low dimensional space. It's unclear to me how this affects the results.

The data values from Monte Carlo approximations in regression problems tend to be unreliable due to greater variability in utility estimates when sample sizes are small. So, is there a specific reason why authors restricted the use of regression problems to utility trade-off scenarios, rather than including them in multiple-utility analyses?

There appears to be a misalignment between the claims made in the introduction and the empirical setup. In particular, the treatment of utility trade-offs in the experiments does not accurately reflect the notion of trade-off presented in the introduction.

There are very few results presented for the utility trade-off scenario, with none in the main paper and few in the appendix. Also, the results relegated to the appendix are neither discussed nor explained. For instance, why do Tables 9–14 only consider p=500? And why are only regression tasks used?

While the robustness metric could, in principle, address the question “How robust are my data valuation results to the choice of utility?”, it is unclear how distinct the observed patterns are from those obtained using Kendall or Spearman correlations. For instance, based on the results for both Kendall and the robustness metric, the cases where Banzhaf outperforms Shapley appear identical. The authors note that, in the absence of ties, the robustness metric “captures how far, in expectation, one must move from a utility direction before the Kendall rank correlation degrades by at least 2p/N.” While this is insightful, the results presented in the tables and figures reveal only broad trends, which seem largely similar across both metrics.



The authors briefly mention in the appendix that “one can alter the utility either by changing the algorithm A or by changing the performance metric,” an important consideration also discussed in several of the cited works. In my opinion, this is a strong consideration, and seeing how the robustness metric performs across datasets would have been helpful. However, the results presented in Table 15 are reported using Spearman correlation rather than the proposed robustness metric. Also, these results are not discussed, and key details of the experimental setup are not given, such as the number of runs. 



The authors should consider adding a discussion section. Overall, the experimental results are insufficiently discussed, making it difficult to highlight the main contributions of the proposed metric, both theoretical and empirical.



Minor.
The references section needs a second look. Cite the published versions of the papers where they exist and not their arXivs. E.g., Jiachen T. Wang, Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia. Rethinking data shapley
for data selection tasks: Misleads and merits, ICML’24


Decision: Accept (Poster)