2D-OOB: Attributing Data Contribution Through Joint Valuation Framework
Authors: Yifan Sun
Jingyan Shen
Yongchan Kwon
Venue: NeurIPS 2024 poster (2024)
Keywords: Data valuation
Cell-level attribution
Outlier detection
Primary area: interpretability_and_explainability
TL;DR: We introduce a joint valuation framework that attributes data contribution through assigning the valuation scores at the cell level.

Abstract: Data valuation has emerged as a powerful framework for quantifying each datum's contribution to the training of a machine learning model. However, it is crucial to recognize that the quality of cells within a single data point can vary greatly in practice. For example, even in the case of an abnormal data point, not all cells are necessarily noisy. The single scalar score assigned by existing data valuation methods blurs the distinction between noisy and clean cells of a data point, making it challenging to interpret the data values. In this paper, we propose 2D-OOB, an out-of-bag estimation framework for jointly determining helpful (or detrimental) samples as well as the particular cells that drive them. Our comprehensive experiments demonstrate that 2D-OOB achieves state-of-the-art performance across multiple use cases while being exponentially faster. Specifically, 2D-OOB shows promising results in detecting and rectifying fine-grained outliers at the cell level, and localizing backdoor triggers in data poisoning attacks.

2D-00B: Attributing Data Contribution Through Joint Valuation Framework
Yifan Sun
Jingyan Shen'
University of Illinois Chhana-Champaign yifan506illinois.co edu
Columbia University
js55448columbia edu
Yongchan Kwon
Columbia University 1.:301@ccolumia.. edd
Abstract
Data valuation has emerged as powerful framework for quantifying each datum's s contribution to the training of a machine learning model. However, it is crucial to recognize that the quality of cells within a single data point can vary greatly in practice. For example, even in the case of an abnormal data point, not all cells are necessarily noisy. The single scalar score assigned by existing data valuation methods blurs the distinction between noisy and clean cells of a data point, making it challenging interpret the data values. In this paper, we propose 2D-00B, out-of-bag estimation framework for jointly determining helpful (or detrimental) samples as well the particular cells that drive them. Our comprehensive periments demonstrate that 2D-00B achieves state-of-the-art performance across multiple use cases while being exponentially faster. Specifically, 2D-00B shows promising results in detecting and rectifying fine-grained outliers at the cell level, and localizing backdoor triggers in data poisoning attacks.
1 Introduction
From customer behavior prediction and medical image analysis to autonomous driving and policy making, machine learning (ML) systems process ever increasing amounts data. In such data-rich regimes, fraction of the samples is often noisy, incorrect annotations are likely to occur, and uniform data quality standards become difficult enforce. To address these challenges, data valuation emerges as research field receiving increasing attention, focusing properly assessing the contribution of These methods have proven useful in identifying low-quality samples that can be detrimental to model performance, as well as selecting subsets of data that are representative of enhanced model performance [52] 30 49 Furthermore, they are widely applicable
each datum ML training
13 26
in data marketplace for fair revenue allocation and incentive design 555
43 5
Nevertheless, existing data valuation methods assign scalar score to each datum, thereby failing to account for the varied roles of individual cells. This leaves the valuation rationale unclear and can be unsatisfactory and sub-optimal various practical scenarios. Firstly, whenever a score is assigned to a data point by particular data valuation method, it is crucial to understand the underlying justifications to ensure transparency and reliability, especially in high-stakes decision making [42]. Secondly. it is important to recognize the fact that even if a data point is of low quality, it is rarely the The absence of detailed insights
case that all the cells within this data point are noisy 40 2g 46
"Equal contribution. CCorreppondigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Age
Income
Experience
Education
Age
Income
Experience
Education
25
50000
3
4
25
50000
3
34
62000
10
6
34
62000
10
45
-1
20
8
45
20
29
35000
5
5
29
35000
5
40
80000
15
100
40
80000
15
100
low valuation
(a) Data valuation
(b) J oint valuation
Figure 1: Comparison of data valuation and joint valuation. (a) Data valuation evaluates the quality of individual data points, whereas (b) joint valuation evaluates the quality of individual cells. Both panels illustrate the same hypothetical dataset, while darker colors indicate higher quality or importance. As illustrated panel (a), data valuation can only identify that the third and fifth data points are of low quality, but it lacks further feature-level attribution. This limitation may result in discarding the entire data point, even when only certain cells are problematic. In contrast, joint valuation provides finer level of attribution than data valuation and aims to reveal how individual features contribute to data values. As shown in panel (b), the joint valuation framework can identify outlier cells (highlighted by blue boxes), such as -1 in "Income" and 100 in "Education", providing detailed interpretations of data values.
into how individual cells contribute to ML training inevitably leads discarding the entire data point. This can result in substantial data waste, particularly when only few cells are noisy and data acquisition is expensive. Finally, data markets, different cells within a data point may originate from different data sellers [T. Consequently, singular valuation for the entire point fails to offer equitable compensation to all contributing parties.
Our contributions In this paper, we propose DD-O0B, a powerful and efficient joint valuation framework that can attribute a data point's value to features. 2D-00B quantifies the importance of each cell in dataset, as illustrated in Figure [ providing interpretable insights into which cells are associated with influential data points. Our method is computationally efficient as well as theoretically
supported by its connections with Data-00B (26]. Moreover, our extensive empirical experiments demonstrate the practical effectiveness of 2D-00B in various use cases. 2D-00B accurately identifies cell outliers and pinpoints which cells to fix to improve model performance. Additionally, 2D-00B enables inspection of data poisoning attacks by precisely localizing the backdoor trigger, an artifact inserted into training sample to induce malicious model behavior [14] 66. On average, 2D-00B is 200 times faster than state-of-the-an methods across all datasets examined.
Preliminaries
Throughout this paper, focus supervised learning settings. For d N, we denote an input space and an output space by X' C IRd and y C R, respectively. We denote training dataset with data points D {1x,, yi) }:)} } where (ara, v:) the i-th pair of the input covariates x; X and its output label yı E y. For an event A, an indicator function 1 (A) ) 1 if A is true, otherwise 0. For j E N. we set |S].
Notations
(1, (1,....]) j.. For set S, we denote its power set by 2 and its cardinality by
DataShapley The primary goal of data valuation is to quantify the contribution of individual data points to model's performance. Leveraging the Shapley value in cooperative game theory [41]. DataShapley [13] measures the average change in utility function U 2n R over all possible
subsets of the dataset that either include or exclude the data point. For i E [n]. DataShapley of i-th datum defined as follows.
0
Shap
Σ ((,,(
(I)
S

---

where D. = (S C DI - DataShapley P; Shap in considers every set S S and computes the average difference in utility USSU ((x,, 9:) U(). It characterizes the impact of data point, but its computation requires evaluating U for all possible subsets of D, rendering precise calculations infeasible. Many efficient computation algorithms have been proposed
[6 28 54 and in these studies, Shapley-based methods have demonstrated better effectiveness in detecting low-quality samples than standard attribution approaches, such as lcave-one-out and influence function methods [22 10
Data-OOB As an alternative efficient data valuation method, Kwon and Zou 26 propose Data-00B, which leverages bagging model and measures the similarity between the nominal label and weak learners' predictions. To be more specific, consider a bagging model consisting of B weak learners, where for E [B), the b-th weak learner hn is given as a minimizer of the weighted empirical risk,
hss == argmin, Σ
who wrallyr h(xx)),
i=1
where é: 3 J + Rs loss function and Whs E N is the number of times the i-th datum (I,,, Mi) selected by the b-th bootstrap dataset. Let wa be a weight vector wa := (Wh1..... Wbn) for all b E [B]. For i E [n]] and {(w,, hn)]
b=1 Data-00B of the i-th datum is defined as follows.
Σe (w (3, ho())))
8 OOB
(2)
b== (w6r 0)
where T(,: hz()))) is a score function evaluated at (æ,, v1). We assume that the higher T, the better the prediction. In classification settings, common choice for T is 1 = h=:)))) and in this case, Data-00B g00B measures the average similarity between a nominal label yi and weak learners' predictions ho(2,) when a datum (I,, y4) is not sampled in a bootstrap dataset. It intuitively captures the quality of a data point. For instance, when (Ix,, Us) a mislabeled sample or an outlier, the label y: is likely to differ from h2 (2;). resulting in dOOB being close to zero.
It is noteworthy that Data-00B in ( can be computed by training a single bagging model, making it computationally efficient. Kwon and Zou (26) show that Data-008 -OBB can easily scale to millions of data points, but for DataShapley this is often very impractical. In addition, Data-00B is typically comparable to or even more effective than DataShapley in detecting mislabeled data points and selecting helpful data points (26 17].
3 Attributing Data Contribution through Joint Valuation Framework
Data valuation quantifies the utility of data points, however, it fails to identify which features contribute to these data values and to what extent. For instance, in anomaly detection tasks, data valuation methods can be deployed detect anomalous data points but cannot explain why they are considered abnormal, which is generally not desirable in practice. To address this challenge, we introduce a joint valuation framework that assigns cell score each feature of data point. Here, cell score is designed to quantify how each feature affects the value of an individual data point, thereby attributing the data value to specific features.
To the best of the author's 's knowledge, Liu et al. (32) were the first to consider the concept of joint val- uation in the literature, proposing 2D-Shapley ey as means quantitatively interpret DataShapley To formalize this, we denote a 2D utility function by u [n] x [d] - IR, which takes as input a subset of data points S [n] and subset of features F C [d], measuring the utility of fragment of the given dataset consisting of cells {i..j) JiesJeF. where tuple (i.j) denotes a cell at the i-th datum
and the j-th column. Then, Shaalee is defined
n
2D Shap
Σ
(3)
nd
(E)
(L.))
where ((i.)) == ((S. (4)) - 1} and = UU
3

---

The function M allows us to quantify how much removing a specific cell at (ž.j) from given set (SU (1), F (j)) affects the overall utility, and 2D 2D-Shapley vi 2happ evaluates the average Maa
across all possible data fragments (S, F) E D (i.))
k.
Similar to DataShapley, the permutation of all rows and columns required for exact 2D DD-Shapley calculations presents significant computational challenges. To address this, Liu et al. [2] develop 2D-KNN, which utilizes k- k-ncarest-neighbor models surrogates to approximate 2D-Shapley val- ues. However, the approximation methods can compromise the accuracy of valuations 26 17 Additionally, 2D-KNN still faces challenges scaling to large-scale datasets and high-dimensional settings.
To address these limitations, we propose 2D-00B, an efficient and model-agnostic joint valuation framework that leverages out-of-bag estimation attribute data contribution. We also illustrate how 2D-00B is connected to Data-00B, thereby facilitating sample-wise feature-level interpretation for data valuation, discussed in Section 3.2
3.1 2D-OOB: efficient joint valuation framework
Our idea builds upon the subset bagging model [55] which is well recognized as an earlier version of Breiman's 's random forest model [4) A key distinction from a standard bagging model is that weak learner
subset bagging model is trained on a randomly selected subset of features. For b E [B]. we denote the b-th random feature subset by Ss. S [d]. Then, the b-th weak learner of subset bagging model given as follows.
fo == argmin, Σ wurllyy. f(x.,s.))
where it.s, a subvector z; that only takes elements in subset Sg. This difference enables us to assess the impact of which features are more influential: if S₁ includes helpful (or detrimental) feature,
expect the out-of-bag prediction fxx,,s,) to be good (or poor). We formalize this intuition and propose 2D-00B, For i E [n]. j E [d] and {(wb, Sh, fo)] the 2D-00B for the j-th cell of the i-th data point is defined as follows,
ΣE b=1 1wwb 0,J E S.)T fo((..s))
OOB Vis
Σ (1hb 0,j 8.)
where T' y' x R is utility function that scores the performance of the weak learner fo(z,.s.) on the i-th datum (z,, 34). Specifically, for binary or multi-class classification problems, we can adopt T(y:; (xra,s))) Il (y fu(3r,,s)). ))- In this case, 2D-00B measures the average accuracy score of out-of-bag predictions (specifically, when the i-th data point is out-of-bag) if the j-th feature is used in training f.. For regression problems, we can use the negative squared error loss function, defined as Twy: (x(,ss,) -y: - fu(,.,)))) In practice, X' could also be incorporated into T to suit the specific use case.
While Data-00F 00B in 2 aims to assess the impact of the i-th datum, 2D-00B in (4) further provides interpretable insights by evaluating the data point with various combinations of features, revealing which cells are influential to model performance. By leveraging the subset bagging scheme, 2D-00B only requires a single training of the bagging model, making it computationally efficient.
3.2 Connection to Data-OOB
We now present interpretable expressions how 2D-00B connects to Data-00B the following proposition. To begin with, we denote a set of subsets of [d] by S = {S S [d]. With {(w,, fa) we define the i-th Data-00B when a particular subset S is used as follows and denote it by o, 50B (S).
OOB -== 1 (wb OyT fo(x.s))
Σ -= 1 (w 0)
Proposition 3.1. For all i e [n] and j [d]. 1i OOn can be expressed as follows.
2DB = Egg (0) OOB (S) 1j S].

---

where Fs is an empirical distribution with respect to S induced by the sampling process.
proof is given in Appendix C Proposition 3.1 shows that 2D-00B vij 2D- can be expressed as a conditional empirical expectation of UUB provided that the j-th feature is used in Data-00B computation. It provides intuitive interpretations: for fixed i and j / k; 1 - > Vik 2/2--00 implies that the cell zij is more helpful to achieve a higher OOB score than the cell zik- where the OOB score serves an indicator of model performance. By distinguishing the contributions of individual cells, 2D-00B effectively realizes joint valuation, providing a finer granularity of analysis that links feature-level importance individual data quality.
OOB
4 Experiments
In this section, we empirically show the effectiveness of 2D-00B across multiple use cases of the joint valuation: cell-level outlier detection, cell fexation, and backdoor trigger detection. To the best of our knowledge, the latter two use cases are introduced here for the first time within the joint valuation framework. As a summary, 2D-00B can precisely identify anomalous cells that should be prioritized for examination and subsequent fixation to improve model performance. In the context of backdoor trigger detection, 2D-00B demonstrates its efficacy by accurately identifying different types of triggers within poisoned data, showcasing its proficiency in detecting non-random, targeted anomalies. Our method also exhibits high computational efficiency through run-time comparison.
Throughout all of our experiments, 2D-00B uses subset bagging model with = 1000 decision trees. We randomly select a fixed ratio of features to build each decision tree. Unless otherwise specified, we utilize half of the features for each weak learner and set Ty, f(r,,s)) 1 (=r,s,)). The run time is measured on single Intel Xeon Gold 6226 2.9 GHz CPU processor. We provide a detailed ablation study of key hyperparameters in Section 4.4
4.1 Cell-Ievel outlier detection
Experimental setting In practical situations, even when dealing with abnormal data points, it not realistic settings, introduce noise to certain cells in the following two-step process: First, we randomly select 20% rows for each dataset. We then select 20% columns uniformly at random, allowing each selected row to have different set of perturbed cells. We inject noises sampled from the low-probability region into these cells, following Du et al. [9] and Liu et al. [32 Details the outlier injection process can be found in Appendix [A.3
always the case that all cells are noisy [40 32 23]. To simulate
We use 12 publicly accessible binary classification datasets from OpenML, encompassing range of both low and high-dimensional datasets, which have been widely used in the literature [13 25 26 Details on these datasets are presented in Appendix A. For each dataset, 1000 and 3000 data points are randomly sampled for training and test datasets, respectively. For the bascline method, we consider DNNKNN, fast and performant variant of 2D-22Dpppp Shapley (32). We incorporate distance regularization term in the utility function T for enhanced performance.
We calculate the valuations for each cell using our joint valuation framework. Ideally, the outlier cells should receive low valuation. We then arrange the cell valuations in ascending order and inspect those cells with the lowest values first
Results
The detection rate curve of inserted outlier is shown in Figure 2
For all datasets, 20-00B successfully identifies over 90% of the outlier cells by inspecting only 30% of the bottom cells. In comparison, BN- requires examining nearly 90% of the cells to achieve the same detection level.
We also evaluate the area under the curve (AUC) as quantitative metric and measure the run- run-time. As Table [ shows, 2D-00B achieves an average AUC of 0.83 across 12 datasets, compared to 0.67 for 2D-KNN, while being significantly faster. For high-dimensional datasets such as the musk dataset, which comprises 166 features, 2D-KNN would take more than hour to process, while 2D-00B can finish seconds. Furthermore, we present additional results on multi-class classification datasets in Appendix B.I demonstrating the consistently superior performance and efficiency of 2D-00B.
5

---

Dotaset: nomao
Dataset musk
Ostaset: Mini:coNE
Dataset: drft
Dataset: feddd
Dataset: anris
Deleciin Rate
Detechenn Rate.
1,00
1.00
Detection Rate
1.00
Detecrann Rate
6.75
9,75
0:75-
0:75
0.30
0.50
8:3
0.0
0.25
0.5
10-011
0.25
22-081
20-00
0.25
23
0.25-
1949
50
100
50
120
50
100
100
100
0
50
150
Inspection Percerkage
Inspection Peccentage
Inspection Percentage
Inspection Percentage
Inspection Peecentage
Inspection Parcertage
Datsset: pol
Datoret: lowrcholl
Dotaset: veticl zerelT
Dstsoet: electrichy
Dstaret: 2dplanes
Dataset creatteard
1.00
1.00
Dutocien Rate
0.75
Detecten Bate
0.75
Dateccton Wad
6.75
Dutection Rute
9.75
0.75
0.75
3.50-
0.50
0.50
0.25
025
0.25
0.25
27
0.23
0
50
100
50
100
50
100
10>
100
50 150
Inspection Narcenkage
Inapection Parcensage
Inspection Percentage
Inspection Parcentage
Irspection Parceriage
Inspection Parcertage
Figure 2: Cell-Jevel outlier detection rate curves for 2D-00B, 2D-KNN, and Random. The x-axis represents the pereentage of inspected cells. The y-axis represents the detection rate, defined as the ratio of the number of detected outlier cells to the total number of outlier cells present in dataset. The error bars show a 95% confidence interval based on 30 independent experiments. We examine closer the
the cells in ascending order, starting from those with the lowest values, and thus left-top corner indicates better performance. 2D-00B efficiently detects the majority of outlier cells by examining only small fraction of the total cells, while 2D-KNN and Random require scanning nearly all the cells.
Table 1: Cell-level outlier detection results. AUC and run-time comparison between 2D-00B and 2D-KNN across twelve binary classification datasets. The average and standard error of the AUC and run-time (in seconds) based on 30 independent experiments are denoted by "average + standard error". Bold numbers denote the best method. The AUC value for the Random method consistently remains at 0.5 across all datasets. In every dataset, 2D-00B achieves significantly higher AUC while being orders of magnitude faster than 2D-KNN.
AUC
Run-time |
Dataset
2D-00B (ours)
2D-KNN
2D-00B (ours)
2D-XMN
lawschool
0.88+ 0.0027
0.75+ 0.0011
3.33 + 0.06
177.56 + 1.92
electricity
0.77+ 0.0072
0.68 0.0014
3.39 + 0.07
191.38 t 2.60
fried
0.914 0.0015
0.61 0.0005
3.97 + 0.10
322.79 + 2.98
2dplanes
0.87+ 0.0015
0.62+ 0.0005
3,46 + 0.05
295.25 2.37
creditcard
0.72.1 0.0028
0.69 0.0011
4.56 + 0.10
662.34 7.12 7.12
0.8211 0.0014
0.671 0.0006
4.34 0.05
759.33 4.37
MiniBooNH
0.77+ 0.0058
0.63+ 0.0019
7.46 0.06
1507.83 14.50
jannis
0.83-t 0.0042
0.55 0.0004
7.98 0.07
1753.10 + 12.35
nomao
0.79.1 0.0021
0.67 0.0009
7.69
2564.58 + 23.11
vehicle sensiT
0.81+ 0.0014
0.61 0.0005
9.87 + 0.08
3113.65 + 24.54
gas gas_drift drift
0,86:1 0.0010
0.841 0.0017
11.28 0.10
3878.31
+ 40.72
musk
0.8811 0.0008
0.71 0.0006
14.09 1 0.11
4415.45 1 22.96
Average
0.83
0.67
6.78
1636.80
4.2 Cell fixation experiment
Experimental setting A naive strategy to handle cell-level outliers is to eliminate data points that contain outliers. This method, however, risks substantial data loss, particularly when outliers scattered and data points are costly to collect. We instead consider a cell fixation experiment, where we assume that the ground-truth annotations of outlier cells can be restored with external expert knowledge, At each step, we "fix" certain number of cells by substituting them with their ground-trui truth annotations, prioritizing cells that have the lowest valuations. Then we fit a logistic mode and evaluate the model's 's performance with test set of 3000 samples. It is important to note that correcting normal cells has no effect, whereas fixing outlier cells expected to enhance the model's performance. We adopt the same datasets and implementations in Section 4.
'Loiisicc regression is chosen because it is simple yet powerful machine learning model commonly used to test data separability, which is standard practice in this field [17 32].

---

Detwcet: nOmIO
2.002
Danasel muse
18.0%
6,4225
Dar seaC HnBrook
35:000
Detwoeet 016 4Hh
Darsset Red
20:00
Detacet sardd
0522
TeC Rscaccurg
2.223
2.22
Rundors
0.6665
Tec arrurraa
0-222-
233
lendur
0.734
Raidean
2.22
6:915-
6660
23:09
0,732
0012
0.2200
0.9650
0.71
8,413
Roodion perertage
Footion Sercentage
Fostion percentage
percetiage
Raxt on percertage
Fication percentage
Figure 3: Cell fixation experiment results (test accuracy curves) for 2D-00B, 2D-KNN, and Random. replace cells with their ground-truth annotations, starting with those cells assigned the lowest valuations. The results for 6 datasets are presented, and additional results for other datasets are available Appendix B.2 We conduct 30 independent trials and report the average results. A higher curve indicates better performance. 2D-00B demonstrates superior capability in accurately identifying and rectifying cell-level outliers.
ship vs. truck
deer dog
airplane automobile
frog horse
bird cat
1.0
1.0
1.0
1.0
1.0
Detection Rate
0.5
0.5-
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0
100
0
100
100
100
100
1.0
1.0
1.0
1.0
1.0
Detection Rate
0.5-
0.5-
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0
100
0
100
0
100
0
100
100
Inspection Percentage Inspection Percentage Inspection Percentage Inspection Percentage Inspection Percentagr
2D-008
2D-KNN
Random
Figure 4: Backdoor trigger detection rate curves for 2D-00B, 2D-KNN, and Random. Panels (top) and B (bottom) correspond to the Trojan square and BadNets square, respectively. We inspect the cells within each poisoned sample in descending order of their valuation scores. The detection rate curve shows the average detection rate across all poisoned samples, with error bars representing 95% confidence interval based on 15 independent runs. 2D-00B demonstrates superior performance in detecting the cells implanted with triggers.
Results Figure illustrates the anticipated trend in the performance of 2D-00B, validating our method's capability to accurately identify and prioritize the most impactful outliers for correction. As cells with the lowest valuations are progressively fixed, 2D-00B demonstrates a consistent improve- ment in model accuracy. In contrast, when applying the same procedure with 2D-KNN, such notable performance enhancements are not observed.
Additionally, we investigate a scenario where ground -round-truth truth annotations remain unavailable. We adopt the setup from Liu et al. [32], where we replace the outlier cells with the average of other cells in the same feature column. 2D-00B uniformly demonstrates significant superiority over its counterparts. Results are provided in Appendix B.2
4.3 Backdoor trigger detection
A common strategy of data poisoning attacks involves inserting predefined trigger (e.g., a specific pixel pattern in an image) into subset of the training data [14 31. These malicious manipulations can be challenging to detect as they only infect specific, targeted samples. Even when poisoned data are present, it could be difficult to discern the root cause of the attacks, since manually reviewing images is expensive and time-consuming. In this experiment, we introduce a novel task enabled by the joint valuation framework: localizing backdoor triggers data poisoning attacks. Distinet from the random outliers investigated previously, this type of cell contamination is targeted and deliberate.
7

---

Paisaned Image
Cell Valusaions
Paisoned Image
Call Valuations
Paisaned Image
Cell Valuaaions
Paisoned Image
Call Valuaaions
5.0
622 021 080
Figure 5: Qualitative examples for 2D-00B in the backdoor trigger detection task. Each pair of images consists of poisoned image and its corresponding cell valuation heatmap. The color of the heatmap indicates importance, with red cells representing higher importance and blue cells representing lower importance. In the first two pairs, the class 'bird" is relabeled as "cat", while in the latter two pairs, the class "deer" is relabeled as "cat". The heatmaps clearly show that higher cell valuations are predominantly concentrated in the regions containing triggers, while areas featuring actual objects receive lower valuations. This pattern suggests that 2D-00B effectively captures the triggers as the impactful features responsible for the misclassification of the poisoned samples.
We consider two popular backdoor attack algorithms: BadNets 14] and Trojan Attack [31]. During training, the poisoned samples, relabeled as the adversarial target class, are mixed up with clean data. As a result, the model learns to incorrectly associate the trigger with the target class. At test time, inputs containing the trigger are misclassified the target class. In this context, our goal is to effectively pinpoint the triggers by recognizing them influential features through our joint valuation framework.
Experimental setting We select 5 pairs of classes from CIFAR-10 24 For each pair, one class is designated as the target attack class, while the other serves as the source class. The training dataset comprises 1000 images. For each attack, we contaminate 15% of the training samples from the source class and relabel them to the target class. Two types of attack triggers are implemented: the Trojan square and the BadNets square [14] (3, 31. These triggers are placed in the lower right corner of the original images to minimize occlusion. Further details about these attacks are available in Appendix A.4 In our experiment, the ratio of poisoned cells is approximately 1 1%, and each weak learner the subset bagging model is built by sampling 25% the features.
Results We adopt the same detection scheme and baseline methods as in Section 4.1 Ideally, the poisoned cells should receive high valuation scores given that such data points have been relabeled. We present the detection rate curves for five datasets in Figure 4 2D-00B significantly outperforms 2D-KNN in detecting both types of triggers. Overall, 2D-00B achieves an average detection AUC of 0.95 across all datasets and attack types, compared to 0.83 for 2D-XNN. It is worth noting that conventional data valuation methods are fundamentally unable to localize backdoor triggers; at most, they can only identify poisoned data points.
Qualitative examples dissre[s the heatmaps for poisoned samples based on cell valuations of 2D-00B, Areas with higher cell valuations (marked in dark red) precisely indicate the trigger locations within these samples, demonstrating the effectiveness of our joint valuation framework.
Additional examples can be found in Appendix B.3
4.4 Ablation study
We conduct ablation studies on the cell-level outlier detection task, as outlined in Section 4.1 to examine the impact of key hyperparameters 2D-00B estimations, including the selection and number of weak learners, as well as the feature subset ratio.
Selection of weak learners Although our study primarily employs decision trees weak learners, it is important to note that 2D-00B is model-agnostic, enabling the use of any class of machine learning models as weak learners. Specifically, examine decision trees, logistic regression, single-layer MLP with 64 units, and a two-layer MLP with 64 and 32 units, respectively, as weak learners compare their performance.
Table presents a comparison of detection AUC across 12 datasets with different choices of weak learners, indicating that 2D-00B is not model-free. The selection of weak leamners slightly affects the

---

Table 2: Ablation study results of weak learner types. The average and standard error of the detection AUC based on 30 independent experiments are denoted by "average = standard error". Results from 2D-KNN are included for comparison. The choice of weak learner leads to variations in cell values, yet the performance of the detection task remains robust.
Dataset
Decision Tree
Logistic Regression
MLP (single-lly))))
MLP (two-layer)
2D-KNN (Baselinc)
lawschool
0.88 + 0.0027
0.81 0.0014
0.83 0.0023
0.86 + 0.0049
0.75 0.001
electricity
0.77 + 0.0072
0.75 + 0.0029
0.75 0.0039
0.74 + 0.0064
0.68 = 0.0014
fried
0,91 + 0.0015
0.82 + 0.0023
0.85 = 0.0020
0.88 + 0.0027
0.61 + 0.0005
2dplanes
0.87 + 0.0015
0.82 0.0026
0.86 0.0026
0.88 0.0037
0.62 0.0005
creditcard
0.72 1: 0.0028
0.74 0.0023
0.74 0.0026
0.74 0.0071
0.69 0.0011
pol
0.82 1: 0.0014
0.79 0.0029
0.85 0.0014
0.86 0.0019
0.67 0.0006
MiniBooNE
0.77 + 0.0058
0.77 + 0.0059
0.80 0.0057
0.81 +: 0,0119
0.63 + 0.0019
jannis
0.83 +: 0.0042
0.76 + 0.0040
0.79 0.0048
0.80 +: 0.0108
0.55 + 0.0004
nomao
0.79 1: 0.0021
0.82 1 0.0012
0.83 0.0010
0.83 1 0.0017
0.67 1 0.0009
vehicle-sensLT
0.81 + 0.0014
0.81 0.0026
0.80 0.0025
0.82 0.0037
0.61 0.0005
gas-drift
0.86 + 0.0010
0.89 0.0005
0.88 0.0005
0.88 0.0006
0.84 0.0017
musk
0.88 + 0.0008
0.87 + 0.0005
0.88 0.0005
0.88 + 0.0008
0.71 + 0.0006
Average
0.83
0.80
0.82
0.83
0.67
Table 3: Ablation study results of (a) the number of base learners and (b) the feature subset ratio K/d. The average and standard error of the detection AUC based on 30 independent runs denoted by "average + standard error." (a) Increasing the number of base learners from 1000 to 3000 does not yield notable performance improvement. (b) Our method's joint valuation capacity remains relatively stable regardless of the selected feature subset ratio.
(u) Ablation on the number of base learners B.
(b) Ablation on feature subset ratio K/d.
Dataset
B 500
A
B 1000
F 3000
Dataset
K/d= 0.25
K/d== = 0.50
AUC
K/d 0.75
lawschool
0.0035
0.0027
0.0026
lawschool
0.86:2 0.0026
0.88: 0.0027
0.88: 0.0024
elcetricity
0.77 :: 0.0062
0.77 0.0072
0.77 0.00P0
electricity
0.79:1 0.0070
0.771 0.0072
0.73 0.0070
fried
0.87 1 0.0022
0.91 1: 0.0015
0.91 = 0.0014
fried
0,86+ 0.0024
0.91+ 0.0015
0.89-
2dplanes
0.87 0.0016
0.87 0.0015
0.87 0.0015
2dplanes
0.82:: 0.0015
0.075
0.88: 0.0014
0.72 0.0025
0.72
0.72 0.0028
creditcard
0.73:1 0.0029
0.72.1 0.0028
0.713 0.0028
pol
0.78 + (100122
0.82 + 0.0014
(.82 + 0.0014
0.66-t 0.0031
0.82+ 0,0014
0.82: 0.0014
MiniBooNE
(.77 + 0.0044
0.77 + 0.0058
0.77 0.0058
MiniBooNE
0,78:t 0.0076
0.77:: 0.0058
0.77:: 0.0049
jonnis
0,0045
0.83
0.0039
jannis
0,84:1 0.0035
0,83.1 0.0042
0.82 0.0043
nomao
0.79 1: 0.0018
0.79 1: 0.0021
0.79 1: 0.0020
nomso
0.0019
0.0021
0.78:
vchicle
0.80 ::: 0..0021
0.81 : 0.0014
0.81 : 0.0014
vehicle_sensIT senssT
0.81:1 0,0014
0.8111 0.0014
0.80:1 0.0015
gas drift
0.86 0.0007
0.86 0.0010
0.86 0.0010
gas diftt
0,88: 0.0009
0,86 0.0010
0.86- 0.0009
musk
0.0008
0.0008
0.89: 0.0008
0.0008
0.88 0.0008
Avenae
0.81
0.83
0.83
Average
0.81
0.83
0.82
valuation results, with more complex models generally yielding better performance. Nonetheless, all variations of 2D-00B outperform 2D-KNN, highlighting the clear advantages of the 2D-00B approach.
Number of weak learners Increasing the number of weak learners allows for a greater number of data-feature subset pairs to be explored, potentially leading to more accurate estimates. However, as shown in Table 3al when we increase the number of base learners from 500 to 3000, the detection AUC for each dataset remains relatively unchanged, suggesting convergence of the estimation beyond a certain threshold. Typically, 1000 base learners are sufficient achieve an equitable joint valuation.
Feature subset ratio K/d The feature subset ratio K/d refers to the fraction of the total number of features d that are randomly selected to build each weak learner, where K is the number of selected features. In previous experiments, we used a fixed ratio of 0.50 (unless otherwise specified). To further investigate the impact of this ratio, we now test two additional values: 0.25 and 0.75. The results in Table suggest that in general, the joint valuation capacity of our method is robust to the choice of feature subset ratio.
Apart from the experiments discussed above, we showease that marginalization of 2D-00B can either match surpass oftte-of-the-ar data valuation methods on standard benchmarks in Appendix D

---

Related work
Data contribution estimation In addition to the marginal contribution- based methods discussed in Section 2 many other approaches are emerging in the area of data valuation. Just et al. [19] develop a non-conventional class-wise Wasserstein distance between the training and validation sets and use the gradient information to evaluate each data point, an approach that has also been applied to data selection [20]. Wu al. [5]] extend data valuation to deep neural networks, introducing training-free data valuation framework based on neural tangent kernel theory. Yoon et al. [52] leverage reinforcement learning techniques automatically learn data valuation scores by training regression model. However, all these data valuation methods do not assign importance scores to cells, whereas our method provides additional insights into how individual cells contribute to the data valuations.
Feature attribution Feature attribution is pivotal research domain explainable machine learning that primarily aims to provide insights into how individual features influence model predictions. Various effective methods have been proposed, including -HAP-based explanation [33 34 27 8 7
counterfactual explanation 44 18 39 35 36), and Dackpropagation-bat baseed explanation | 45
44 133.
Among these methods, the SHHPP-asee explanation stands out as the most widely adopted approach, utilizing cooperative game theory principles compute the Shapley value [41]. While feature attribution offers a potential method to attribute data valuation scores across individual cells, empirical experiments in Appendix [B. reveal that this two-stage scheme falls short in efficacy compared to our proposed joint valuation paradigm, which integrates data valuation and feature attribution in simultaneous process.
6 Conclusion
We propose 2D- 2D-00B, an efficient joint valuation framework that assigns score to each cell in dataset, thereby facilitating finer attribution of data contributions and enabling a deeper understanding of the dataset. Through comprehensive experiments, we show that 2D-00B is computationally efficient and competitive over state-of-the-art methods multiple joint valuation use cases.
Discussion We emphasize that the primary objective of the joint valuation framework is to evaluate the quality cells within the dataset, rather than to optimize model performance. The model used serves as a proxy for this evaluation, and it is important to note that high-performing machine learning method does not necessarily ensure a justified valuation framework.
Limitation and future work While our study primarily explores random forest models applied to tabular datasets and simple image datasets, the potential application of neural network models within the 2D-00B framework for more complex vision and language tasks presents promising avenue for future investigation. For instance, text datasets, tokens or words can be treated as cells. 2D-00B can be easily integrated into any bagging training scheme that uses language models.
Overall, we believe that our work will inspire further exploration in the field joint valuation, with the broader goal of improving the transparency and interpretability of machine learning, as well as developing an equitable incentive mechanism for data sharing.
Acknowledgement
We acknowledge computing resources from Columbia University's 's Shared Research Computing Facility project, which is supported by NIH Research Facility Improvement Grant IG20RR030893. 01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.
10

---

References
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods deep neural networks. arXiv preprint arXiv:1711.06104 2017.
[2] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non non-linear classifier decisions by fayer-wise relevance propagation. PloS one, 10(7)=003 2015.
[3] Jens Bleiholder and Felix Naumann. Data fusion. ACM computing surveys (CSUR). 41,1):1-41,
[4] Leo Breiman. Random forests. Machine learning. 45:5-32, 2001.
[5] Lingjiao Chen, Bilge Acun, Newsha Ardalani, Yifan Sun, Feiyang Kang, Hanrui Lyu, Yongchan Kwon, Ruoxi Jia, Carole-Jean Wu, Matei Zaharia, al. Data acquisition: new frontier in data-centric ai. arXiv preprint 1371223 2023.
[6] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 17:1:1712.. 2017.
[7] lan Covert and Su-Ia Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pages 3457-3465. PMLR, 2021.
[8] Ian Covert, Scott M Lundberg, and Su-In Lec. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33:17212- 17223, 2020.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197 2022.
[10] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems,
2020.
[11] Raul Castro Fernandez, Pranav Subramaniam, and Michael J Franklin. Data market platforms: Trading data assets to solve data problems. arXiv preprint 27Xiv2200.000047, 2020.
[12] Matthias Feurer, Jan N Van Rijn, Arlind Kadra, Picter Gijsbers, Necratyoy Mallik, Sahithya Ravi, Andreas Müller, Joaquin Vanschoren, and Frank Hutter. Opthon--ython: extensible python api for openml. The Journal Machine Learning Research, 22(1):45733457 2021.
[13] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMIRR 2019.
[14] Tianyu Gu, Brendan Davitt,,, and Siddharth Garg, Badnets: Identifying vulnerabilities in the machinc learning model supply chain. arXiv preprint 1rXiv:1708.0675: 2017.
[15] Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume I, pages 278-282. IEEF, 1995.
[16] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithmss. arXiv preprint /908: 1908. 08619, 2019.
[17] Kevin Fu Jiang, Weixin Liang, James Zou, and Yongchan Kwon. Opendataval: a unified henchmark for data valuation. arXiv preprint arXiv:2306.105: 10577. 10577,2023. 2023.
[18] Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijilbenjaronk, Been Kim, and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv: 1907.0961, 2019.
[19] Hoang Anh Just, Feiyang Kang. Jiachen T Wang. Yi Zeng. Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.0000 2023.
[20] Feiyang Kang, Hoang Aah Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Ku- mar Sahu, and Ruoxi Jia, Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.0m 027744, 2024.
[21] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. https://archive.. ics uci edu, 2017. The UCI Machine Learning Repository.

---

[22] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference muchine learning, pages 1885-1894. PMLR, 2017.
[23] Hans-Peter Kriegel, Peer Kröger, Erich Schubert, and Arthur Zimek. Outlier detection in axis-parallel subspaces of high dimensional data. In Advances in Knowledge Discovery and Data Mining: 13th Pacific- Asia Conference, PAKDD 2009 Bangkok, Thailand, April 27-30, 2009 Proceedings 13,, pages 831-838. Springer, 2009.
[24] Alex Krizhevsky, Geoffry Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110. 2021.
[26] Yongchan Kwon and James Zou. Data-OOB: Out-of bag estimate as simple and efficient data value. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedingy Machine Learning Research, pages 18135-18152. PMLR, 23-29 Jul 2023. URL https://proceecdings mlr, wwwpprreesss0. h20/////2222/2eeeee2 .ttml
[27] Yongchan Kwon and James Y Zou. Weightedshap: analyzing and improving shaplcy based feature attributions. Advances in Neural Information Processing Systems, 15:34333-33776 2022.
[28] Yongchan Kwon, Manuel A. Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 793-801. PMLR, 13-15 Apr 2021. URL httthttt:///pooeeediii kwon21a. html
mlr /vess/vv30//
[29] Andy Leung, Hongyang Zhang, and Ruben Zamar. Robust regression estimation and inference in the presence cellwise and casewise contamination. Computational Statistics & Data Analysis, 99:1-11, 2016.
[30] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4 (S):699-777) 2022.
[31] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and X. Zhang. Trojaning attack neural networks. In Network and Distributed System Security Symposium, 2018. URL http://pp...mmaticccchh....eee http:/accccooooaoooooooppsllll/a :ttg//oorpssD::110000 31806516
[32] Zhihong Liu, Hoang Anh Just, Xiangyu Chang. Xi Chen, and Ruoxi Jia. 2d-shapley: A framework for fragmented data valuation. arXiv preprint arXiv:2306. 10473, 2023.
[33] Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[34] Scott M Lundherg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint 180 03888, 2018
[35] Divyat Mahajan, Chenhao Tan, and Amit Sharma. Preserving causal constraints in counterfactual explana- tions for machine learning classifiers. arXiv preprint arXiv:1912.0321 1912.03277, 2019.
[36] Ramaravind Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountabiliry and transparency, pages 607-617. 2020.
[37] Maria-Irina Nicolac, Mathicu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018. URL Uttphttp::// //avi org/påf/1807
01069
[38] Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, and Ting Wang. Trojanzoo: Towards unified, holistic, and practical evaluation of ncural backdoors. Proceedings of IEEE European Symposium on Security and Privacy (Euro S&P), 2022.
[39] Rafael Poyiadzi, Kacper Sokol, Raul Santos- Rodriguez, Tijl De Bie, and Peter Flach. Face: feasible and actionable counterfactual explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 444-350, 2020.
12

---

[40] Peter J Rousseeuw and Wannes Van Den Bossche. Detecting deviating data cells. Technumetrics, 60(2): 135-145, 2018.
[41] Lloyd S Shapley et al. A value for n-person games. 1953.
[42] Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learn- "igg"iineeeiems"" stralegies, and open challenges. In Proc. IJCAI, 2022.
[43] Rachael Hwee Ling Sim, Yehong Zhang, Trong Nghia Hoang, Xinyi Xu, Bryan Kian Hsiang Low, and Patrick Jaillet. Incentives in private collaborative machine learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[44] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[45] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arX::::442..666 2014.
[46] Peng Su, Garth Tarr, and Samuel Muller. Robust variahle selection under cellwise contamination. Journal of Statistical Computation and Simulation, pages 1-17, 2023.
[47] Sandra Wachter, Brent Mittelstadt, and Chris Russell, Counterfactual explanations without opening the black box: Automated decisions and the gdpt. Harv. JL & Tech., 31:841, 2017.
[48] Jiachen T Wang, Prateek Mittal, and Ruoxi Jia. Efficient data shapley for weighted nearest neighbor algorithms. arXiv preprint arXiv:2401. 10 2024.
[49] Jiachen T Wang. Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia. Rethinking data shapley for data selection tasks: Misleads and merits. arXiv preprint trXiv:2405.03877 2024.
[50] Tianhao Wang and Ruoxi Jia. Data banzhaf: A data valuation framework with maximal robustness lcarning stochasticity. arXiv preprint arXiv:2205. 15466, 2022.
[51] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference Machine Learning, pages 24150-24176. PMIR, 2022.
[52] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842 10851. PMLR, 2020.
[53] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerlond, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014.
[54] Jaaya Zhang, Qiheng Sun, Jinfei Liu, Li Xiong, Jian Pei, and Kui Ren. Efficient sampling approaches to shapley value approximation. Proc. ACM Manag. Data, 1(1), may 2023. doi: 10.1 155/34888888. URI.
attps://áoi.... orggooggg.o 10. 1145/3588728
[55] Boxin Zhao, Boxiang Lyu, Raul Castro Fernandez, and Mladen Kolar. Addressing budget allocation and revenue allocation in data market environments using an adaptive sampling algorithm. arXiv preprint arXiv:2336.02554. 2023.
13

---

Supplementary Materials
In the supplementary materials, we provide implementation details, additional experimental results, rigorous formalized proofs and data valuation experiment results. Code repository can be found at Joint- vetttlalliin
A Implementation details
A.1 Datasets
Tabular datasets We use 12 binary classification datasets obtained from OpenML 12) A summary of all the datasets provided in Table These datasets are used in Section 4.1 Appendix
and
For each dataset, we first employ standard normalization procedure. where each feature normalized to have zero mean and unit standard deviation. After preprocessing, we randomly partition subset of the data into two non-overlapping sets: a training dataset and a test dataset, which consists of 1000 and 3000 samples respectively. The training dataset is used to obtain the joint (or marginal) valuation for each cell (or data point). The test dataset is exclusively used for the cell fixation (or point removal) experiment when evaluating the test accuracy. Note that for methods that need validation dataset such as KNNShapley and DataShapley, we additionally sample a separate validation dataset (disjoint from training dataset and test dataset) to evaluate the utility function. The size of the validation dataset is set to 10% of the training sample size.
Image datasets We create datasets by pairing CIFAR-10 classes, each pair consisting of target attack class and a source class. The training and test dataset comprises 1000 and 2000 samples, respectively. The size of the validation dataset is 10% of the training sample size. To manage the computational challenges posed by the baseline method, we employ the siper-pixel technique to transform the (32,32.3) image into 256-dimensional vector. Specifically, we first average the pixel values across the three channels for each pixel. Then, we partition these transformed images into equal-sized x grids. Average pooling is applied within each grid reduce pixel values single cell value, which is then arranged into flattened input vector. A cell is annotated as poisoned if at least 25% of its corresponding grid area contains the trigger.
A.2 Implementation details for different methods
2D-00B 2D-00B involves fitting subset random forest model with B 1000 decision trees based on the package "seikit-learn". When constructing each decision tree, we fix the feature subset size ratio 0.5 (unless otherwise specified). For Section
4. and Appendix D we simply adopt (y: = (xr,,s))) For Section 4.1 and 4.2 further calculate the normalized
T f(r,ss) negative L2 distance between covariates and the class-specitic mean in the bootstrap dataset, denoted as dnorm (m,, y:). Then we use T(v,, f (x..s.) (1: fxr,,s)) + dnorm (2i, V:).
KN-KNN 2D-KNN employs KNN as a surrogate model to approximate 2D- AD-Shapley. We set the number of nearest neighbors as 10 and the number of permutations 1000. The hyperparameters are selected based on convergence behavior, and the run time is measured until the values converge.
A.3 Implementation details for cell-level outlier generation
Following Du et al. [9] and Liu al. 32 replace given cell with outlier value. Here, the outlier value is randomly generated from the two-sided "tails" of the Gaussian distribution fitted to the column's 's mean and standard deviation. where the probability of the two-sided tail area is set be 1%. In total, 4% (20% x 20%) of the cells are replaced with the corresponding outlier values.
A.4 Implementation details for backdoor trigger generation
Following the prior work 14 we generate the BadNets square and the Trojan square trigger. For BadNets, we adopt the implementation in Nicolae et al. [37]. For Trojan Attack, use
14

---

pretrained ResNet-18 18 model on the CIFAR-10 dataset and employ the implementation in Pang et al. [8]. For each attack, evaluate its effectiveness by training a decision tree model on the poisoned dataset. The accuracy on clean test set remains nearly unchanged compared to model trained on an uncontaminated training set, while the attack success rate on a held-out poisoned test set guaranteed to exceed 75%.
Table 4: A summary of all the datasets used in 4.1 4.2, and Appendix been commonly used in previous literature [13 25 26
D These datasets have
Name
Total sample size
Input dimension
Majority class proportion
OpenML ID
lawschool
20800
0.679
43890
electricity
38474
0.500
44080
fried
40768
10
0.502
901
2dplanes
40768
0,501
727
creditcard
30000
23
0.779
42477
pol
15000
0.664
722
MiniBooNE
72998
50
0,500
43974
jannis
57580
54
0.500
43977
nomao
34465
89
0.715
1486
vehicle_seni sensIT
98528
100
0.500
357
gas_drift
5935
128
0,507
1476
musk
6598
166
0.846
1116
B Additional experimental results
B.1 Additional results for cell-level outlier detection
Additional results on multi-class classification datasets We conducted cell-level outlier detection experiments (as described in Section 4.1 on three multi-class classification datasets from the UCI Machine Learning repository 21 As shown in the Table 2D-00B displays superior detection performance and efficiency.
Table 5: Cell-level outlier detection results on multi-class classification datasets. The average and standard error of the AUC and run-time (in seconds) based 30 independent experiments denoted by "average + standard error".
AUC
Run-time 1
Dataset
2D-001 (ours)
2D-KNN
2D-00B (ours)
2D-XNN
Covertype
0.81=0.0116
0.63--0.0183
0.5874
962.34 1,3388
Dry Bean
0.88 0.0059 0.81=10.0178
0.851
3.31 0.4586
347 247.80--220012
Wine Quality
0.57+0.0252
2.90=0.1240
269.1411.1825
two-stage attribution Once we obtain the data valuation scores, an alternative solution approach determining cell-level attributions involves leveraging feature attribution methods such as SHAP 33 We explore an additional baseline method building upon this idea: initially, Data-00B (or any other data valuation method) is computed for the i-th data point, denoted as duy. Subsequently, TreeSHAP [34] is fitted. using du; as the target and the concatenation of and y, (denoted as I2 @ yı) as the predictor. The derived local feature attributions are then interpreted as joint valuation results, We refer to this method "two-stage attribution".
Additional baseline:
Table Tableaagindicats that 2D-00B substantially outperforms its two-stage counterpart. We hypothesize that the superiority of our method stems from integrating data valuation and feature attribution into cohesive framework. Conversely, attribution as separate processes, potentially resulting in sub-optimal outcomes. Furthermore, due to the computational complexity of TreeSHAP, the two-stage approach notably slower compared to method.
two-stage method treats data valuation and feature
15

---

Table 6: Cell-level outlier detection results (AUC) of 2D-00B and the two-stage attribution. Our method shows a better performance than the alternative method by significant performance margin.
AUC f
Dataset
2D-00B (ours)
Two-stage attributtin
lawschool
0.88 0.0027
0.831 0.0064
electricity
0.77- 0.0072
0.64+ 0.0093
fried
0.91 0.0015
0.82+ 0.0068
2dplancs
0.87= 0.0015
0.80+ 0.0058
creditcard
0.72.1 0.0028
0.67.1 0.0051
pol
0,82 0,0014
0.781 0.0042
MiniBooNE
0.77 0.0058
0.70 0.004|
jannis
0.83= 0.0042
0.62+ 0.0043
nomao
0.79- 0.0021
0.71+ 0.0041
vehicle sensiT
0.81 0.0014
0.641 0.0033
gas drift
0.86 0.0010
0.73+ 0.0143
musk
0.88 0.0008
0.68 0.0028
Average
0.83
0.72
Table 7: Cell-level outlier detection results (AUC) of different joint valuation methods when the row outlier ratio and column outlier ratio are both 50%. Our method consistently outperforms 2D-KNN even in the presence of significant noise.
AUC
Dataset
2D-00B (ours)
2D-KNN
lawschool
0.75+ 0.0084
0.60# 0.0144
electricity
0.64 0.0155
0.60 0.0106
fried
0.74+t 0.0087
0.54+ 0.0027
2dplanes
0.74+ 0.0063
0.55+ 0.0033
creditcard
0.63-t 0.0055
0.61 0.0053
pol
0.69-t 0.0069
0.60 0.0042
MiniBooNE
0.67-t 0.0128
0.60= 0.0048
jannis
0.70=t 0.0113
0.53= 0.0014
nomao
0.70-t 0.0088
0.58+ 0.0052
vehicle_scme sensiT
0.70-t 0.0075
0.55+ 0.0031
gas_drift
0.73-t 0.0077
0.65 0.0114 14
musk
0.77 0.0063
0.64 0.0038
Average
0.71
0.59
A noisy setting with more outlier cells We consider more challenging scenario with increased outlier levels, where both the row outlier ratio and column outlier ratio increase from 20% (as in Section 4.1) to 50%. Consequently, this leads to 25% (50% x 50%) of the cells being replaced with outlier values. We follow the same outlier generation procedure outlined in Appendix [A.3] The findings, presented in Table 7 demonstrate that method maintains significantly superior performance over 2D-KNN, even under such a noisy setting.
B.2 Additional results for cell fixation experiment
Figure 6 presents the results for the cell fixation experiment on 6 additional datasets. 2D-00B excels precisely detecting and correcting relevant cell outliers.
The scenario without ground-truth knowledge Following Liu al. [32], we examine a situation where externll information on the ground-truth annotations of outlier cells is not accessible. In this scenario, we address these outliers by substituting them with the average of other cells in same feature column. This procedure starts by addressing cells with the lowest valuations, based on the hypothesis that correcting these cells likely to maintain or potentially improve the model's performance. As depicted in Figure 2D-00B conforms this expected trend, demonstrating the
16

---

Dae pd
Dacaset bewehool
Datseet velidle cWneT
Detaver: elecricty
Darsset 2dhlenes
Datscet credbrand
0.092
3,65
10:000
21:00
692
Tee Rec.eeriiN
2.43
Rutton
Tee: accurruueyy
0:032
0:400
Rukdam
10:53
Teal Lec
(250
22:59
0.76
2.02
0.79
0.22
0.03
0.1000
6.764
6:827
Flodton pereerage
Foation bercentage
Fostion percentage
Reat percettage
Raxt on percentage
Faxtton percentage
Figure 6: Cell fixation experiment results (test accuracy curves) for 2D-00B, 2D-KNN and random baseline. We replace cell values with ground-truth values from the cells with the lowest valuation to the highest valuation. The results from 6 additional datasets are displayed. We conduct 30 independent trials and report the average results. A higher curve indicates better performance. 2D-00B sets itself apart by its remarkable precision in detecting and rectifying relevant cell outliers.
Datasen: musk
0.22
MiriBookE
dis.dii
Datasstt fried
Dotaset jane's
0.51
Teet accuaryy
6.77
0.70
20:05
DCB
25:008
0.69
0.00
3.84
22
22
10
10
23
10
20
20
percentage
pertettage
perceniage
percentage Dwaset eloctr
Dutaset: (peschoo
Datage:: veticle seaul
Doraset: 25pranes
Dataset: credicord
1.00
3
0.50
11:00
3,00
0.78
3,800
32
10
0.78
20 25
0,975
10
20
20
Fxotion percemtag:
Exztion porcettage
Reot on perzerlage
Exotion percenzage
Reation porcentage
Reation percentaage
Figure 7: Cell fixation experiment (without ground-truth knowledge) results (test accuracy curves) for 2D-00B, 2D-KNN and a random baseline. We replace cell values with column mean imputations from cells with the lowest value to the highest value. We report the average results of 30 independent trials from 12 datasets. A higher curve indicates better performance.
effectiveness of our method in joint valuation. Conversely, 2D-KNN fails show similar performance improvements.
B.3 Additional results for backdoor trigger experiment
We provide additional qualitative examples of the backdoor trigger detection experiment in Figure g
Poisoned Image
Cell Valuations
Poisoned Image Call Valuations
Poisaned Image
Cell Valuations
Poisoned Image Call Valuations
Figure 8: Qualitative results on more datasets for the backdoor trigger detection experiment. The first two images are from the class "airplane" but have been relabeled as "automobile", while the latter two images are from the class "frog" and have been relabeled as "horse".
C Proof of Proposition 3.1
Proof. For simplicity, we denote g00 as (S) and 2/2 2D-0BB as thi in the proof. Let S == {S C [d]} represent the set of all feature subsets, where S is feature subset. We denote the cardinality of S as := (S] 2d. Let be weight vector 7 := (7b1,..., Tot) for all b E [B]. where Tiu E {0, 1} and "/22 indicates the I-th subset used in the b-th weak learner. With
(wo, nb. fo we can denote the i-th DoB on the 1-th feature subset St as
B b-
11 1w ()M 1)T (x
(si))
Σb 1 (wn 0) 1)
17

---

With slight abuse of notation, the formulation of 2D-00B in 4 can be expressed as follows.
4/ Ei=1 Σ 12
1
(wy = 0)1(m) 1)1(j E ( f((o.s)))
Ei=1
Σ b=1 (wbl 0))((1: = 1] E Si)
B
1wwi 0)1(1m 1)T (o(().,)))
1(j E Si)
--
- 2x=1 1w 0)1 1)1(j E Si)
Ee= N(wbi 0)1(>>=1 =
Σ b= 1 = 0)1(1m = 1)T(Us. fo((s.s)))
Σ 1(j E Si)
D1=1 2d= 1wwbi = 000(1>111100000000000)00 = 1)1(j S.)
Σ [ww =0)1 = =)
where aij x 1(j E Si) B
1w 07H = 1), [m], E [d], E [L] and Σi= Dij. 1. Define (S E St, (wmi) )=) = Oijjd which specifies empirical distribution of the feature subset S, conditioned on j E S and the bootstrap sampling process. Here, [ E St) indicates that the distribution is conditioned on the inclusion of the j-th feature within the feature subset Sy. whi indicates whether the i-th sample is out-of-bag in the b-th bootstrap, and 7i indicates whether the I-th feature subset is selected in the b-th weak learner. Thus, the point mass is determined by the sampling process, leading to:
Viy Bpg (8)
D Data valuation experiment
In this section, show that 2D-00B-data, the marginalization of 2D-00B, offers effective approach to data valuation. This serves as the basis our enhanced performance in joint valuation.
Marginalization 2D-00B aims to attribute data contribution through cells. Consequently, by summing up 2D-00B over all columns, we can derive data contribution values. For E [m]]. we define the 2D-00B-aata wdaaa as follows.
d
data
OOB
(5)
j=1
Based on discussions in Section 3.2 the marginalizations also connect with Data-00B:
Proposition D.1. For all i [n]. the marginalizations wigtt can be expressed follows.
data
E R&g (0) (00B (S)]
where the notations follow the same definitions as Proposition
3.1
Proof. Based on the definition of 2D-00B-Data for i E |n].
1 d
1
vidato
2D
OOB
0
00B (OOB
d
1j
d
1=1
j=1
a ) OOB (Si),
1=1
where angit is defined in Appendix C We have Σts Σ )يري) = E o,j. 1.
Denote P:(Si) (ors (/ EE= Oigijl which induces the empirical expectation of Data-00B with respect to St.
18

---

Proposition D.I indicates the i-th data point. As result, AD--0B-dataa is expected inherit the advanced ability of Data-00F in terms of data valuation, as will be empirically examined next.
vit can be expressed as the average Data- 00B value for
Experimental setting Following the standard protocol in Kwon and Zou 25 26 and Jiang et al. T7]. we randomly select 10% of the data points and flip its label to the other class. For joint valuation methods, we calculate the valuation of each cell and perform the marginalization over features to obtain the data valuation scores. Mislabeled data detection and data removal experiments are examined based on this setting. For the baseline methods, we further incorporate several state- of-the-art data valuation methods including DataShapley [13], KNNShapley [16], DataBanzhaf [50], LAVA [9], and Data-00B [26]. Implementation details are listed below. To guarantee fair comparison, we also employ the decision tree as the base model in DataShapley and DataBanzhaf.
We adopt the same 12 datasets as outlined in Section 4.1
Data-00B Data-00B involves fitting a random forest model without feature subset sampling, consisting of 1000 decision trees.
DataShapley We use a Monte Carlo-based algorithm. The Gelman-Rubil Rubin statistics is computed to determine the termination criteria of the algorithm. Following Jiang et al. [17. We adopt the threshold be 1.05.
KNNShapley et al. [16].
We the number of nearest neighbors be 10% of the sample size following Jia
LAVA We calculate the class-wise Wasserstein distance following Just et al. 19 The "OTDD" framework is adopted complete the optimal transport calculation.
DataBanzhaf We adopt the implementation from Jiang al. 17 We set "the number of models to train" as 1000.
D.1 Mislabeled data detection
We calculate the precision-recall curve by comparing the actual annotations, which denote whether data points are mislabeled, against the data valuation scores computed by differrnnt methods. Misla- beled data typically have a detrimental impact on model performance. Therefore, data points that receive lower valuation score are considered to have higher chance of being mislabeled. We then determine AUCPR (the AUC of the precision-recall curve) quantitative metric to assess the detection efficacy.
As shown in Table [8 2D-00B-data consistently outperforms ID-KNN-date across all datasets, suggesting its superior ability detect mislabeled data points. It worth noting that 2D-00B-das -0ta's results are on par with Data-O0B, while significantly exceeding the performance of other data valuation methods. These results in line with our theoretical analysis regarding the resemblance between Data -0BB and AD-00B-date However, it is important to highlight that applying Data-00B the joint tasks is not feasible as mentioned earlier, underscoring the necessity for the development of 2D-00B.
D.2 Point removal experiment
Removing low-quality data points has the potential to enhance model performance. Based on this idea, we employ the point removal experiment, widely used benchmark data valuation (2 13 25). According to the calculated data valuation scores, we progressively remove data points from the dataset ascending order. Specifically, we begin by removing the data points with the lowest data valuations. Each time we remove datum, we fit a logistic model and use the held-out test set consisting of 3000 instances to evaluate the model performance. The expected behavior is that model performance will improve initially the detrimental data points are gradually eliminated from the training process. Removing an excessive number data points may result a drastically altered dataset. Consequently, we opt to remove the bottom 20% data points.
19

---

Table 8: Point-level mislabeled data detection results. AUCPR of different data valuation and (marginalized) joint valuation methods. The average and standard error of the AUCPR based on 30 independent experiments denoted by "average + standard error". Bold numbers denote the best method, for data valuation and joint valuation respectively. The AUCPR value for the Random method consistently remains 0.5 across all datasets. 2D-00B-data exhibits performance comparable to Data-QOB, while significantly surpassing 2D-KNN-data (the marginalization of 2D-KNN) and all other data valuation methods.
Data Valuation DataBanzhaf
Joint Valuation (Marginalized)
Dataset
KKNShapley
LAVA
DataShapley ay
Data-COB
2D-KKl-data
20-00B-data (ours)
lawschool
0.008
0.88 0.007
1,00
0.000
0.46 0.011
electricity
0.22 +: 0.008
0.11 # 0.002
0.18 = 0.005
0.26 0.007
0.44 + 0.007
0.20 = 0.006
0.39 :: 0.007
fried
0:40 0.014
0.11 + 0.002
0.22 0.007
0.35 0.009
0.76
0.007
0.34 0.010
0.73 0.008
2áplanes
0.46 1. 0.016
0,12 1. 0,002
0.32 0.007
0.54 0.009
0.78 0.008
0.44 0.011
0.68 0.010
creditcard
0.37 0.007
0,11 = 0.003
0.16 = 0.004
0.28 0.006
0.40 0.007
0,20 0.005
0.40 0.007
pol
0.017
MiniBooNE
0.41 1: 0.013
0.13 + 0,006
0.23 = 0.007
0.41 1 0.010
0.78 :: 0.007
0,36 = 0.008
0.78 1: 0.007
jannis
0.002
0.005
0.17 0.005
0.38
0.006
0.37
nomao
0.61 +:0.11
0,14 0.003
0.33 + 0.010
0.58 0.009
0.87 0.006
0.33 0.011
0.88 0.005
vehicle sensiT
0.221:0.000
0,11 1: 0.002
0.007
0.33 0.011
0.56 0.010
0.14 0.005
0.56 0.010
gus drill
0.07::0.0113
0,16 1: 0,006
0.42 1 0.009
0.75 0.008
0.98
0.002
0.88 0.006
0.98 0.002
musk
0.010
0,11 = 0.003
0.31 0.007
0.47 0.012
0.85
0.005
0,21 0,008
0.85 0,005
Average
0.41
0.12
0.28
0.47
0.73
0.34
0.71
Test accuracy curves throughout the data removal process are shown for 12 datasets (Figure A higher curve signifies better performance in terms of data valuation. Overall, 2D-00B-date demonstrates similar performance to Data-00B, while significantly outperforming all other data valuation methods and the random baseline. When few data points with poor quality are removed, the test performance of 2D-00B-data exhibits an evident increase. However, such a positive trend does apply other popular data valuation methods including DataShapley and LAVA. These findings highlight the potential of 2D-00B-date in selecting a subset of critical data points that can maintain model performance when the dataset is pruned.
20

---

Dataset: nomao
Dataset: musk
Dataset: MiniBooNE
0.92
0.830
0.92
0.825
0.91
0.91
Test accuracy
0.90
0.815
0.89
0.810
0.88
0.805
0.87
0.800
0.86
0.86
0.795
0.85
0
10
15
20
0
10
15
20
0
10
15
20
Removal percentage
Removal percentage
Removal percentage
Dataset: gas drift
Dataset: fried
Dataset: jannis
0.830
0.99
accuracy
0.98
Test accuracy
Test accuracy
0.820
0.97
0.815
0.96
0.805
0.95
0.800
0
10
15
20
10
15
20
0
10
15
20
Removal percentage Dataseti pol
Removal percentage
Removal percentage
Dataset: lawschool
Dataset: vehicle sensiT
1.00
0.98
0.83
Test accuracy
accuracy
0.96
accuracy
0.82
0.81
0.92
0.80
0.90
0.79
0.78
0.88
0.77
10
20
10
15
20
10
15
20
Removal percentage electricity
Removal percentage
Removal percentage Datasel: creditcard
Datasel: 2dplanes
0.834
0.8100
0.8075
0.730
Test accuracy
Test accuracy
0.830
Test accuracy
0.8050
0.725
0.8025
0.828
0.720
0.8000
0.715
0.826
0.7975
0.710
0.824
0.7950
0.822
0.7925
0.7900
10
20
0
10
15
20
10
15
20
Removal percentage
Removal percentage
Removal percentage
UAVA
Datallanchaf Data COB
2D-KNN-data
DataShapley
Random
KXNShapley
20-008-data
Figure 9: Point removal experiment results (test accuracy curves) of 7 data valuation methods- 2D-00B-data, 2D-KNN-data, Data-00B, LAVA, DataBanzhaf, DataShapley, KNNShapley and random baseline. We remove data points from the lowest valuation to the highest valuation. The results from 6 binary classification datasets are displayed. For each dataset, we conduct 30 independent trials and report the average results. A higher curve indicates better performance. 2D DD-O0B-data demonstrates superior ability in finding a set of helpful data points.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions have been clearly stated in the abstract and section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper has included discussion about limitations in section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

The paper discusses theoretical interpretation in section 3.2 and the proof has been included in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All implementation details have been included in section 4 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: The paper uses open-source datasets, detailed in Appendix A. and the code repository is included in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil wwwppbblccgguiicuceccccccoe.oopp /eegguddssseCeebbbinninn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment settings have been clearly stated in section 4 and Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars for all experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides information on compute resources in section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper fully adheres to the NeurIPS Code of Ethics as outlined in the provided guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Appendix A for dataset citations.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "hifZ8n2KXs", "reviewer": "Reviewer_6nYm", "summary": "This paper studies data valuation in a fine-grained fashion: determine helpful samples as well as the particular cells that drive them. The proposed method is tested with 12 datasets and has application in backdoor detection.", "questions": "Could you please discuss what are the *unique* challenges of 2D-OOB that do not arise for 2D-KNN and Data-OOB?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It makes sense to use cells rather than single data samples as the smallest unit for data valuation. \n- The proposed framework is clearly stated. \n- The backdoor trigger detection is an interesting and useful application of the proposed framework. \n- The proposed method is significantly more efficient than 2D-KNN.", "weaknesses": "- The proposed method seems to be a combination of existing algorithms 2D-KNN and Data-OOB. The novelty and contribution may be relatively marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness: The novelty and contribution may be relatively marginal.**\n\n**Re.:** Thank you for your feedback. We acknowledge that 2D-OOB may initially appear to be a natural extension of existing methods, as it is built on them. However, we believe our contribution is substantial in the field of data valuation, as our proposed method addresses both practical and conceptual challenges of 2D-Shapley and Data-OOB.\n\nCompared to 2D-Shapely, our work significantly improves **computational efficiency**, as demonstrated by the empirical experiments in the submitted paper and the new results in our general response. In addition, we identify **new** use cases of the joint valuation framework, such as attributing data contribution through cell values, correcting erroneous cells, and detecting backdoor triggers, which have not been discussed in 2D-Shapley.\n\nIn comparison with Data-OOB, which only provides data-level values, our method is designed to assess cell-level values. This finer granularity in valuation presents a richer interpretation of data values by providing **attribution for data values**. Also, it enables new practical applications.\n\n----\n\n### **Question: Could you please discuss what are the unique challenges of 2D-OOB that do not arise for 2D-KNN and Data-OOB?**\n\n**Re.:** The main technical contribution of 2D-OOB lies in the adoption of a **subset bagging model**, rather than the standard bagging model used in Data-OOB. This requires more careful formulation (as detailed in Equation 4) and presents additional challenges, such as more complex hyperparameter selection (for example, the feature subset ratio, which we have thoroughly examined in the ablation study in Appendix B.4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0yCvf9lNx", "reviewer": "Reviewer_4K7L", "summary": "This paper proposes a joint valuation framework for not only obtaining data value but also attributing a data points value to its individual features (cells). The framework is build on top of data-OOB. The authors compared the proposed 2D-OOB with 2D-KDD on several tasks, including cell-level outlier detection and backdoor trigger detection in data poisoning attacks.", "questions": "1. What does fitting a logistic model in line 187 on page 6 mean? Why change from bagging decision trees to a logistic model in this specific cell fixation experiment?\n\n2. It would be helpful to further explain the difference between Trojan and BadNets trigger methods, as it seems that 2D-OOB is much more effective than 2D-KNN under the Trojan square. This would help us understand in which scenario 2D-OOB works better.\n\n3. Regarding the ablation study: I am also curious about the original prediction performance for each task, not just the outlier detection performance.\n\n4. What is the implication of the connection to data-OOB in Proposition 3.1? Are there any insights we can gain after establishing the connection with data-OOB?", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) This paper targets an important problem that is useful for improving the interpretability of data valuation. The joint valuation provides more fine-grained information. Identifying outliers at the cell level is informative for determining which cell to fix in the next step when improving data quality, especially useful when data acquisition is expensive.\n\n(+) The proposed method naturally integrates the idea of random forest - random feature selection in the construction of weak learners - with data-OOB, offering a new solution for joint evaluation. This approach addresses a relatively underexplored problem, with few existing works in the area. The proposed 2D-OOB method demonstrates greater computational efficiency compared to 2D-KDD.\n\n(+) The improvement in backdoor trigger detection is promising, as cell contamination in this context is targeted and deliberate, unlike artificial random outliers. \n\n(+) The authors provide an ablation study to assess the impact of the selection and number of weak learners, as well as the feature subset ratio, which helps in understanding the stability of the proposed method.", "weaknesses": "(-) Despite the strengths mentioned above, the entire framework is built on top of the OOB framework. This brings a main limitation, as the OOB framework requires the training algorithm to be a bagging method with weak learners. In many applications, bagging may not be the best-performing method, and data valuation under this method may not be of interest.\n\n(-) Typically, the number of randomly selected features for each weak learner is an important hyperparameter to fine-tune. What is the impact when the number of random features is small? While an ablation study was provided in Appendix, certain variation was still observed. Should we correspondingly update the bagging size? Some practical guidance would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness-1: the OOB framework requires the training algorithm to be a bagging method with weak learners**\n\n**Re.:** The primary objective of the joint valuation framework is to evaluate the quality of cells rather than to optimize model performance. The model used serves as a proxy for this evaluation, and it is important to note that a high-performing machine learning method does not necessarily ensure a justified valuation framework.\n\nAdditionally, we believe the bagging framework can be useful in many domains[1-3], and 2D-OOB can be easily integrated into these applications.\n\n[1] Smaida, Mahmoud, and Serhii Yaroshchak. \"Bagging of Convolutional Neural Networks for Diagnostic of Eye Diseases.\" COLINS. 2020.  \n[2] Dong, Yan-Shi, and Ke-Song Han. \"A comparison of several ensemble methods for text categorization.\" IEEE International Conference onServices Computing, 2004.(SCC 2004). Proceedings. 2004. IEEE, 2004.  \n[3] Xinqin, L. I., et al. \"Application of bagging ensemble classifier based on genetic algorithm in the text classification of railway fault hazards.\" 2019 2nd International Conference on Artificial Intelligence and Big Data (ICAIBD). IEEE, 2019.\n \n----\n\n### **Weakness-2: Impact of number of random features and bagging size**\n\n**Re.:** If the number of random features is **too small**, each weak learner becomes nearly random, which significantly diminishes the joint valuation capacity. This issue is independent of the bagging size, as nearly random weak learners, regardless of their quantity, fail to provide meaningful signals for effective evaluation.\n\nIn practice, for low-dimensional datasets, we recommend using a higher feature subset ratio to ensure sufficient capacity of weak learners. For high-dimensional datasets, a lower or moderate feature subset ratio might be more effective in distinguishing the impact of features.\n\n----\n\n### **Question-1: Clarification of fitting a logistic model in line 187 on page 6**\n\n**Re.:** The cell fixation experiment is divided into two separate stages. In the first stage (attribution), we use a subset bagging model with decision trees as weak learners to fit the joint valuation framework. In the second stage (evaluation), our objective is to assess whether low-quality cells were effectively removed, rather than to optimize model performance. Logistic regression was chosen because it is a simple yet powerful machine learning model commonly used to test data separability, which is a common practice in this field [1][2].\n\n[1] Jiang, Kevin, et al. \"Opendataval: a unified benchmark for data valuation.\" NeurIPS. 2023.\n[2] Liu, Zhihong, et al. \"2D-Shapley: A Framework for Fragmented Data Valuation.\" ICML. 2023.\n\n----\n\n### **Question-2: Difference between Trojan and BadNets trigger**\n\n**Re.:** BadNets attack is implemented by directly inserting a white square into the original images. In contrast, Trojan attack specifically targets a pre-trained neural network model. The Trojan trigger is generated by initializing the trigger as a white square and tuning the pixel values in the square to strongly activate some selected neurons in the neural network [1]. We believe the similar performance of the two methods on BadNets square is likely due to the trigger acting as a significant feature. In other words, the trigger pixels significantly differ from the other pixels, making it easily detectable by both methods. The result on Trojan square suggests that 2D-OOB has a stronger detection capability against non-random outliers in applications compared to 2D-KNN, as KNN algorithms focus on local proximity and can fail to effectively capture global patterns in feature behavior. We will include this discussion in the revision.\n\n[1] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and X. Zhang. 352 Trojaning attack on neural networks. In Network and Distributed System Security Symposium, 2018. URL 353 https://api.semanticscholar.org/CorpusID:31806516.\n\n----\n\n### **Question-3: Original prediction performance for each task**\n\n**Re.:** The average accuracy (across B=1000 weak learners) of various weak learners are summarized in the table below. There are certain variations across the datasets. It is important to note that the original prediction performance is not correlated with the joint valuation performance.\n\n|Dataset|DecisionTree|LogisticRegression|MLP(single-layer)|MLP(two-layer)|\n|-------|------------|------------------|-----------------|--------------|\n|lawschool|0.78|0.84|0.83|0.83|\n|electricity|0.65|0.67|0.68|0.68|\n|fried|0.64|0.71|0.71|0.71|\n|2dplanes|0.67|0.70|0.71|0.70|\n|creditcard|0.71|0.79|0.80|0.79|\n|pol|0.78|0.74|0.83|0.84|\n|MiniBooNE|0.80|0.74|0.81|0.84|\n|jannis|0.61|0.68|0.66|0.64|\n|nomao|0.88|0.91|0.92|0.91|\n|vehicle_sensIT|0.73|0.78|0.81|0.79|\n|gas_drift|0.98|0.99|1.00|1.00|\n|musk|0.88|0.90|0.93|0.93|\n\n----\n\n### **Question-4: The implication of the connection to data-OOB in Proposition 3.1.**\n\n**Re.:** Proposition 3.1 shows that the 2D-OOB $\\psi_{ij} ^{\\mathrm{2D-OOB}}$ can be interpreted as the empirical expectation of the Data-OOB, conditioned on the inclusion of the j-th feature. \n\nTo clarify further, let's consider a simple case: assume that for a given data point, each specific feature subset is selected an equal number of times when it is not selected in a bootstrap dataset, i.e., it is in an out-of-bag. Under this assumption, the point mass of $j$-th cell within the data point is $\\frac{1}{\\sum_{l=1}^{L}\\mathbb{1}(j\\in S_l)}$, which uniformly considers all feature coalitions including $j$-th feature. \n\nIt justifies why the definition in Eqn. (4) (line 120) effectively attributes a data point's value to features. It shows that for a fixed $i$ and $ j \\neq k$, $\\psi_{ij} ^{\\mathrm{2D-OOB}}>\\psi_{ik} ^{\\mathrm{2D-OOB}}$​ implies that the cell $x_{ij}$ is more helpful in achieving a high OOB score, serving as an indicator of model performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iRkrciP4bn", "reviewer": "Reviewer_nD8c", "summary": "The paper proposes 2D-OOB, an out-of-bag estimation framework for jointly detecting both samples as well as their cells that are outliers. In other words, 2D-OOB allows attribution of the value of a data sample to its individual features.  Evaluation shows that the framework is also able to localize backdoor triggers in data poisoning attacks. \nIn comparison to the 2D-Shapley model, which also performs joint valuation of a sample and its features, the 2D-OOB is a more efficient and model-agnostic formulation and builds on the subset bagging model, where each weak learner is trained on a randomly selected subset of features. In the experiments, decision trees as used as the weak learners and trained using randomly selected features. The framework measures (using Eqn. (4)), the average accuracy of the bag of predictors for a selected sample and its features.", "questions": "* One of the arguments mentioned in the description of Fig. 2 is that 2D-OOB detects a majority of outlier cells by examining a small fraction of the total cells. It would be great if the authors could further explain this statement and which part of the formulation in (4) motivates this argument.\n\n* In Fig. 3 (showing 2D-OOBs ability to identify and prioritize the outliers to fix), the test accuracy for the Jannis dataset is relatively low compared to the others and the performance is also similar to 2D-KNN. Any insights as to why this is happening?\n\n* In Section 4.4, were the experiments also performed by varying B using a 2-layer MLP? What is the overhead of training an out-of-bag estimator versus a single model (maybe an autoencoder) and using that for outlier detection. \n\n* Are there any other stronger baselines against which comparison can be made?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written.\n* Given the background in moving fro DataShapley to the Data-OOB formulation, the transition from 2D-Shapley to 2D-OOB was intuitive and easy to follow. \n* The proposed method can identify data poisoning attack triggers (square patches placed in the original images to elicit misclassification) which could be very useful.", "weaknesses": "* The paper is low on novelty. Given, OOB-Data formulation and the 2D-Shapley for joint valuation, the 2D-OOB formulation seems like a natural extension -- interesting, but not high on novelty.\n\n* In Fig. 4, panel B, for BadNets poisoning attack, the detection performance of 2D-OOB and 2D-KNN (proxy for 2D-Shapley) are very similar. It is difficult to assess how 2D-OOB will perform for more stronger poisoning attacks. A more detailed study on this aspect as well as analysis showing that 2D-OOB is more likely to identify such triggers, even when attack takes such a detection mechanism into account, will be a good indicator of the strength of this framework.\n\n* Table 1, shows the average run time is much improved from 2D-KNN, however, most of the datasets used (Appendix A, Table 3) have small input dimensions (only three have >= 100 but have small sample size) and hence it is difficult to assess how the technique would scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness-1: Novelty**\n\n**Re.:** Thank you for your feedback. We acknowledge that 2D-OOB may initially appear to be a natural extension of existing methods, as it is built on them. However, we believe our contribution is substantial in the field of data valuation, as our proposed method addresses both practical and conceptual challenges of 2D-Shapley and Data-OOB.\n\nCompared to 2D-Shapely, our work significantly improves **computational efficiency**, as demonstrated by the empirical experiments in the submitted paper and the new results in our general response. In addition, we identify **new** use cases of the joint valuation framework, such as attributing data contribution through cell values, correcting erroneous cells, and detecting backdoor triggers, which have not been discussed in 2D-Shapley.\n\nIn comparison with Data-OOB, which only provides data-level values, our method is designed to assess cell-level values. This finer granularity in valuation presents a richer interpretation of data values by providing **attribution for data values**. Also, it enables new practical applications.\n\n----\n\n### **Weakness-2: Detailed study on different attacks**\n\n**Re.:**  BadNets and Trojan attacks are popular data poisoning techniques that maliciously contaminate the training datasets by injecting samples patched with a pre-defined trigger [1,2]. To expand our analysis, we explored the impact of varying types of triggers in additional experiments. All variations we used in the experiments are motivated by [1,2]. 2D-OOB consistently demonstrates superior detection capabilities over 2D-KNN across all trigger variations. This result suggests that 2D-OOB has a stronger detection capability against non-random outliers in applications compared to 2D-KNN, as KNN algorithms focus on local proximity and can fail to capture global patterns in feature behavior effectively.\n\nTable: Additional experiment results on backdoor trigger detection. Five independent experiments are conducted on each dataset and the average detection AUC across all datasets is presented. The average and standard error of the detection AUC are denoted by “average ± standard error”.\n\n| Trigger         | 2D-OOB (ours) | 2D-KNN |\n|-------------|-------------|-------------|\n| BadNets White Square  |         0.97$\\pm$0.01      |  0.90$\\pm$0.03 |\n| BadNets Yellow Square |   0.98 $\\pm$0.004      |  0.81$\\pm$0.004  |\n| BadNets White Triangle |  0.95 $\\pm$0.01         |  0.91$\\pm$0.02  |\n| Trojan White Square |  0.92 $\\pm$0.005       |  0.76$\\pm$0.006 |\n| Trojan Black Apple Logo |  0.88$\\pm$0.008      |  0.78$\\pm$0.04 |\n| Trojan White Apple Logo |  0.87$\\pm$0.006      |  0.76$\\pm$0.04 |\n\n\n[1] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017.\n[2] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and X. Zhang. 352 Trojaning attack on neural networks. In Network and Distributed System Security Symposium, 2018. URL 353 https://api.semanticscholar.org/CorpusID:31806516.\n\n\n----\n\n### **Weakness-3: Scalability**\n\n**Re.:** We have included the additional experimental results on the scalability of 2D-OOB in the overall response above. Please let us know if there are any other aspects you'd like us to discuss further.\n\n----\n\n### **Question-1: Fig. 2 statement that 2D-OOB detects a majority of outlier cells by examining a small fraction of the total cells**\n\n**Re.:** 2D-OOB assigns a quality score to each cell, where smaller values indicate poorer quality and a higher likelihood of being outliers. Consequently, by checking cells with low scores, we can effectively scan out most outliers, as shown in Fig 2. \n\nFor Equation 4, the underlying intuition is that if a particular feature of a data point (i.e., a cell) consistently results in a high OOB error when used in weak learners, it indicates that the cell is of poor quality and potentially a harmful outlier, thus deserving a low score.\n\n----\n\n### **Question-2: Fig. 3 results**\n\n**Re.:** The relatively low test accuracy observed for the Jannis dataset can be attributed to inherent variations in data separability across different datasets. The Jannis dataset may exhibit more complex patterns or higher noise levels. For example, the covariates may not be useful enough to predict their labels. The performance similarity between 2D-OOB and 2D-KNN indicates that both methods might face similar challenges with this dataset.\n\n----\n\n### **Question-3: Section 4.4 results**\n\n**Re.:** For different weak learners (including 2-layer MLP), we use a consistent B of 1000, which is large enough to guarantee convergence empirically. \n\nAn autoencoder has the potential to identify outliers, while our out-of-bag joint valuation framework can provide **richer** information (i.e., identify both low-quality and **high-quality** cells). Given the differences in the objective of these methods, comparing the overhead of training an out-of-bag estimator versus a single model like an autoencoder might not be directly meaningful.\n\nWe hope this addresses your concern. Please let us know if there are any other aspects you'd like us to discuss further.\n\n----\n\n### **Question-4: Stronger baselines**\n\n**Re.:** Thank you for the question. Given that the joint valuation problem is still emerging, with the first concept introduced in 2023 [1], there is no established baseline beyond 2D-KNN. The proposed 2D-OOB demonstrates both practical effectiveness and computational efficiency compared to 2D-KNN in many downstream tasks. \n\n[1] Liu, Zhihong, et al. \"2D-Shapley: A Framework for Fragmented Data Valuation.\" ICML. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzyofYd4Zs", "reviewer": "Reviewer_ECZV", "summary": "This paper studies data attribution. Unlike the previous works that only quantifies the contribution of each datum, this work also considers the attribution of the cells within every datum. The proposed the joint valuation framework, 2D-OOB, can accurately identify cell outliers, as well as the poisoning attacks. The experiments verify the authors claims and show the efficiency of the proposed method.", "questions": "See weaknesses", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method has been theoretically generalized and connected to the existing works.\n3. The experiments are comprehensive and show the efficiency of the proposed method.", "weaknesses": "1. In section 4.3, the experiments did not mention that the classification label is altered or not after poisoning (e.g, from correct to incorrect).\n2. The paper does not state clearly about the differences of the application on image, comparing with the other model attribution methods? They both assign a value to each feature/pixel/cell in the input data. Why would the proposed method be better in this scenario?\n3. How about the scalability of the proposed method? The experiments are either conducted on tabular data or small images. The large images or high-dimensional data are not mentioned.\n4. What is the distance regularization term mentioned in L168 in the utility function $T$?\n5. The size of CIFAR-10 is 32x32, but the cell valuation displayed in Figure 5 is 16x16. Why is that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness-1: Section 4.3 Clarification**\n\n**Re.:**  In line 216 on page 7, we explicitly stated that the poisoned samples are relabeled from the source class to the target class. \n\n----\n\n### **Weakness-2: Difference between 2D-OOB and image feature attribution methods**\n\n**Re.:** Thank you for pointing this out. As the reviewer mentioned, both feature attribution methods and the joint valuation method we studied in this paper assign a score to each cell. However, we would like to clarify that they serve distinct purposes and are conceptually different.\n\nFeature attribution methods measure the impact of a particular data point’s features on its model prediction. In the case of image classification, these methods explain which parts of an image are most influential for its model's prediction.\nIn contrast, the joint valuation method measures the impact of each cell in the training dataset on the model training process. Hence, it explains which parts of training images are most influential or potentially harmful to the model's performance.\n\n----\n\n### **Weakness-3: Scalability**\n\n**Re.:** We have included the additional experimental results on the scalability of 2D-OOB in the general response.\n\n----\n\n### **Weakness-4:Regularization term**\n\n**Re.:** We calculate the normalized (min-max normalization) negative L2 distance between covariates and the class-specific mean in the bootstrap dataset, as stated in L439-440. The normalization is performed across those samples within the same class. We will present a more explicit formulation in the revision.\n\n----\n\n### **Weakness-5: Super-pixel**\n\n**Re.:**  We employ a standard super-pixel technique to transform the (32,32,3) image into a 256-dimensional vector (This is explicitly stated in line 427 page 13 of our submitted manuscript). Specifically, we partitioned the images into equally sized $2\\times2$ grids and used average pooling to reduce the pixel values to a single cell value. The super-pixel technique we employed has been widely used in numerous previous explainable machine learning works [1-4] to highlight key factors in a simplified illustration. \n\n[1] Dardouillet, Pierre, et al. \"Explainability of image semantic segmentation through shap values.\" International Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2022.   \n[2] Vermeire, Tom, et al. \"Explainable image classification with evidence counterfactual.\" Pattern Analysis and Applications 25.2 (2022): 315-335.   \n[3] Shah, Sumeet S., and John W. Sheppard. \"Evaluating explanations of convolutional neural network image classifications.\" 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.  \n[4] Jethani, Neil, et al. \"Fastshap: Real-time shapley value estimation.\" International Conference on Learning Representations. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers and AC for their dedicated work. We are appreciative that reviewers agreed our work to be **well-motivated, clearly-written, and include extensive experiments**. The concept of joint valuation was recognized **reasonable** (6nYm) and **important** (4K7L). Our proposed framework was considered **easy to follow** (4K7L, nD8c) and **theoretically justified** (ECZV). Reviewers 6nYm, 4K7L and ECZV commended **the algorithm's superior computational efficiency compared to existing methods**. Reviewers 6nYm, 4K7L and nD8c acknowledged the backdoor trigger detection experiment as **useful, interesting and promising**.

As major additions, we have added **a new set of experiments on the scalability of 2D-OOB**.

## Experiment on the scalability of 2D-OOB (Reviewers 4K7L, nD8c)

We evaluate the computational efficiency of our proposed approach using a synthetic binary classification dataset. For $d \in \lbrace 20, 100, 500, 1000 \rbrace$, input data $X \in \mathbb{R}^d$ is sampled from a multivariate Gaussian distribution with a mean of zero and an identity covariance matrix. The output labels Y in $Y \in \{0, 1\}$ are generated using a Bernoulli distribution with a success probability determined by $p(X) := 1/(1 + exp(-X^T\eta))$, where each component of the vector $\eta \in \mathbb{R}^d$ is sampled from a standard Gaussian distribution. We experiment with sample sizes $n \in \lbrace10^3, 10^4, 10^5\rbrace$. We record the elapsed time using the same computational device. The elapsed time for 2D-OOB includes the training time for the subset bagging model to ensure a fair comparison.

As shown in the table below, our 2D-OOB enjoys remarkable scalability, in terms of both sample size n and feature dimension d. We also examined 2D-KNN under the same settings. However, for larger datasets, 2D-KNN required more than 5 hours to complete (which are omitted from the table). Notably, 2D-KNN with n=10000 and d=20 took 7556.32 seconds, which is even slower than our method with n=100000 (10x larger) and d=1000 (50x larger), which took 7423.87 seconds.


| n      | d    | 2D-OOB (ours) | 2D-KNN |
|--------|------|---------------|--------|
| 1000   | 20   | 10.16         |   714.89     |
| 1000   | 100  | 17.35         |   3564.51     |
| 1000   | 500  | 56.67         |   /     |
| 1000   | 1000 | 113.06        |    /    |
| 10000  | 20   | 53.89         |   7556.32     |
| 10000  | 100  | 165.55        |   /     |
| 10000  | 500  | 681.60        |   /     |
| 10000  | 1000 | 1303.33       |  /      |
| 100000 | 20   | 604.38        |   /     |
| 100000 | 100   | 845.83        |      /  |
| 100000 | 500   | 3927.31       |    /    |
| 100000 | 1000   | 7423.87       |    /    |

Table: Runtime (in seconds) for 2D-OOB under different data sizes (n) and input dimensions (d). We omitted the runtime for cases where the computation exceeded 5 hours.

Meta-review: The rebuttal provided by the authors have addressed most of the reviewers' questions, by clarifying parts of the submission and providing additional experiments. However, some of the reviewers also point out that the novelty of the submission and its applicability may be limited.
Decision: Accept (poster)